Xilinx Runtime (XRT) Core Library

Xilinx Runtime (XRT) Library Interface Definitions

Header file xrt.h defines data structures and function signatures exported by Xilinx Runtime (XRT) Library. XRT is part of software stack which is integrated into Xilinx reference platform.

typedef xclDeviceHandle

opaque device handle

Description

A device handle of xclDeviceHandle kind is obtained by opening a device. Clients pass this device handle to refer to the opened device in all future interaction with XRT.

XRT Device Management APIs

---

XCL_DRIVER_DLLESPEC unsigned xclProbe()

Enumerate devices found in the system

Parameters

Return

count of devices found

XCL_DRIVER_DLLESPEC xclDeviceHandle xclOpen(unsigned deviceIndex, const char * logFileName, enum xclVerbosityLevel level)

Open a device and obtain its handle.

Parameters

unsigned deviceIndex

Slot number of device 0 for first device, 1 for the second device…

const char * logFileName

Log file to use for optional logging

enum xclVerbosityLevel level

Severity level of messages to log

Return

Device handle

XCL_DRIVER_DLLESPEC void xclClose(xclDeviceHandle handle)

Close an opened device

Parameters

xclDeviceHandle handle

Device handle

XCL_DRIVER_DLLESPEC int xclResetDevice(xclDeviceHandle handle, enum xclResetKind kind)

Reset a device or its CL

Parameters

xclDeviceHandle handle

Device handle

enum xclResetKind kind

Reset kind

Return

0 on success or appropriate error number

Reset the device. All running kernels will be killed and buffers in DDR will be purged. A device may be reset if a user’s application dies without waiting for running kernel(s) to finish.

XCL_DRIVER_DLLESPEC int xclGetDeviceInfo2(xclDeviceHandle handle, struct xclDeviceInfo2 * info)

Obtain various bits of information from the device

Parameters

xclDeviceHandle handle

Device handle

struct xclDeviceInfo2 * info

Information record

Return

0 on success or appropriate error number

XCL_DRIVER_DLLESPEC int xclGetUsageInfo(xclDeviceHandle handle, struct xclDeviceUsage * info)

Obtain usage information from the device

Parameters

xclDeviceHandle handle

Device handle

struct xclDeviceUsage * info

Information record

Return

0 on success or appropriate error number

XCL_DRIVER_DLLESPEC int xclGetErrorStatus(xclDeviceHandle handle, struct xclErrorStatus * info)

Obtain error information from the device

Parameters

xclDeviceHandle handle

Device handle

struct xclErrorStatus * info

Information record

Return

0 on success or appropriate error number

XCL_DRIVER_DLLESPEC int xclLoadXclBin(xclDeviceHandle handle, const struct axlf * buffer)

Download FPGA image (xclbin) to the device

Parameters

xclDeviceHandle handle

Device handle

const struct axlf * buffer

Pointer to device image (xclbin) in memory

Return

0 on success or appropriate error number

Download FPGA image (AXLF) to the device. The PR bitstream is encapsulated inside xclbin as a section. xclbin may also contains other sections which are suitably handled by the driver.

XCL_DRIVER_DLLESPEC int xclLoadXclBinMgmt(xclDeviceHandle handle, const struct axlf * buffer)

Download FPGA image (xclbin) to the device via mgmtpf

Parameters

xclDeviceHandle handle

Device handle

const struct axlf * buffer

Pointer to device image (xclbin) in memory

Return

0 on success or appropriate error number

Download FPGA image (AXLF) to the device. The PR bitstream is encapsulated inside xclbin as a section. xclbin may also contains other sections which are suitably handled by the driver.

XCL_DRIVER_DLLESPEC int xclGetSectionInfo(xclDeviceHandle handle, void * info, size_t * size, enum axlf_section_kind kind, int index)

Get Information from sysfs about the downloaded xclbin sections

Parameters

xclDeviceHandle handle

Device handle

void * info

Pointer to preallocated memory which will store the return value.

size_t * size

Pointer to preallocated memory which will store the return size. kind: axlf_section_kind for which info is being queried index: The (sub)section index for the “kind” type.

enum axlf_section_kind kind

undescribed

int index

undescribed

Return

0 on success or appropriate error number

Get the section information from sysfs. The index corrresponds to the (section) entry of the axlf_section_kind data being queried. The info and the size contain the return binary value of the subsection and its size.

XCL_DRIVER_DLLESPEC int xclReClock2(xclDeviceHandle handle, unsigned short region, const unsigned short * targetFreqMHz)

Configure PR region frequncies

Parameters

xclDeviceHandle handle

Device handle

unsigned short region

PR region (always 0)

const unsigned short * targetFreqMHz

Array of target frequencies in order for the Clock Wizards driving the PR region

Return

0 on success or appropriate error number

XCL_DRIVER_DLLESPEC int xclReClockUser(xclDeviceHandle handle, unsigned short region, const unsigned short * targetFreqMHz)

Configure PR region frequncies via userpf

Parameters

xclDeviceHandle handle

Device handle

unsigned short region

PR region (always 0)

const unsigned short * targetFreqMHz

Array of target frequencies in order for the Clock Wizards driving the PR region

Return

0 on success or appropriate error number

XCL_DRIVER_DLLESPEC int xclLockDevice(xclDeviceHandle handle)

Get exclusive ownership of the device

Parameters

xclDeviceHandle handle

Device handle

Return

0 on success or appropriate error number

The lock is necessary before performing buffer migration, register access or bitstream downloads.

XCL_DRIVER_DLLESPEC int xclUnlockDevice(xclDeviceHandle handle)

Release exclusive ownership of the device

Parameters

xclDeviceHandle handle

Device handle

Return

0 on success or appropriate error number

XCL_DRIVER_DLLESPEC int xclOpenContext(xclDeviceHandle handle, uuid_t xclbinId, unsigned int ipIndex, bool shared)

Create shared/exclusive context on compute units

Parameters

xclDeviceHandle handle

Device handle

uuid_t xclbinId

UUID of the xclbin image running on the device

unsigned int ipIndex

IP/CU index in the IP LAYOUT array

bool shared

Shared access or exclusive access

Return

0 on success or appropriate error number

The context is necessary before submitting execution jobs using xclExecBuf(). Contexts may be exclusive or shared. Allocation of exclusive contexts on a compute unit would succeed only if another client has not already setup up a context on that compute unit. Shared contexts can be concurrently allocated by many processes on the same compute units.

XCL_DRIVER_DLLESPEC int xclCloseContext(xclDeviceHandle handle, uuid_t xclbinId, unsigned ipIndex)

Close previously opened context

Parameters

xclDeviceHandle handle

Device handle

uuid_t xclbinId

UUID of the xclbin image running on the device

unsigned ipIndex

IP/CU index in the IP LAYOUT array

Return

0 on success or appropriate error number

Close a previously allocated shared/exclusive context for a compute unit.

XCL_DRIVER_DLLESPEC int xclBootFPGA(xclDeviceHandle handle)

Boot the FPGA from PROM

Parameters

xclDeviceHandle handle

Device handle

Return

0 on success or appropriate error number

This should only be called when there are no other clients. It will cause PCIe bus re-enumeration

XCL_DRIVER_DLLESPEC int xclLogMsg(xclDeviceHandle handle, enum xrtLogMsgLevel level, const char * tag, const char * format, ...)

Send message to log file as per settings in ini file.

Parameters

xclDeviceHandle handle

Device handle

enum xrtLogMsgLevel level

Severity level of the msg

const char * tag

Tag supplied by the client, like “OCL”, “XMA”, etc.

const char * format

Format of Msg string to write to log file

...

All other arguments as per the format

Return

0 on success or appropriate error number

XRT Buffer Management APIs

---

Buffer management APIs are used for managing device memory and migrating buffers between host and device memory

XCL_DRIVER_DLLESPEC unsigned int xclAllocBO(xclDeviceHandle handle, size_t size, enum xclBOKind domain, unsigned flags)

Allocate a BO of requested size with appropriate flags

Parameters

xclDeviceHandle handle

Device handle

size_t size

Size of buffer

enum xclBOKind domain

Memory domain

unsigned flags

Specify bank information, etc

Return

BO handle

XCL_DRIVER_DLLESPEC unsigned int xclAllocUserPtrBO(xclDeviceHandle handle, void * userptr, size_t size, unsigned flags)

Allocate a BO using userptr provided by the user

Parameters

xclDeviceHandle handle

Device handle

void * userptr

Pointer to 4K aligned user memory

size_t size

Size of buffer

unsigned flags

Specify bank information, etc

Return

BO handle

XCL_DRIVER_DLLESPEC void xclFreeBO(xclDeviceHandle handle, unsigned int boHandle)

Free a previously allocated BO

Parameters

xclDeviceHandle handle

Device handle

unsigned int boHandle

BO handle

XCL_DRIVER_DLLESPEC size_t xclWriteBO(xclDeviceHandle handle, unsigned int boHandle, const void * src, size_t size, size_t seek)

Copy-in user data to host backing storage of BO

Parameters

xclDeviceHandle handle

Device handle

unsigned int boHandle

BO handle

const void * src

Source data pointer

size_t size

Size of data to copy

size_t seek

Offset within the BO

Return

0 on success or appropriate error number

Copy host buffer contents to previously allocated device memory. seek specifies how many bytes to skip at the beginning of the BO before copying-in size bytes of host buffer.

XCL_DRIVER_DLLESPEC size_t xclReadBO(xclDeviceHandle handle, unsigned int boHandle, void * dst, size_t size, size_t skip)

Copy-out user data from host backing storage of BO

Parameters

xclDeviceHandle handle

Device handle

unsigned int boHandle

BO handle

void * dst

Destination data pointer

size_t size

Size of data to copy

size_t skip

Offset within the BO

Return

0 on success or appropriate error number

Copy contents of previously allocated device memory to host buffer. skip specifies how many bytes to skip from the beginning of the BO before copying-out size bytes of device buffer.

XCL_DRIVER_DLLESPEC void * xclMapBO(xclDeviceHandle handle, unsigned int boHandle, bool write)

Memory map BO into user’s address space

Parameters

xclDeviceHandle handle

Device handle

unsigned int boHandle

BO handle

bool write

READ only or READ/WRITE mapping

Return

Memory mapped buffer

Map the contents of the buffer object into host memory To unmap the buffer call POSIX unmap() on mapped void * pointer returned from xclMapBO

XCL_DRIVER_DLLESPEC int xclSyncBO(xclDeviceHandle handle, unsigned int boHandle, enum xclBOSyncDirection dir, size_t size, size_t offset)

Synchronize buffer contents in requested direction

Parameters

xclDeviceHandle handle

Device handle

unsigned int boHandle

BO handle

enum xclBOSyncDirection dir

To device or from device

size_t size

Size of data to synchronize

size_t offset

Offset within the BO

Return

0 on success or standard errno

Synchronize the buffer contents between host and device. Depending on the memory model this may require DMA to/from device or CPU cache flushing/invalidation

XCL_DRIVER_DLLESPEC int xclCopyBO(xclDeviceHandle handle, unsigned int dstBoHandle, unsigned int srcBoHandle, size_t size, size_t dst_offset, size_t src_offset)

Copy device buffer contents to another buffer

Parameters

xclDeviceHandle handle

Device handle

unsigned int dstBoHandle

Destination BO handle

unsigned int srcBoHandle

Source BO handle

size_t size

Size of data to synchronize

size_t dst_offset

dst Offset within the BO

size_t src_offset

src Offset within the BO

Return

0 on success or standard errno

Copy from source buffer contents to destination buffer, can be device to device or device to host. Always perform WRITE to achieve better performance, destination buffer can be on device or host require DMA from device

XCL_DRIVER_DLLESPEC int xclExportBO(xclDeviceHandle handle, unsigned int boHandle)

Obtain DMA-BUF file descriptor for a BO

Parameters

xclDeviceHandle handle

Device handle

unsigned int boHandle

BO handle which needs to be exported

Return

File handle to the BO or standard errno

Export a BO for import into another device or Linux subsystem which accepts DMA-BUF fd This operation is backed by Linux DMA-BUF framework

XCL_DRIVER_DLLESPEC unsigned int xclImportBO(xclDeviceHandle handle, int fd, unsigned flags)

Obtain BO handle for a BO represented by DMA-BUF file descriptor

Parameters

xclDeviceHandle handle

Device handle

int fd

File handle to foreign BO owned by another device which needs to be imported

unsigned flags

Unused

Return

BO handle of the imported BO

Import a BO exported by another device. * This operation is backed by Linux DMA-BUF framework

XCL_DRIVER_DLLESPEC int xclGetBOProperties(xclDeviceHandle handle, unsigned int boHandle, struct xclBOProperties * properties)

Obtain xclBOProperties struct for a BO

Parameters

xclDeviceHandle handle

Device handle

unsigned int boHandle

BO handle

struct xclBOProperties * properties

BO properties struct pointer

Return

0 on success

This is the prefered method for obtaining BO property information.

XRT Legacy Buffer Management APIs

---

Do not develop new features using the following 5 API’s. These are for backwards compatibility with classic XRT interface and will be deprecated in future. New clients should use BO based APIs defined above

XRT_DEPRECATED XCL_DRIVER_DLLESPEC uint64_t xclAllocDeviceBuffer(xclDeviceHandle handle, size_t size)

Allocate a buffer on the device

Parameters

xclDeviceHandle handle

Device handle

size_t size

Size of buffer

Return

Physical address of buffer on device or 0xFFFFFFFFFFFFFFFF in case of failure

Allocate a buffer on the device DDR and return its address. This API is deprecated and will be removed in future release. Use xclAllocBO() in all new code.

XRT_DEPRECATED XCL_DRIVER_DLLESPEC uint64_t xclAllocDeviceBuffer2(xclDeviceHandle handle, size_t size, enum xclMemoryDomains domain, unsigned flags)

Allocate a buffer on the device on a specific DDR

Parameters

xclDeviceHandle handle

Device handle

size_t size

Size of buffer

enum xclMemoryDomains domain

Memory domain

unsigned flags

Desired DDR bank as a bitmap.

Return

Physical address of buffer on device or 0xFFFFFFFFFFFFFFFF in case of failure

Allocate a buffer on a specific device DDR and return its address. This API is deprecated and will be removed in future release. Use xclAllocBO() in all new code.

XRT_DEPRECATED XCL_DRIVER_DLLESPEC void xclFreeDeviceBuffer(xclDeviceHandle handle, uint64_t buf)

Free a previously buffer on the device

Parameters

xclDeviceHandle handle

Device handle

uint64_t buf

Physical address of buffer

Description

The physical address should have been previously allocated by xclAllocDeviceBuffe() or xclAllocDeviceBuffer2(). The address should point to the beginning of the buffer and not at an offset in the buffer. This API is deprecated and will be removed in future release. Use xclFreeBO() together with BO allocation APIs.

XRT_DEPRECATED XCL_DRIVER_DLLESPEC size_t xclCopyBufferHost2Device(xclDeviceHandle handle, uint64_t dest, const void * src, size_t size, size_t seek)

Write to device memory

Parameters

xclDeviceHandle handle

Device handle

uint64_t dest

Physical address in the device

const void * src

Source buffer pointer

size_t size

Size of data to synchronize

size_t seek

Seek within the segment pointed to physical address

Return

Size of data moved or standard error number

Copy host buffer contents to previously allocated device memory. seek specifies how many bytes to skip at the beginning of the destination before copying size bytes of host buffer. This API is deprecated and will be removed in future release. Use xclSyncBO() together with other BO APIs.

XRT_DEPRECATED XCL_DRIVER_DLLESPEC size_t xclCopyBufferDevice2Host(xclDeviceHandle handle, void * dest, uint64_t src, size_t size, size_t skip)

Read from device memory

Parameters

xclDeviceHandle handle

Device handle

void * dest

Destination buffer pointer

uint64_t src

Physical address in the device

size_t size

Size of data to synchronize

size_t skip

Skip within the segment pointed to physical address

Return

Size of data moved or standard error number

Copy contents of previously allocated device memory to host buffer. skip specifies how many bytes to skip from the beginning of the source before copying size bytes of device buffer. This API is deprecated and will be removed in future release. Use xclSyncBO() together with other BO APIs.

XRT Unmanaged DMA APIs

---

Unmanaged DMA APIs are for exclusive use by the debuggers and tools. The APIs allow clinets to read/write from/to absolute device address. No checks are performed if a buffer was allocated before at the specified location or if the address is valid. Users who want to take over the full memory managemnt of the device may use this API to synchronize their buffers between host and device.

XCL_DRIVER_DLLESPEC ssize_t xclUnmgdPread(xclDeviceHandle handle, unsigned flags, void * buf, size_t size, uint64_t offset)

Perform unmanaged device memory read operation

Parameters

xclDeviceHandle handle

Device handle

unsigned flags

Unused

void * buf

Destination data pointer

size_t size

Size of data to copy

uint64_t offset

Absolute offset inside device

Return

size of bytes read or appropriate error number

This API may be used to perform DMA operation from absolute location specified. Users may use this if they want to perform their own device memory management – not using the buffer object (BO) framework defined before.

XCL_DRIVER_DLLESPEC ssize_t xclUnmgdPwrite(xclDeviceHandle handle, unsigned flags, const void * buf, size_t size, uint64_t offset)

Perform unmanaged device memory read operation

Parameters

xclDeviceHandle handle

Device handle

unsigned flags

Unused

const void * buf

Source data pointer

size_t size

Size of data to copy

uint64_t offset

Absolute offset inside device

Return

size of bytes written or appropriate error number

This API may be used to perform DMA operation to an absolute location specified. Users may use this if they want to perform their own device memory management – not using the buffer object (BO) framework defined before.

XCL_DRIVER_DLLESPEC size_t xclWrite(xclDeviceHandle handle, enum xclAddressSpace space, uint64_t offset, const void * hostBuf, size_t size)

Perform register write operation

Parameters

xclDeviceHandle handle

Device handle

enum xclAddressSpace space

Address space

uint64_t offset

Offset in the address space

const void * hostBuf

Source data pointer

size_t size

Size of data to copy

Return

size of bytes written or appropriate error number

This API may be used to write to device registers exposed on PCIe BAR. Offset is relative to the the address space. A device may have many address spaces. This API will be deprecated in future. Please use this API only for IP bringup/debugging. For execution management please use XRT Compute Unit Execution Management APIs defined below.

XCL_DRIVER_DLLESPEC size_t xclRead(xclDeviceHandle handle, enum xclAddressSpace space, uint64_t offset, void * hostbuf, size_t size)

Perform register read operation

Parameters

xclDeviceHandle handle

Device handle

enum xclAddressSpace space

Address space

uint64_t offset

Offset in the address space

void * hostbuf

Destination data pointer

size_t size

Size of data to copy

Return

size of bytes written or appropriate error number

This API may be used to read from device registers exposed on PCIe BAR. Offset is relative to the the address space. A device may have many address spaces. This API will be deprecated in future. Please use this API only for IP bringup/debugging. For execution management please use XRT Compute Unit Execution Management APIs defined below

XRT Compute Unit Execution Management APIs

These APIs are under development. These functions will be used to start compute units and wait for them to finish.

XCL_DRIVER_DLLESPEC int xclExecBuf(xclDeviceHandle handle, unsigned int cmdBO)

Submit an execution request to the embedded (or software) scheduler

Parameters

xclDeviceHandle handle

Device handle

unsigned int cmdBO

BO handle containing command packet

Return

0 or standard error number

Submit an exec buffer for execution. The exec buffer layout is defined by struct ert_packet which is defined in file ert.h. The BO should been allocated with DRM_XOCL_BO_EXECBUF flag.

XCL_DRIVER_DLLESPEC int xclExecBufWithWaitList(xclDeviceHandle handle, unsigned int cmdBO, size_t num_bo_in_wait_list, unsigned int * bo_wait_list)

Submit an execution request to the embedded (or software) scheduler

Parameters

xclDeviceHandle handle

Device handle

unsigned int cmdBO

BO handle containing command packet

size_t num_bo_in_wait_list

Number of BO handles in wait list

unsigned int * bo_wait_list

BO handles that must complete execution before cmdBO is started

Return

0 or standard error number

Submit an exec buffer for execution. The BO handles in the wait list must complete execution before cmdBO is started. The BO handles in the wait list must have beeen submitted prior to this call to xclExecBufWithWaitList.

XCL_DRIVER_DLLESPEC int xclExecWait(xclDeviceHandle handle, int timeoutMilliSec)

Wait for one or more execution events on the device

Parameters

xclDeviceHandle handle

Device handle

int timeoutMilliSec

How long to wait for

Return

Same code as poll system call

Wait for notification from the hardware. The function essentially calls “poll” system call on the driver file handle. The return value has same semantics as poll system call. If return value is > 0 caller should check the status of submitted exec buffers Note that if you perform wait for the same handle from multiple threads, you may lose wakeup for some of them. So, use different handle in different threads.

XCL_DRIVER_DLLESPEC int xclRegisterInterruptNotify(xclDeviceHandle handle, unsigned int userInterrupt, int fd)

register eventfd file handle for a MSIX interrupt

Parameters

xclDeviceHandle handle

Device handle

unsigned int userInterrupt

MSIX interrupt number

int fd

Eventfd handle

Return

0 on success or standard errno

Support for non managed interrupts (interrupts from custom IPs). fd should be obtained from eventfd system call. Caller should use standard poll/read eventfd framework in order to wait for interrupts. The handles are automatically unregistered on process exit.

XRT Stream Queue APIs

These functions are used for next generation DMA Engine, QDMA. QDMA provides not only memory mapped DMA which moves data between host memory and board memory, but also stream DMA which moves data between host memory and kernel directly. XDMA memory mapped DMA APIs are also supported on QDMA. New stream APIs are provided here for preview and may be revised in a future release. These can only be used with platforms with QDMA engine under the hood. The higher level OpenCL based streaming APIs offer more refined interfaces and compatibility between releases.

Performance Monitoring Operations

These functions are used to read and write to the performance monitoring infrastructure. OpenCL runtime will be using the BUFFER MANAGEMNT APIs described above to manage OpenCL buffers. It would use these functions to initialize and sample the performance monitoring on the card. Note that the offset is wrt the address space