XOCL (PCIe User Physical Function) Driver Interfaces¶

A GEM style driver for Xilinx PCIe based accelerators

File xocl_ioctl.h defines ioctl command codes and associated structures for interacting with xocl PCI driver for Xilinx FPGA platforms.

Device memory allocation is modeled as buffer objects (bo). For each bo driver tracks the host pointer backed by scatter gather list – which provides backing storage on host – and the corresponding device side allocation of contiguous buffer in one of the memory mapped DDRs/BRAMs, etc.

Exection model is asynchronous where execute commands are submitted using command buffers and POSIX poll is used to wait for finished commands. Commands for a compute unit can only be submitted after an explicit context has been opened by the client.

“xocl” driver allows user land to perform mmap on multiple entities distinguished by offset: - page offset == 0: whole user BAR is mapped - page offset > 0 and <= 128: one CU reg space is mapped, offset is used as CU index - page offset >= (4G >> PAGE_SHIFT): one BO is mapped, offset should be obtained from drm_xocl_map_bo()

xocl driver functionality is described in the following table. All the APIs are multi-threading and multi-process safe.

#	Functionality	ioctl request code	data format
1	Allocate buffer on device	DRM_IOCTL_XOCL_CREATE_BO	drm_xocl_create_bo
2	Allocate buffer on device with userptr	DRM_IOCTL_XOCL_USERPTR_BO	drm_xocl_userptr_bo
3	Prepare bo for mapping into user’s address space	DRM_IOCTL_XOCL_MAP_BO	drm_xocl_map_bo
4	Synchronize (DMA) buffer contents in requested direction	DRM_IOCTL_XOCL_SYNC_BO	drm_xocl_sync_bo
5	Obtain information about buffer object	DRM_IOCTL_XOCL_INFO_BO	drm_xocl_info_bo
6	Update bo backing storage with user’s data	DRM_IOCTL_XOCL_PWRITE_BO	drm_xocl_pwrite_bo
7	Read back data in bo backing storage	DRM_IOCTL_XOCL_PREAD_BO	drm_xocl_pread_bo
8	Open/close a context on a compute unit on the device	DRM_IOCTL_XOCL_CTX	drm_xocl_ctx
9	Create a hw context on a slot for a xclbin on the device	DRM_IOCTL_XOCL_CREATE_HW_CTX	drm_xocl_create_hw_ctx
10	Destroy a hw context on a slot for a xclbin on the device	DRM_IOCTL_XOCL_DESTROY_HW_CTX	drm_xocl_destroy_hw_ctx
11	Open a context on a compute unit on the device	DRM_IOCTL_XOCL_OPEN_CU_CTX	drm_xocl_open_cu_ctx
12	Close a context on a compute unit on the device	DRM_IOCTL_XOCL_CLOSE_CU_CTX	drm_xocl_close_cu_ctx
13	Unprotected write to device memory	DRM_IOCTL_XOCL_PWRITE_UNMGD	drm_xocl_pwrite_unmgd
14	Unprotected read from device memory	DRM_IOCTL_XOCL_PREAD_UNMGD	drm_xocl_pread_unmgd
15	Send an execute job to a compute unit	DRM_IOCTL_XOCL_EXECBUF	drm_xocl_execbuf
16	Send an execute job to a compute unit	DRM_IOCTL_XOCL_HW_CTX_EXECBUF	drm_xocl_hw_ctx_execbuf
17	Register eventfd handle for MSIX interrupt	DRM_IOCTL_XOCL_USER_INTR	drm_xocl_user_intr
18	Update device view with a specific xclbin image	DRM_IOCTL_XOCL_READ_AXLF	drm_xocl_axlf
19	Obtain info of bo	DRM_IOCTL_XOCL_INFO	drm_xocl_info_bo
20	Obtain bo related statistics	DRM_IOCTL_XOCL_OCL_USAGE_STAT	drm_xocl_usage_stat
21	Perform hot reset	DRM_IOCTL_XOCL_HOT_RESET	N/A
22	Perform clock scaling	DRM_IOCTL_XOCL_RECLOCK	drm_xocl_reclock_info
23	Allocate buffer on host memory	DRM_IOCTL_XOCL_ALLOC_CMA	drm_xocl_alloc_cma_info
24	Free host memory buffer	DRM_IOCTL_XOCL_FREE_CMA	N/A
25	Copy bo buffers	DRM_IOCTL_XOCL_COPY_BO	drm_xocl_copy_bo

struct drm_xocl_create_bo: Create buffer object used with DRM_IOCTL_XOCL_CREATE_BO ioctl

Definition

struct drm_xocl_create_bo {
  uint64_t size;
  uint32_t handle;
  uint32_t flags;
  uint32_t type;
};

Members

size: Requested size of the buffer object
handle: bo handle returned by the driver
flags: DRM_XOCL_BO_XXX flags
type: The type of bo

struct drm_xocl_userptr_bo: Create buffer object with user’s pointer used with DRM_IOCTL_XOCL_USERPTR_BO ioctl

Definition

struct drm_xocl_userptr_bo {
  uint64_t addr;
  uint64_t size;
  uint32_t handle;
  uint32_t flags;
  uint32_t type;
};

Members

addr: Address of buffer allocated by user
size: Requested size of the buffer object
handle: bo handle returned by the driver
flags: DRM_XOCL_BO_XXX flags
type: The type of bo

struct drm_xocl_map_kern_mem: Map a buffer object to linux kernel memory (sgl or virtual address) used with DRM_IOCTL_XOCL_MAP_KERN_MEM ioctl. WARNING: INTERNAL USE ONLY. NOT FOR PUBLIC CONSUMPTION. For use with Linux kernel space specific IOCTLs.

Definition

struct drm_xocl_map_kern_mem {
  uint32_t handle;
  uint64_t addr;
  uint64_t size;
  uint32_t flags;
};

Members

handle: bo handle returned by the driver
addr: Address of sgl or kernel buffer allocated by user
size: Requested size of the buffer object
flags: DRM_XOCL_BO_XXX flags

struct drm_xocl_map_bo: Prepare a buffer object for mmap used with DRM_IOCTL_XOCL_MAP_BO ioctl

Definition

struct drm_xocl_map_bo {
  uint32_t handle;
  uint32_t pad;
  uint64_t offset;
};

Members

handle: bo handle
pad: Unused
offset: ‘Fake’ offset returned by the driver which can be used with POSIX mmap

struct drm_xocl_sync_bo: Synchronize the buffer in the requested direction between device and host used with DRM_IOCTL_XOCL_SYNC_BO ioctl

Definition

struct drm_xocl_sync_bo {
  uint32_t handle;
  uint32_t flags;
  uint64_t size;
  uint64_t offset;
  enum drm_xocl_sync_bo_dir dir;
};

Members

handle: bo handle
flags: Unused
size: Number of bytes to synchronize
offset: Offset into the object to synchronize
dir: DRM_XOCL_SYNC_DIR_XXX

struct drm_xocl_sync_bo_cb: Synchronize the buffer in the requested direction between device and host used with DRM_IOCTL_XOCL_SYNC_BO_CB ioctl (linux kernel only) WARNING: INTERNAL USE ONLY. NOT FOR PUBLIC CONSUMPTION. For use with Linux kernel space specific IOCTLs.

Definition

struct drm_xocl_sync_bo_cb {
  uint32_t handle;
  uint32_t flags;
  uint64_t size;
  uint64_t offset;
  enum drm_xocl_sync_bo_dir dir;
  uint64_t cb_func;
  uint64_t cb_data;
};

Members

handle: bo handle
flags: Unused
size: Number of bytes to synchronize
offset: Offset into the object to synchronize
dir: DRM_XOCL_SYNC_DIR_XXX
cb_func: Pointer to callback function(void(*fn)(long,int))
cb_data: Pointer to context that callback needs to be invoked with

struct drm_xocl_info_bo: Obtain information about an allocated buffer obbject used with DRM_IOCTL_XOCL_INFO_BO IOCTL

Definition

struct drm_xocl_info_bo {
  uint32_t handle;
  uint32_t flags;
  uint64_t size;
  uint64_t paddr;
};

Members

handle: bo handle
flags: Flags passed while creating BO (out)
size: Size of buffer object (out)
paddr: Physical address (out)

struct drm_xocl_copy_bo: Device memory to memory copy bo used with DRM_IOCTL_XOCL_COPY_BO IOCTL

Definition

struct drm_xocl_copy_bo {
  uint32_t dst_handle;
  uint32_t src_handle;
  uint64_t size;
  uint64_t dst_offset;
  uint64_t src_offset;
};

Members

dst_handle: dst bo handle
src_handle: src bo handle
size: bo size in bytes
dst_offset: dst offset
src_offset: src offset

struct drm_xocl_set_cu_range: Set CU range used with DRM_IOCTL_XOCL_SET_CU_READONLY_RANGE

Definition

struct drm_xocl_set_cu_range {
  uint32_t cu_index;
  uint32_t start;
  uint32_t size;
};

Members

cu_index: Index of the compute unit in the device inage for which
start: Start offset of the range
size: Size of the range

struct argument_info: Kernel argument information

Definition

struct argument_info {
  char name;
  uint32_t offset;
  uint32_t size;
  uint32_t dir;
};

Members

name: argument name
offset: argument offset in CU
size: argument size in bytes
dir: input or output argument for a CU

struct kernel_info: Kernel information

Definition

struct kernel_info {
  char name;
  uint32_t range;
  int anums;
  int features;
  struct argument_info args;
};

Members

name: kernel name
range: kernel register range
anums: number of argument
features: number of argument
args: argument array

struct drm_xocl_kinfo_bo: Get a buffer object’s kernel virtual address used with DRM_IOCTL_XOCL_KINFO_BO ioctl. WARNING: INTERNAL USE ONLY. NOT FOR PUBLIC CONSUMPTION. For use with Linux kernel space specific IOCTLs.

Definition

struct drm_xocl_kinfo_bo {
  uint32_t handle;
  uint32_t flags;
  uint64_t size;
  uint64_t paddr;
  uint64_t vaddr;
};

Members

handle: bo handle of BO whose info is required
flags: Unused
size: Size of buffer object
paddr: Physical address (BO’s Device address)
vaddr: Kernel Virtual address of BO

struct drm_xocl_kds: KDS user configuration

Definition

struct drm_xocl_kds {
  uint32_t slot_size;
  uint32_t ert:1;
  uint32_t polling:1;
  uint32_t cu_dma:1;
  uint32_t cu_isr:1;
  uint32_t cq_int:1;
  uint32_t dataflow:1;
  uint32_t rw_shared:1;
};

Members

slot_size: CQ slot size
ert: enable embedded HW scheduler
polling: poll for command completion
cu_dma: enable CUDMA custom module for HW scheduler
cu_isr: enable CUISR custom module for HW scheduler
cq_int: enable interrupt from host to HW scheduler
dataflow: enable dataflow mode
rw_shared: allow xclRegWrite/xclRegRead access shared CU

struct drm_xocl_axlf: load xclbin (AXLF) device image used with DRM_IOCTL_XOCL_READ_AXLF ioctl

Definition

struct drm_xocl_axlf {
  struct axlf * xclbin;
  int ksize;
  char * kernels;
  struct drm_xocl_kds kds_cfg;
  uint32_t flags;
};

Members

xclbin: Pointer to user’s xclbin structure in memory
ksize: size of kernels in bytes
kernels: pointer of argument array
kds_cfg: kds configuration
flags: flags passed while programming xclbin

NOTE

This ioctl will be removed in next release

struct drm_xocl_pwrite_bo: Update bo with user’s data used with DRM_IOCTL_XOCL_PWRITE_BO ioctl

Definition

struct drm_xocl_pwrite_bo {
  uint32_t handle;
  uint32_t pad;
  uint64_t offset;
  uint64_t size;
  uint64_t data_ptr;
};

Members

handle: bo handle
pad: Unused
offset: Offset into the buffer object to write to
size: Length of data to write
data_ptr: User’s pointer to read the data from

struct drm_xocl_pread_bo: Read data from bo used with DRM_IOCTL_XOCL_PREAD_BO ioctl

Definition

struct drm_xocl_pread_bo {
  uint32_t handle;
  uint32_t pad;
  uint64_t offset;
  uint64_t size;
  uint64_t data_ptr;
};

Members

handle: bo handle
pad: Unused
offset: Offset into the buffer object to read from
size: Length of data to read
data_ptr: User’s pointer to write the data into

struct drm_xocl_ctx: Open or close a context on a compute unit on device used with DRM_XOCL_CTX ioctl

Definition

struct drm_xocl_ctx {
  enum drm_xocl_ctx_code op;
  xuid_t xclbin_id;
  uint32_t cu_index;
  uint32_t flags;
  uint32_t handle;
};

Members

op: Alloc or free a context (XOCL_CTX_OP_ALLOC_CTX/XOCL_CTX_OP_FREE_CTX)
xclbin_id: UUID of the device image (xclbin)
cu_index: Index of the compute unit in the device inage for which the request is being made
flags: Shared or exclusive context (XOCL_CTX_SHARED/XOCL_CTX_EXCLUSIVE)
handle: Unused

struct drm_xocl_create_hw_ctx: Create a hw context on a slot on device used with DRM_XOCL_CREATE_HW_CTX ioctl

Definition

struct drm_xocl_create_hw_ctx {
  struct drm_xocl_axlf * axlf_ptr;
  uint32_t qos;
  uint32_t hw_context;
};

Members

axlf_ptr: axlf pointer which need to download
qos: QOS information
hw_context: Returns Context handle

struct drm_xocl_destroy_hw_ctx: Close/Destroy a hw context on a slot on device used with DRM_XOCL_DESTROY_HW_CTX ioctl

Definition

struct drm_xocl_destroy_hw_ctx {
  uint32_t hw_context;
};

Members

hw_context: Context handle which need to close

struct drm_xocl_open_cu_ctx: Open a cu context under a hw context on device used with DRM_XOCL_OPEN_CU_CTX ioctl

Definition

struct drm_xocl_open_cu_ctx {
  uint32_t hw_context;
  char cu_name;
  uint32_t flags;
  uint32_t cu_index;
};

Members

hw_context: Open CU under this hw Context handle
cu_name: Name of the compute unit in the device image for which the open request is being made
flags: Shared or exclusive context (XOCL_CTX_SHARED/XOCL_CTX_EXCLUSIVE)
cu_index: Return the acquired CU index. This will require for close cu context

struct drm_xocl_close_cu_ctx: Open a cu context under a hw context on device used with DRM_XOCL_CLOSE_CU_CTX ioctl

Definition

struct drm_xocl_close_cu_ctx {
  uint32_t hw_context;
  uint32_t cu_index;
};

Members

hw_context: Open CU under this hw Context handle
cu_index: Index of the compute unit in the device image for which the close request is being made

struct drm_xocl_pwrite_unmgd: unprotected write to device memory used with DRM_IOCTL_XOCL_PWRITE_UNMGD ioctl

Definition

struct drm_xocl_pwrite_unmgd {
  uint32_t address_space;
  uint32_t pad;
  uint64_t paddr;
  uint64_t size;
  uint64_t data_ptr;
};

Members

address_space: Address space in the DSA; currently only 0 is suported
pad: Unused
paddr: Physical address in the specified address space
size: Length of data to write
data_ptr: User’s pointer to read the data from

struct drm_xocl_pread_unmgd: unprotected read from device memory used with DRM_IOCTL_XOCL_PREAD_UNMGD ioctl

Definition

struct drm_xocl_pread_unmgd {
  uint32_t address_space;
  uint32_t pad;
  uint64_t paddr;
  uint64_t size;
  uint64_t data_ptr;
};

Members

address_space: Address space in the DSA; currently only 0 is valid
pad: Unused
paddr: Physical address in the specified address space
size: Length of data to write
data_ptr: User’s pointer to write the data to

struct drm_xocl_usage_stat: obtain device memory usage and DMA statistics used with DRM_IOCTL_XOCL_USAGE_STAT ioctl

Definition

struct drm_xocl_usage_stat {
  unsigned dma_channel_count;
  unsigned mm_channel_count;
  uint64_t h2c;
  uint64_t c2h;
  struct drm_xocl_mm_stat mm;
};

Members

dma_channel_count: How many DMA channels are present
mm_channel_count: How many storage banks (DDR) are present
h2c: Total data transferred from host to device by a DMA channel
c2h: Total data transferred from device to host by a DMA channel
mm: BO statistics for a storage bank (DDR)

struct drm_xocl_execbuf: Submit a command buffer for execution on a compute unit used with DRM_IOCTL_XOCL_EXECBUF ioctl

Definition

struct drm_xocl_execbuf {
  uint32_t ctx_id;
  uint32_t exec_bo_handle;
  uint32_t deps;
};

Members

ctx_id: Pass 0
exec_bo_handle: BO handle of command buffer formatted as ERT command
deps: Upto MAX_DEPENT_CMD_BO dependency command BO handles this command is dependent on for automatic event dependency handling by ERT

struct drm_xocl_hw_ctx_execbuf: Submit a command buffer for execution on a compute unit used with DRM_IOCTL_XOCL_HW_CTX_EXECBUF ioctl with new hw context parameters

Definition

struct drm_xocl_hw_ctx_execbuf {
  uint32_t hw_ctx_id;
  uint32_t exec_bo_handle;
  uint32_t deps;
};

Members

hw_ctx_id: Pass the HW Context id
exec_bo_handle: BO handle of command buffer formatted as ERT command
deps: Upto MAX_DEPENT_CMD_BO dependency command BO handles this command is dependent on for automatic event dependency handling by ERT

struct drm_xocl_execbuf_cb: Submit a command buffer for execution on a compute unit used with DRM_IOCTL_XOCL_EXECBUF_CB ioctl with a callback (linux kernel only) WARNING: INTERNAL USE ONLY. NOT FOR PUBLIC CONSUMPTION. For use with Linux kernel space specific IOCTLs.

Definition

struct drm_xocl_execbuf_cb {
  uint32_t ctx_id;
  uint32_t exec_bo_handle;
  uint32_t deps;
  uint64_t cb_func;
  uint64_t cb_data;
};

Members

ctx_id: Pass 0
exec_bo_handle: BO handle of command buffer formatted as ERT command
deps: Upto 8 dependency command BO handles this command is dependent on for automatic event dependency handling by ERT
cb_func: Pointer to callback function(void (*fn)(long,int)) upon exec completion
cb_data: Pointer to context that callback needs to be invoked with

struct drm_xocl_user_intr: Register user’s eventfd for MSIX interrupt used with DRM_IOCTL_XOCL_USER_INTR ioctl

Definition

struct drm_xocl_user_intr {
  uint32_t ctx_id;
  int fd;
  int msix;
};

Members

ctx_id: Pass 0
fd: File descriptor created with eventfd system call
msix: User interrupt number (0 to 15)

struct drm_xocl_reclock_info: perform clock scaling

Definition

struct drm_xocl_reclock_info {
  unsigned region;
  unsigned short ocl_target_freq;
};

Members

region: Region
ocl_target_freq: clock scacling request array

struct drm_xocl_alloc_cma_info: Alloc buffer on host memory

Definition

struct drm_xocl_alloc_cma_info {
  uint64_t total_size;
  uint64_t entry_num;
  uint64_t * user_addr;
};

Members

total_size: total size
entry_num: number of entries
user_addr: user space address