ZOCL Driver Interfaces¶

A GEM style driver for Xilinx edge based accelerators

File zynq_ioctl.h defines ioctl command codes and associated structures for interacting with zocl driver for Xilinx FPGA platforms(Zynq/ZynqMP/Versal).

Accelerator memory allocation is modeled as buffer objects (bo). zocl supports both SMMU based shared virtual memory and CMA based shared physical memory between PS and PL. zocl also supports memory management of PL-DDRs and PL-BRAMs. PL-DDR is reserved by zocl driver via device tree. Both PS Linux and PL logic can access PL-DDRs

Execution model is asynchronous where execute commands are submitted using command buffers and POSIX poll is used to wait for finished commands. Commands for a compute unit can only be submitted after an explicit context has been opened by the client for that compute unit.

zocl driver functionality is described in the following table.

#	Functionality	ioctl request code	data format
1	Allocate buffer on device	DRM_IOCTL_ZOCL_CREATE_BO	drm_zocl_create_bo
2	Allocate buffer on device with userptr	DRM_IOCTL_ZOCL_USERPTR_BO	drm_zocl_userptr_bo
3	Get the buffer handle of given physical address	DRM_IOCTL_ZOCL_GET_HOST_BO	drm_zocl_host_bo
4	Prepare bo for mapping into user’s address space	DRM_IOCTL_ZOCL_MAP_BO	drm_zocl_map_bo
5	Synchronize (DMA) buffer contents in requested direction	DRM_IOCTL_ZOCL_SYNC_BO	drm_zocl_sync_bo
6	Obtain information about buffer object	DRM_IOCTL_ZOCL_INFO_BO	drm_zocl_info_bo
7	Update bo backing storage with user’s data	DRM_IOCTL_ZOCL_PWRITE_BO	drm_zocl_pwrite_bo
8	Read back data in bo backing storage	DRM_IOCTL_ZOCL_PREAD_BO	drm_zocl_pread_bo
9	Update device view with a specific xclbin image	DRM_IOCTL_ZOCL_PCAP_DOWNLOAD	drm_zocl_pcap_download
10	Read the xclbin and map the compute units.	DRM_IOCTL_ZOCL_READ_AXLF	drm_zocl_axlf
11	Send an execute job to a compute unit	DRM_IOCTL_ZOCL_EXECBUF	drm_zocl_execbuf
12	Get the soft kernel command (experimental)	DRM_IOCTL_ZOCL_SK_GETCMD	drm_zocl_sk_getcmd
13	Create the soft kernel (experimental)	DRM_IOCTL_ZOCL_SK_CREATE	drm_zocl_sk_create
14	Report the soft kernel state (experimental)	DRM_IOCTL_ZOCL_SK_REPORT	drm_zocl_sk_report
15	Get Information about Compute Unit (experimental)	DRM_IOCTL_ZOCL_INFO_CU	drm_zocl_info_cu
16	Create a hw context on a slot for a xclbin on the device	DRM_IOCTL_ZOCL_CREATE_HW_CTX	drm_zocl_create_hw_ctx
17	Destroy a hw context on a slot on a device	DRM_IOCTL_ZOCL_DESTROY_HW_CTX	drm_zocl_destroy_hw_ctx
18	Open cu context	DRM_IOCTL_ZOCL_OPEN_CU_CTX	drm_zocl_open_cu_ctx
19	Close cu context	DRM_IOCTL_ZOCL_CLOSE_CU_CTX	drm_zocl_close_cu_ctx
20	Send an execute job to a CU with hw context	DRM_IOCTL_ZOCL_HW_CTX_EXECBUF	drm_zocl_hw_ctx_execbuf
21	Open graph context	DRM_IOCTL_ZOCL_OPEN_GRAPH_CTX	drm_zocl_open_graph_ctx
22	Close graph context	DRM_IOCTL_ZOCL_CLOSE_GRAPH_CTX	drm_zocl_close_graph_ctx

struct drm_zocl_create_bo: Create buffer object used with DRM_IOCTL_ZOCL_CREATE_BO ioctl

Definition

struct drm_zocl_create_bo {
  uint64_t size;
  uint32_t handle;
  uint32_t flags;
};

Members

size: Requested size of the buffer object
handle: bo handle returned by the driver
flags: DRM_ZOCL_BO_XXX flags

struct drm_zocl_userptr_bo: Create buffer object with user’s pointer used with DRM_IOCTL_ZOCL_USERPTR_BO ioctl

Definition

struct drm_zocl_userptr_bo {
  uint64_t addr;
  uint64_t size;
  uint32_t handle;
  uint32_t flags;
};

Members

addr: Address of buffer allocated by user
size: Requested size of the buffer object
handle: bo handle returned by the driver
flags: DRM_XOCL_BO_XXX flags

struct drm_zocl_map_bo: Prepare a buffer object for mmap used with DRM_IOCTL_ZOCL_MAP_BO ioctl

Definition

struct drm_zocl_map_bo {
  uint32_t handle;
  uint32_t pad;
  uint64_t offset;
};

Members

handle: bo handle
pad: Unused
offset: ‘Fake’ offset returned by the driver which can be used with POSIX mmap

struct drm_zocl_sync_bo: Synchronize the buffer in the requested direction via cache flush/invalidation. used with DRM_ZOCL_SYNC_BO ioctl.

Definition

struct drm_zocl_sync_bo {
  uint32_t handle;
  enum drm_zocl_sync_bo_dir dir;
  uint64_t offset;
  uint64_t size;
};

Members

handle: GEM object handle
dir: DRM_ZOCL_SYNC_DIR_XXX
offset: Offset into the object to write to
size: Length of data to write

struct drm_zocl_info_bo: Obtain information about buffer object used with DRM_IOCTL_ZOCL_INFO_BO ioctl

Definition

struct drm_zocl_info_bo {
  uint32_t handle;
  uint32_t flags;
  uint64_t size;
  uint64_t paddr;
};

Members

handle: GEM object handle
flags: User BO flags
size: Size of BO
paddr: physical address

struct drm_zocl_host_bo: Get the buffer handle of given physical address used with DRM_IOCTL_ZOCL_GET_HOST_BO ioctl

Definition

struct drm_zocl_host_bo {
  uint64_t paddr;
  size_t size;
  uint32_t handle;
};

Members

paddr: physical address
size: Size of BO
handle: GEM object handle

struct drm_zocl_set_cu_range: Set CU range used with DRM_IOCTL_ZOCL_SET_CU_READONLY_RANGE

Definition

struct drm_zocl_set_cu_range {
  uint32_t cu_index;
  uint32_t start;
  uint32_t size;
};

Members

cu_index: Index of the compute unit
start: Start offset of the range
size: Size of the range

struct drm_zocl_pwrite_bo: Update bo with user’s data used with DRM_IOCTL_ZOCL_PWRITE_BO ioctl

Definition

struct drm_zocl_pwrite_bo {
  uint32_t handle;
  uint32_t pad;
  uint64_t offset;
  uint64_t size;
  uint64_t data_ptr;
};

Members

handle: GEM object handle
pad: Padding
offset: Offset into the object to write to
size: Length of data to write
data_ptr: Pointer to read the data from (pointers not 32/64 compatible)

struct drm_zocl_pread_bo: Read data from bo used with DRM_IOCTL_ZOCL_PREAD_BO ioctl

Definition

struct drm_zocl_pread_bo {
  uint32_t handle;
  uint32_t pad;
  uint64_t offset;
  uint64_t size;
  uint64_t data_ptr;
};

Members

handle: GEM object handle
pad: Padding
offset: Offset into the object to read from
size: Length of data to wrreadite
data_ptr: Pointer to write the data into (pointers not 32/64 compatible)

struct drm_zocl_info_cu: Get information about Compute Unit (experimental) used with DRM_IOCTL_ZOCL_INFO_CU ioctl

Definition

struct drm_zocl_info_cu {
  uint64_t paddr;
  int apt_idx;
  int cu_idx;
};

Members

paddr: Physical address
apt_idx: Aperture index
cu_idx: CU index

struct drm_zocl_execbuf: Submit a command buffer for execution on a compute unit (experimental) used with DRM_IOCTL_ZOCL_EXECBUF ioctl

Definition

struct drm_zocl_execbuf {
  uint32_t ctx_id;
  uint32_t exec_bo_handle;
};

Members

ctx_id: Pass 0
exec_bo_handle: BO handle of command buffer formatted as ERT command

struct drm_zocl_hw_ctx_execbuf: Submit a command buffer for execution on a CU used with DRM_IOCTL_ZOCL_HW_CTX_EXECBUF ioctl

Definition

struct drm_zocl_hw_ctx_execbuf {
  uint32_t hw_ctx_id;
  uint32_t exec_bo_handle;
};

Members

hw_ctx_id: pass the hw context id
exec_bo_handle: BO handle of command buffer formatted as ERT command

struct argument_info: Kernel argument information

Definition

struct argument_info {
  char name;
  uint32_t offset;
  uint32_t size;
  uint32_t dir;
};

Members

name: argument name
offset: argument offset in CU
size: argument size in bytes
dir: input or output argument for a CU

struct kernel_info: Kernel information

Definition

struct kernel_info {
  char name;
  uint32_t range;
  int anums;
  struct argument_info args;
};

Members

name: kernel name
range: kernel range
anums: number of argument
args: argument array

struct drm_zocl_kds: KDS user configuration

Definition

struct drm_zocl_kds {
  uint32_t polling:1;
};

Members

polling: poll for command completion

struct drm_zocl_axlf: Read xclbin (AXLF) device image and map CUs (experimental) used with DRM_IOCTL_ZOCL_READ_AXLF ioctl

Definition

struct drm_zocl_axlf {
  struct axlf * za_xclbin_ptr;
  uint32_t za_flags;
  int za_ksize;
  char * za_kernels;
  uint32_t za_slot_id;
  uint8_t hw_gen;
  uint32_t partition_id;
};

Members

za_xclbin_ptr: Pointer to xclbin (AXLF) object
za_flags: platform flags
za_ksize: size of kernels in bytes
za_kernels: pointer of argument array
za_slot_id: xclbin slot
hw_gen: aie generation
partition_id: aie partition id

struct drm_zocl_create_hw_ctx: Create a hw context on a slot on device used with DRM_IOCTL_ZOCL_CREATE_HW_CTX ioctl

Definition

struct drm_zocl_create_hw_ctx {
  struct drm_zocl_axlf * axlf_ptr;
  uint32_t qos;
  uint32_t hw_context;
};

Members

axlf_ptr: axlf pointer which need to be downloaded
qos: QOS information
hw_context: Returns context handle

struct drm_zocl_destroy_hw_ctx: Destroy a hw context on a slot on device used with DRM_IOCTL_ZOCL_DESTROY_HW_CTX ioctl

Definition

struct drm_zocl_destroy_hw_ctx {
  uint32_t hw_context;
};

Members

hw_context: Context handle that needs to be closed

struct drm_zocl_open_cu_ctx: Opens a cu context under a hw context on the device used with DRM_IOCTL_ZOCL_OPEN_CU_CTX

Definition

struct drm_zocl_open_cu_ctx {
  uint32_t hw_context;
  char cu_name;
  uint32_t flags;
  uint32_t cu_index;
};

Members

hw_context: Open a cu context under this hw context handle
cu_name: Name of the cu on the device image for which the open context is being made
flags: Shared or Exclusive context (ZOCL_CTX_SHARED/ZOCL_CTX_EXCLUSIVE)
cu_index: Reture the acquired cu index. This will be required for closing

struct drm_zocl_close_cu_ctx: Closes a cu context opened under a hw context on device used with DRM_IOCTL_ZOCL_CLOSE_CU_CTX

Definition

struct drm_zocl_close_cu_ctx {
  uint32_t hw_context;
  uint32_t cu_index;
};

Members

hw_context: close cu context under this hw context handle
cu_index: Index of the cu on the device image for which the close request is being made

struct drm_zocl_open_graph_ctx: Opens a graph context under a hw context on the device used with DRM_IOCTL_ZOCL_OPEN_GRAPH_CTX

Definition

struct drm_zocl_open_graph_ctx {
  uint32_t hw_context;
  uint32_t flags;
  uint32_t graph_id;
};

Members

hw_context: Open a graph context under this hw context handle
flags: Shared or Exclusive context (ZOCL_CTX_SHARED/ZOCL_CTX_EXCLUSIVE)
graph_id: graph id

struct drm_zocl_close_graph_ctx: Closes a graph context opened under a hw context on device used with DRM_IOCTL_ZOCL_CLOSE_GRAPH_CTX

Definition

struct drm_zocl_close_graph_ctx {
  uint32_t hw_context;
};

Members

hw_context: close graph context under this hw context handle

struct drm_zocl_sk_getcmd: Get the soft kernel command (experimental) used with DRM_IOCTL_ZOCL_SK_GETCMD ioctl

Definition

struct drm_zocl_sk_getcmd {
  uint32_t opcode;
  uint32_t start_cuidx;
  uint32_t cu_nums;
  char name;
  int bohdl;
  int meta_bohdl;
  unsigned char uuid;
};

Members

opcode: opcode for the Soft Kernel Command Packet
start_cuidx: start index of compute units
cu_nums: number of compute units in program
name: symbol name of soft kernel
bohdl: BO to hold soft kernel image
meta_bohdl: BO to hold metadata
uuid: UUID for the xclbin

struct drm_zocl_aie_cmd: Get the aie command used with DRM_IOCTL_ZOCL_AIE_GETCMD and DRM_IOCTL_ZOCL_AIE_PUTCMD ioctl

Definition

struct drm_zocl_aie_cmd {
  uint32_t opcode;
  uint32_t size;
  char info;
};

Members

opcode: opcode for the Aie Command Packet
size: size in bytes of info data
info: information to transfer

struct drm_zocl_sk_create: Create a soft kernel (experimental) used with DRM_IOCTL_ZOCL_SK_CREATE ioctl

Definition

struct drm_zocl_sk_create {
  uint32_t cu_idx;
  int handle;
};

Members

cu_idx: Compute unit index
handle: Buffer object handle

struct drm_zocl_sk_report: Report the Soft Kernel State (experimental) used with DRM_IOCTL_ZOCL_SK_REPORT ioctl

Definition

struct drm_zocl_sk_report {
  uint32_t cu_idx;
  enum drm_zocl_scu_state cu_state;
};

Members

cu_idx: Compute unit index
cu_state: State of the Soft Compute Unit