Vitis Accelerator Flow Example: Replacing DPU with Filter2d Accelerator in a SmartCam Application

This page walks you through an example of replacing pre-processing and DPU in a SmartCam application with a simple accelerator (in this case, filter2d) in the Vitis Accelerator Flow. This particular example makes use of the Makefile environment provided with the KV260 Vitis Repository. Before trying this example, first read the Vitis Accelerator Flow.

This example was created with 2022.1 tools and BSP and is based on Ubuntu. You need Vitis. This application assumes that you are familiar with the Smartcam application and have already booted Linux with a Smartcam application running. If you have not yet done that, go through Smartcam Application Deployment.

This example generates two files: .bit.bin (PL bitstream) and .xclbin file for the specific accelerator. You reuse the .dtbo (device tree overlay), and shell.json file from the Smartcam platform because you are using the same platform.

Obtaining Platform

Because you are altering a Smartcam application only and not the platform, you first need to get the platform Smartcam is based on, kv260_ispMipiRx_vcu_DP. A detailed tutorial is at Creating Vitis Platform. Use the following commands to generate the platform:

git clone --branch xlnx_rel_v2022.1 --recursive https://github.com/Xilinx/kria-vitis-platforms.git
cd $workdir/kria-vitis-platforms/kv260/platforms
source <vitis path>/settings64.sh
make platform PFM=kv260_ispMipiRx_vcu_DP

The platform is now available in kria-vitis-platforms/kv260/platforms/xilinx_kv260_ispMipiRx_vcu_DP_202110_1/.

Obtaining the filter2d Kernel

The filter2d RTL kernel is found here. The repository targets VCK190, a board containing a Versal device, but because it is a soft IP, it also works for the Zynq MPSoC. In a separate folder, download and extract filter2d_pl, and copy it to the smartcam folder:

cd $workdir
git clone --branch 2022.1 https://github.com/Xilinx/vck190-base-trd
cd vck190-base-trd
cp overlays/filter2d/kernels/filter2d_pl/ ../kria-vitis-platforms/kv260/overlays/examples/smartcam/ -r

In $workdir/kria-vitis-platforms/kv260/overlays/examples/smartcam//overlays/filter2d/kernels/filter2d_pl/Makefile, update the LINK_CFG with a zynqmp.cfg instead of a vck190.cfg:

LINK_CFG ?= zynqmp.cfg

The zynqmp.cfg is placed at the same level as the Makefile and vck190.cfg and has the following content:

[advanced]
param=compiler.addOutputTypes=hw_export
[vivado]
prop=run.impl_1.strategy=Performance_ExploreWithRemap

This helps meeting timing on the Zynq MPSoC instead of the Versal based VCK190.

Altering Application/Overlay

The next step is to alter the SmartCam overlay. This is the overlay folder:

cd $workdir/kria-vitis-platforms/kv260/overlays/examples/smartcam/

A Makefile now displays in that level. Open the Makefile, remove DPU, and add filter2d. You first need to remove the generation of the DPU and its preprocessing. This means removing the packing of the .xo files for those two RTL kernels.

1. Remove the following lines for the DPU (in three separate places):

   dpu_HDLSRCS=kernel_xml/dpu/kernel.xml\
   		scripts/package_dpu_kernel.tcl\
   		scripts/gen_dpu_xo.tcl\
   		../../dpu_ip/Vitis/dpu/hdl/DPUCZDX8G.v\
   		../../dpu_ip/Vitis/dpu/inc/arch_def.vh\
   		../../dpu_ip/Vitis/dpu/xdc/*.xdc\
   		../../dpu_ip/DPUCZDX8G_*/hdl/DPUCZDX8G_*_dpu.sv\
   		../../dpu_ip/DPUCZDX8G_*/inc/function.vh\
   			../../dpu_ip/DPUCZDX8G_*/inc/arch_para.vh
   
   dpu_TCL=scripts/gen_dpu_xo.tcl
   DPU_KERN_NAME = DPUCZDX8G
   dpu_xo = binary_container_1/dpu.xo
   
   
   binary_container_1/dpu.xo: $(dpu_HDLSRCS)
   	@mkdir -p $(@D)
   	-@$(RM) $@
   	$(VIVADO) -mode batch -source $(dpu_TCL) -tclargs $@ $(DPU_KERN_NAME) ${TARGET} mpsoc
   
   
   	cp ./binary_*/link/vivado/vpl/prj/prj.gen/sources_1/bd/*/ip/*_DPUCZDX8G_1_0/arch.json ./binary_*/sd_card
   

2. Remove the following lines for preprocessing (PP):

   KERNEL = pp_pipeline_accel
   KERNEL_XO = $(KERNEL)/$(KERNEL).xo
   
   KERNEL_XO_FLAGS = -k $(KERNEL) -I$(XFOPENCV_INCDIR) -D__SDSVHLS__ -DHLS_NO_XIL_FPO_LIB --advanced.prop kernel.pp_pipeline_accel.kernel_flags="-std=c++0x"
   
   $(KERNEL_XO): xf_pp_pipeline_accel.cpp
   	@mkdir -p $(@D)
   	-@$(RM) $@
   	$(VPP) $(VPP_XO_FLAGS) -o $@ $<
   	-@$(RM) .Xil
   

3. Add the filter2d .xo file generation and Makefile calls in three different places:

   filter2d_xo = filter2d_pl/kernel/filter2d_pl_accel/filter2d_pl_accel.xo
   
   
   $(BINARY_CONTAINER): $(filter2d_xo)
   	$(VPP) $(VPP_LINK_FLAGS) -o $@ $(+)
   	$(XBU) $(XBU_FLAGS) --input $(BINARY_CONTAINER) --output tmp.xclbin
   	-@$(MV) tmp.xclbin $(BINARY_CONTAINER)
   	-@$(RM) .Xil
   
   $(filter2d_xo):
   	$(MAKE) -C filter2d_pl/kernel
   
   
   	$(MAKE) -C filter2d_pl/kernel clean
   

4. Remove the invocation for strip_interconnects.tcl:

   --xp param:compiler.userPostSysLinkOverlayTcl=${DIR_PRJ}/prj_conf/strip_interconnects.tcl 
   

5. Replace the two kernels with the filter2d kernel:

   • Remove dpu_pp_xo = $(KERNEL_XO) $(dpu_xo)

   • Add dd dpu_pp_xo =  $(filter2d_xo)

   The resulting Makefile is found in example Makefile. You can compare it with the original Makefile here.

6. Next, connect the filter2d accelerator to the processor subsystem. Open prj_conf/prj_config_1dpu, remove DPUCZDX8G, pp_pipeline_accel_1 clocks and connectivity, and connect filter2d to HP1, and set the clock to 200 MHz:

   [clock]
   #freqHz=300000000:DPUCZDX8G_1.aclk
   #freqHz=600000000:DPUCZDX8G_1.ap_clk_2
   #freqHz=300000000:pp_pipeline_accel_1.ap_clk
   
   freqHz=200000000:filter2d_pl_accel_0.ap_clk
   
   [connectivity]
   #sp=DPUCZDX8G_1.M_AXI_GP0:HP1
   #sp=DPUCZDX8G_1.M_AXI_HP0:HP1
   #sp=DPUCZDX8G_1.M_AXI_HP2:HPC1
   #sp=pp_pipeline_accel_1.m_axi_gmem1:HP3
   #sp=pp_pipeline_accel_1.m_axi_gmem2:HP3
   #sp=pp_pipeline_accel_1.m_axi_gmem3:HP3
   #sp=pp_pipeline_accel_1.m_axi_gmem4:HP3
   
   nk=filter2d_pl_accel:1:filter2d_pl_accel_0
   sp=filter2d_pl_accel_0.frm_in:HP1
   sp=filter2d_pl_accel_0.frm_out:HP1
   sp=filter2d_pl_accel_0.kernel:HP1
   

You have finished all the modifications needed to replace the preprocessor and DPU with filter2d.

Generate the Bitstream and .xclbin

1. Now, call the top level Makefile to generate the required files:

   cd ../../../
   make overlay OVERLAY=smartcam
   

   The generated Vivado project is located at:

   $working_dir/kria-vitis-platforms/kv260/overlays/examples/smartcam/binary_container_1/link/vivado/vpl/prj
   

   You can open up the Vivado project to see how filter2d is connected into the system.

   The generated bitfile and xclbin will be located at:

   $working_dir/kria-vitis-platforms/kv260/overlays/examples/smartcam/binary_container_1/link/int/system.bit   
   $working_dir/kria-vitis-platforms/kv260/overlays/examples/smartcam/binary_container_1/*.xclbin
   

2. Rename the .xclbin file to filter2d.xclbin

3. Now you need to convert system.bit to a .bit.bin file:

   cd overlays/examples/smartcam/binary_container_1/link/int/
   echo 'all:{system.bit}'>bootgen.bif
   bootgen -w -arch zynqmp -process_bitstream bin -image bootgen.bif
   mv system.bit.bin filter2d.bit.bin
   

Now you have the filter2d.xclbin and filter2d.bit.bin files.

Optional Steps to Generate shell.json and .dtbo file

Because you are reusing the same platform the smartcam application is based on, reuse the smartcam shell.json and kv260-smartcam.dtbo file in the firmware folder as indicated in the following section. However, if you prefer, you can also generate them separatedly.

NOTE: You still need to download and install smartcam for the software dependencies this example uses.

The shell.json is generated using the instructions here.

The filter2d.dtbo file is generated by using the same .dtsi file and compile using dtc:

dtc -@ -O dtb -o filter2d.dtbo kv260-smartcam.dtsi

Compile the Software on Ubuntu Target

1. Next, boot Ubuntu, and compile the software on target. Download the kria-developer container because that contains the environment to compile your code:

   docker pull xilinx/kria-developer:latest
   

2. Then load docker container:

   docker run \
   --env="DISPLAY" \
   -h "xlnx-docker" \
   --env="XDG_SESSION_TYPE" \
   --net=host \
   --privileged \
   --volume="$HOME/.Xauthority:/root/.Xauthority:rw" \
   -v /tmp:/tmp \
   -v /dev:/dev \
   -v /sys:/sys \
   -v /etc/vart.conf:/etc/vart.conf \
   -v /lib/firmware/xilinx:/lib/firmware/xilinx \
   -v /run:/run \
   -it xilinx/kria-developer:latest bash
   

3. In the docker container, change directory to /tmp, clone the git repo, and build the filder2d application. Use /tmp because that folder is accessible inside and outside containers. However, the files there might not persist through reboot.

   cd /tmp/
   git clone --branch 2022.1 https://github.com/Xilinx/vck190-base-trd.git
   cd vck190-base-trd/overlays/filter2d/apps/filter2d-notebooks/filter2d-notebooks
   mkdir build;
   cd build;
   cmake .. && make -j &&make install
   

4. This produces the two files you require. Copy the files to the /tmp/ folder:

   cp /usr/local/lib/vvas/libvvas_xfilter2d_pl.so /tmp/
   cp /usr/local/share/vvas/base-trd/kernel_xfilter2d_pl.json /tmp/
   

Run Application on Target

This application assumes that you are familiar with the SmartCam application and know how to boot Linux with a SmartCam application running. If you have not yet done that, go through SmartCam Application Deployment. The following commands in target (KV260) download SmartCam and their dependencies:

docker pull xilinx/smartcam:2022.1

With the SmartCam application downloaded, you have most of the software dependencies needed to run gstreamer, as well as the .dtbo file and shell.json file.

NOTE: Because you reused the same platform from SmartCam, you can reuse the .dtbo file from SmartCam.

The filter2d accelerator included only takes YuY2 format, while the MIPI sensors are in NV12 format. Therefore, you need to find a compatible video to try the filter2d with. In this case, you can use the bbb_sunflower_1080p_30fps_normal.mp4 from here or here, placing them in /tmp/bbb_sunflower_1080p_30fps_normal.mp4. The /tmp folder is accessible by Ubuntu or by containers.

1. Transfer filter2d.xclbin and filter2d.bit.bin to the target KV260 using SCP or your chosen method. On target, make a new filter2d directory and copy the required files into the directory:

   sudo mkdir /lib/firmware/xilinx/filter2d
   // sudo cp /lib/firmware/xilinx/kv260-smartcam/kv260-smartcam.dtbo /lib/firmware/xilinx/filter2d/filter2d.dtbo
   sudo cp /lib/firmware/xilinx/kv260-smartcam/shell.json /lib/firmware/xilinx/filter2d/
   sudo cp filter2d.dtbo /lib/firmware/xilinx/filter2d/filter2d.dtbo
   sudo cp filter2d.bit.bin /lib/firmware/xilinx/filter2d/
   sudo cp filter2d.xclbin /lib/firmware/xilinx/filter2d/
   

2. Modify /tmp/kernel_xfilter2d_pl.json so that xclbin-location points to the location, ```/lib/firmware/xilinx/filter2d/filter2d.xclbin``.

3. Now load the new application with the filter2d firmware and smartcam container:

   sudo xmutil desktop_disable
   sudo xmutil unloadapp
   sudo xmutil loadapp filter2d
   
   docker run \
   --env="DISPLAY" \
   -h "xlnx-docker" \
   --env="XDG_SESSION_TYPE" \
   --net=host \
   --privileged \
   --volume="$HOME/.Xauthority:/root/.Xauthority:rw" \
   -v /tmp:/tmp \
   -v /dev:/dev \
   -v /sys:/sys \
   -v /etc/vart.conf:/etc/vart.conf \
   -v /lib/firmware/xilinx:/lib/firmware/xilinx \
   -v /run:/run \
   -it xilinx/smartcam:2022.1 bash
   

4. First, try running gstreamer with the accelerator bypassed to make sure things work correctly. Expect to see the monitor displaying what it captured in MIPI. This command assumes that you use a MIPI camera mapped to media0, and that the monitor supports 1920x1080p:

   gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=mmap ! "video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1" ! perf ! kmssink plane-id=39 fullscreen-overlay=true -v
   

   This confirms that the monitor is working as expected.

5. Next, try running using filter2d in the PL. Copy the software to the following folders:

   mkdir /usr/lib/vvas/
   cp /tmp/libvvas_xfilter2d_pl.so /usr/lib/vvas/
   
   gst-launch-1.0 filesrc location="/tmp/bbb_sunflower_1080p_30fps_normal.mp4" ! decodebin ! videoconvert ! video/x-raw, width=1920, height=1080, format=YUY2, framerate=30/1 ! vvas_xfilter kernels-config=/tmp/kernel_xfilter2d_pl.json dynamic-config='{ "filter_preset" : "edge" }' ! perf ! kmssink driver-name=xlnx plane-id=39 fullscreen-overlay=true sync=false -v
   

You should see the video being displayed with the filter_preset set to “edge”. You can try other presets as well. The list can be found here. If there are spaces in the preset, replace them with an underscore. For example, in the above command, horizontal edge would be horizontal_edge. The framerate being displayed is be slow; you have to do format convertation in software, so each frame takes time.

---

_{Copyright © 2023-2025 Advanced Micro Devices, Inc.}

^{Terms and Conditions}