Kria™ KR260 Robotics Starter Kit

Machine Vision Camera Tutorial

Software Architecture of the Accelerator¶

Introduction¶

This document describes the software architecture of the MV-Defect-Detect accelerator application.

Software Architecture¶

In the MV-Defect-Detect application, the Computer Vision Starter Kit can take video inputs from a live or a file source. As shown in the following figure, the video inputs are in the Luma(Y) format, processed and displayed. AMD Vitis™ overlay includes Vitis Vision libraries that process the frames and detect defects in mangoes.

In this reference design, the resolution on the input frames is 1920 x 1080, and the outputs are 1920x1080 on a 1080p display.

../../../_images/defect-detection-process.png

The parts before pre-process plugin and after mv-defect-detect accelerator for data source and sink respectively, use purely official GStreamer plugins, such as filesrc for file input, v4l2src for capture, and Kmssink for the display. Refer to the GStreamer documentation for detailed usage.

The core acceleration tasks are performed by the Pre-Process and CCA libraries, which AMD developed.

The following table lists the GStreamer plugins used in the application.

GStreamer Plugins	Definition	Note
v4l2src	Image capturing from the live camera source	V4l2 source
Kmssink	For the display	Upstream GStreamer
Queue	Simple data queue	Upstream GStreamer
VVAS xfilter	Kernel Library: gaussian_otsu. Vitis Vision library for the Gaussian + OTSU detector. Preserves edges while smoothening and calculates the optimum threshold between foreground and background pixels.	Xilinx Opensource Plugin
VVAS xfilter	Kernel Library: pre-process. Vitis Vision library to convert a grey-scale image to a binary image and filter out noise from the image.	Xilinx Opensource Plugin
VVAS xfilter	Kernel Library: cca_accelerator. Vitis Vision library to determine the defective pixels in the image.	Xilinx Opensource Plugin
VVAS xfilter	Kernel Library: text2overlay. OpenCV software library to calculate the defect density, determine the quality of the mango, and embed text as result into output images.	Xilinx Opensource Plugin
____

The following table lists the Component types used in the application.

Component	Component Type
Gaussian + OTSU Accelerator	PL
Threshold	PL
Defect Decision	CCA
Text2Overlay + Defect Decision	SW

v4l2src¶

Click here to view details

v4l2src is an open source plugin. The underlying GStreamer plugin uses the IMX547 sensor and the AP1302 ISP. The data flow is as follows:

../../../_images/v4l2src-data-flow.png

In-House Plugins¶

The following are the in-house plugins:

Pre-process¶

Click here to view details

The pre-process pipeline is as follows:

../../../_images/preprocessing_pipeline.png

The following figure depicts the Pre-Process plugin data flow.

../../../_images/preprocess-plugin-data-flow.png

Gaussian_OTSU Accelerator

This accelerator has two kernels - Gaussian & OTSU, stitched in a streaming fashion. In general, any smoothing filter smoothens the image, and affects the edges of the image. To preserve the edges while smoothing, use bilateral filter. In an analogous way as the Gaussian filter, the bilateral filter also considers the neighboring pixels with weights assigned to each of them.

These weights have two components, the first of which is the same weighing used by the Gaussian filter. The second component takes into account the difference in the intensity between the neighbouring pixels and the evaluated one.

OTSU threshold is used to automatically perform clustering-based image thresholding or the reduction of a gray-level image to a binary image. The algorithm assumes that the image contains two classes of pixels following bi-modal histogram (foreground pixels and background pixels), it then calculates the optimum threshold separating the two classes.

The following figure depicts the Gaussian + OSTSU plugin software stack.

../../../_images/gaussian-plugin-sw-stack.png

The following figure depicts the Gaussian + OTSU plugin data flow.

../../../_images/gaussian-plugin-dataflow.png

Threshold_Median Accelerator

The grey-scale image should be converted into a binary image with an appropriate threshold value. The threshold function in the Vitis Vision library can perform the thresholding operation on the input image. This should yield an image that has a black background with the mango area in white.

The median blur filter acts as a non-linear digital filter that reduces noise. A filter size of N outputs the median of the NxN neighborhood pixel values, for each pixel. In this design, N is 3.

This plugin accepts the 1920x1080 Y8 image as input. The plugin applies the threshold binary algorithm to convert the Y8 image to binary image by using the threshold value of the pixel. Later, it applies the Median filter to remove salt and pepper noise.

The following figure depicts the Threshold + Median plugin software stack.

../../../_images/threshold-plugin-sw-stack.png

The following figure depicts the Threshold + Median plugin data flow.

../../../_images/threshold-median-plugin-dataflow.png

Threshold and Median Blur kernels are connected together using AXI Stream interface.

CCA¶

Click here to view details

The implemented Connected Component Analysis (CCA), is a custom solution to find the defective pixels in the problem object. This algorithm considers few assumptions that the background must be easily separable from the foreground object.

The custom CCA effectively analyses the components that are connected to the background pixels and removes the background from the object and defective pixels. The aim is to send the following output information from the function:

defect image: image with only defect pixels marked as ‘255’ and both object pixels and background as ‘0’
object_pixels: total non-defective pixels of the object
defect_pixels: total defective pixels

The following figure depicts the CCA plugin software stack.

../../../_images/cca-plugin-sw-stack.png

The following figure depicts the CCA plugin data flow.

../../../_images/cca-plugin-dataflow.png

Defect Decision¶

Click here to view details

The output of the CCA plugin is fed into the Defect Decision block, which determines the defect density and decides the quality of the mango. The block performs the following main operations:

The ratio of blemished pixels to total mango pixels is calculated to determine how much of the mango’s surface area is covered with blemishes.
Defect Decision determines whether the ratio exceeds a user-defined threshold, to decide whether the mango is defected or not.
The results are embedded in the image and output is fed to the next plugin for the display.

The following figure depicts the Defect Decision plugin software stack.

../../../_images/defect-decision-plugin-sw-stack.png

The following figure depicts the Defect Decision plugin data flow.

../../../_images/defect-decision-plugin-dataflow.png

Configuration Files¶

Click here to view details

The mv-defect-detect application uses the following configuration files.

Gaussian_OTSU Accelerator

The otsu-accelerator.json file is as follows:

{
  "xclbin-location": "/lib/firmware/xilinx/kr260-mv-camera/kr260-mv-camera.xclbin",
  "vvas-library-repo": "/opt/xilinx/xlnx-app-kr260-mv-defect-detect/lib",
  "element-mode": "transform",
  "kernels": [
    {
      "kernel-name": "gaussian_otsu_accel:{gaussian_otsu_accel_1}",
      "library-name": "libvvas_otsu.so",
      "config": {
        "debug_level" : 1
      }
    }
  ]
}

* debug_level: Enable or disable debug log for the Kernel library.

Preprocess Accelerator

The preprocess-accelerator.json file is as follows:

{
  "xclbin-location": "/lib/firmware/xilinx/kr260-mv-camera/kr260-mv-camera.xclbin",
  "vvas-library-repo": "/opt/xilinx/xlnx-app-kr260-mv-defect-detect/lib",
  "element-mode": "transform",
  "kernels": [
    {
      "kernel-name": "preprocess_accel:{preprocess_accel_1}",
      "library-name": "libvvas_preprocess.so",
      "config": {
        "debug_level" : 1,
        "max_value": 255
      }
    }
  ]
}

* debug_level: Enable or disable debug log for the Kernel library.
* max_value: Maximum value to use with the THRESH_BINARY thresholding types.
    For more information, click [here](https://docs.opencv.org/master/d7/d1b/group__imgproc__misc.html#gae8a4a146d1ca78c626a53577199e9c57).

Preprocess Accelerator stride

The preprocess-accelerator-stride.json file is as follows:

{
  "xclbin-location": "/lib/firmware/xilinx/kr260-mv-camera/kr260-mv-camera.xclbin",
  "vvas-library-repo": "/opt/xilinx/xlnx-app-kr260-mv-defect-detect/lib",
  "element-mode": "transform",
  "kernels": [
    {
      "kernel-name": "preprocess_accel:{preprocess_accel_1}",
      "library-name": "libvvas_preprocess.so",
      "config": {
        "debug_level" : 1,
        "max_value": 255,
        "stride_value": 2048
      }
    }
  ]
}

CCA Accelerator

The cca-accelarator.json file is as follows:

{
  "xclbin-location": "/lib/firmware/xilinx/kr260-mv-camera/kr260-mv-camera.xclbin",
  "vvas-library-repo": "/opt/xilinx/xlnx-app-kr260-mv-defect-detect/lib",
  "element-mode": "transform",
  "kernels": [
    {
      "kernel-name": "cca_custom_accel:{cca_custom_accel_1}",
      "library-name": "libvvas_cca.so",
      "config": {
        "debug_level" : 1
      }
    }
  ]
}

* debug_level: Enable or disable debug log for the Kernel library.

Text2Overlay

The text2overlay.json file is as follows:

{
  "xclbin-location": "/lib/firmware/xilinx/kr260-mv-camera/kr260-mv-camera.xclbin",
  "vvas-library-repo": "/opt/xilinx/xlnx-app-kr260-mv-defect-detect/lib",
  "element-mode":"inplace",
  "kernels" :[
    {
      "library-name":"libvvas_text2overlay.so",
      "config": {
        "debug_level" : 1,
        "font_size" : 1.0,
        "font" : 3,
        "x_offset" : 0,
        "y_offset" : 100,
        "defect_threshold" : 0.14,
        "is_acc_result" : 0
      }
    }
  ]
}

Note:

1) debug_level: Enable or disable debug log for the Kernel library.

2) font_size: User configuration to change the font size.

3) font: User configuration to change the supported font type.

4) x_offset: The X co-ordinate from where the text starts writing.

5) y_offset: The Y co-ordinate from where the text starts writing.

6) defect_threshold: The defect density threshold to calculate the defect. If the defect value is more than the threshold, it falls under defect category.

7) is_acc_result: Flag to display the accumulated result. If the value is 0, then the accumulated result will not be displayed.
For more information [see](https://docs.opencv.org/3.4/d0/de1/group__core.html#ga0f9314ea6e35f99bb23f29567fc16e11).

Next Steps¶

Hardware Accelerator
Go back to the System Architecture of the Platform