Kria™ KV260 Vision AI Starter Kit Smart Camera Tutorial

Software Architecture of the Accelerator

Software Architecture of the Accelerator

Introduction

This document describes the software architecture of the SmartCam accelerator application.

Software Platform and Dependencies

We leverage the GStreamer multi-media orchestration platform for application software development. As described in following sections about GStreamer pipeline, all the processing units in the stream pipeline are presented to the user as easily pluggable and changeable plugins, enabling a “low touch” customer adaptation of the reference design pipeline within the context of an open-source and non-vendor specific implementation.

Vitis AI 1.3.0 is the core underlying component to access the AI inference capability provides by Xilinx DPU.

To access DPU and other PL hardware accelerator functions from GStreamer, Xilinx developed Intelligent Video Analysis SDK (IVAS) to provide convinient and customizable GStreamer plugins for it.

GStreamer Pipeline

Smart Camera Accelerated Application supports three types of inputs including MIPI, video file and USB camera.

For MIPI source the ISP will output frames with NV12 format directly, and for other sources, there are mature plugins in GStreamer serving to perform the conversion too.

Three AI tasks are supported: Facedetect, Refinedet, SSD. We use ${AI_TASK} to refer to either of them in following part of this document.

• AI Preprocess

  NV12 to BGR conversion, image resizing, and input quantizing blocks are to meet the requirement of the DPU AI inference engine. These 3 steps are done in one dedicate preprocessing IP, which is detailed in the Accelerator IP Modules, to achieve the optimal framerate and latency.

  ivas_xmultisrc GStreamer plugin with customized kernel /opt/xilinx/lib/libivas_xpp.so are used to integrate the accelerator IP functionality into the pipeline.

  Configuration file: /opt/xilinx/share/ivas/smartcam/${AI_TASK}/preprocess.json, contains the info to do quantization on the resized BGR images, these values may vary between different AI tasks configs as the core AI algorithm demands.

    "config": {
        "debug_level" : 0,
        "mean_r": 128,
        "mean_g": 128,
        "mean_b": 128,
        "scale_r": 1,
        "scale_g": 1,
        "scale_b": 1
    }

• AI Inference

  ivas_xfilter GStreamer plugin with kernel library /usr/lib/libivas_dpuinfer.so works as middleware between application which interface with user and underlying Vitis AI library which interface with DPU to do the actual AI inference tasks.

  For the 3 AI tasks already supported, no more customization is needed here. To change the underlying Vitis AI model or otherwise customize the plugin, refer to IVAS documentation.

• AI Rendering

  The results from the AI Inference step (such as coordinates of detected faces) are passed to the AI Rendering plugin which is responsible for drawing bounding boxes around the objects in the original frame.

  ivas_xfilter GStreamer plugin and kernel library /usr/lib/libivas_xboundingbox.so do the AI results rendering work, based on the meta info passed down from AI Inference plugin to here with video buffers.

  • Configuration file:

    /opt/xilinx/share/ivas/smartcam/${AI_TASK}/drawresult.json, provide the way to customize the rendering properties such as font of the label, colours of the bounding box for different classes.

    Facedetect/drawresult.json

        "config": {
            "font_size" : 0.5,
            "font" : 3,
            "thickness" : 2,
            "debug_level" : 0,
            "label_color" : { "blue" : 0, "green" : 0, "red" : 255 },
            "label_filter" : [ "class", "probability" ],
            "classes" : [
            ]
        }

    For detailed customization, please refer to IVAS documentation.

Additional GStreamer component used:

Gst-Rtsp-Server lib is utilized to build up the RTSP server video stream with AI detection results.

Next Steps

• Read Hardware Architecture of the Platform

• Go back to the KV260 SOM Smart camera design start page

License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License.

You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright© 2021 Xilinx