Kria™ KV260 Vision AI Starter Kit Defect Detection Tutorial

Setting up the Board and Application Deployment

Setting up the Board and Application Deployment

Introduction

This document shows how to set up the board and run the defect-detect application.

Introduction to the Test Environment

Note: Ensure that you have the Gstreamer packages installed on your Linux PC. If you are using Ubuntu as your Linux distribution, ensure that the Ubuntu version is atleast 16.04.

You need to download all the sample mango images from the Cofilab site to the Linux PC.

As the downloaded images are in JPG format, you need to convert them to GRAY 8 (Y8) format using the following steps.

1. Unzip the rar file that you downloaded.

2. In the Linux PC, go to DB_mango.

3. Copy and save the following script as convert_jpeg_y8.sh:

   for file in ./*; do       
   f=$(echo "${file##*/}");
     filename=$(echo $f| cut  -d'.' -f 1); #file has extension, it return only filename
       echo $filename
         gst-launch-1.0 filesrc location=$file ! jpegdec  ! videoconvert  ! videoscale ! video/x-raw, width=1280, height=800, format=GRAY8 ! filesink location=$filename.y8
   done
   cat Mango_*.y8 > input_video.y8
   

4. Make the script executable: chmod +x convert_jpeg_y8.sh

5. Run the script convert_jpeg_y8.sh as follows:

   ./convert_jpeg_y8.sh >& file.txt
   

   When complete, the script produces input_video.y8 as input to the Defect Detect application.

6. Copy input_video.y8 from the Linux PC to the SOM board rootfs path=/home/petalinux.

   NOTE: You can delete all files except input_video.y8.

   You can also perform the preceding conversion on the SOM board, but will need to do a sudo dnf install gstreamer1.0-plugins-good-jpeg.aarch64 first, and use sudo with all the commands.

   NOTE: For setups without access to the internet, it is possible to download and use the packages locally. Refer to the Install from a local package feed for instructions.

The Defect Detection application’s design processes these images and sends the results and the processed images on to the monitor.

See Known Issues with the Defect Detect application.

Setting up the SOM Board

Note: Skip Step 1, if you have already flashed the SD Card with the KV260 Vision AI Starter Kit Image (kv260-sdcard.img.wic)

1. Flash the SD Card

   • Download the SD Card Image and save it on your computer.

   • Connect the microSD to your computer.

   • Download the Balena Etcher tool (recommended; available for Window, Linux, and macOS) required to flash the SD card.

   • Follow the instructions in the tool and select the downloaded image to flash onto your microSD card.

     

   • Eject the SD card from your computer.

     If you are looking for other OS specific tools to write the image to the SD card, refer to KV260 Getting Started Page

2. Hardware Setup:

   

   • microSD: Insert the SD card into slot at J11.

   • Monitor: Before booting, connect the 4k monitor to the board using either DP or HDMI port.

   • UART/JTAG Interface: For interacting and viewing boot-time information, connect the micro-USB cable to J4.

   • Network Connection: Connect the Ethernet cable from J10 on SOM board to your local network that is DHCP enabled, and ensure that you have Internet access as otherwise the package feed will not be accessible.

3. Boot the Linux Image:

   NOTE: Only perform this step if you are booting the Starter kit for the first time. Otherwise, log in with the PetaLinux username and the password that was previously set.

   The Linux image will boot into the following login prompt:

   xilinx-k26-starterkit-2020_2 login:
   

   Use the petalinux user for login. You will be prompted to set a new password on the first login.

   xilinx-k26-starterkit-2020_2 login: petalinux
   You are required to change your password immediately (administrator enforced)
   New password:
   Retype new password:
   

   The petalinux user does not have root privileges. Most commands used in subsequent tutorials must be run using sudo and you may be prompted to enter your password.

   Note: For security, by default, the root user is disabled. If you want to login as root user, perform the following steps. Use the petalinux user’s password on the first password prompt, then set a new password for the root user. You can now login as root user using the newly set root user password.

   xilinx-k26-starterkit-2020_2:~$ sudo su -l root
   
   We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things:
   
   #1) Respect the privacy of others.
   #2) Think before you type.
   #3) With great power comes great responsibility.
   
   Password:
   root@xilinx-k26-starterkit-2020_2:~# passwd
   New password:
   Retype new password:
   passwd: password updated successfully
   

Installing the Defect Detect package

Install the latest application packages.

1. Check the package feed for new updates.

   sudo dnf update
   

   Confirm with “Y” when prompted to install new or updated packages.

   Sometimes it is needed to clean the local dnf cache first. To do so, run:

   sudo dnf clean all
   

2. Get the list of available packages in the feed:

   sudo xmutil getpkgs
   

3. Install each package:

   sudo dnf install packagegroup-kv260-defect-detect.noarch
   

NOTE: For setups without access to the internet, it is possible to download and use the packages locally. Refer to the Install from a local package feed for instructions.

Firmware Loading

The Defect Detection application firmware consist of bitstream, device tree overlay (dtbo) and the xclbin file. The Defect Detection firmware is loaded dynamically on user request once Linux is fully booted. Use the xmutil utility to list and load the firmware.

To list the available accelerator applications, run:

sudo xmutil listapps

You should see similar output to this:

Accelerator                   Type           Active
kv260-smartcam                XRT_FLAT         0
base                          XRT_FLAT         0
kv260-aibox-reid              XRT_FLAT         0
kv260-defect-detect           XRT_FLAT         0

Note: The Active column shows the accelerator application that is currently loaded in the system. The value is changed to 1 after the firmware is loaded.

To load the Defect Detection application firmware consisting of PL bitstream, device tree overlay, and the xclbin file, run the following command:

sudo xmutil loadapp kv260-defect-detect

Running the Defect Detect Application

There are two ways to invoke the Defect Detection application: Jupyter Notebook or command line.

Jupyter Notebook

Jupyter notebook works as an example to show the capability of the defect-detect application for specific configurations.

Jupter Notebook Defect Detect application supports File Source & File Sink.

Jupyter Notebook requires an IP address. If an IP address is not assigned by default, then perform steps 1-2.

1. If using a direct connection (no DHCP), see public documentation on how to configure your PC with a static IP on the same subnet.

2. For the SOM target, set the desired IP address within the same subnet using ifconfig, like the following example:

   ifconfig eth0 <user defined IP> netmask <user defined netmask>
   

To launch Jupyter notebook on the target, run the following command. Use Chrome web-browser to interact with the platform.

$ jupyter-lab --ip=ip-address &
// fill in ip-address from ifconfig

Output example:

xilinx-SOM-multi-cc-2020_2:~$ jupyter-lab --ip=xxx.xx.x.xxx &
[1] 1530
xilinx-SOM-multi-cc-2020_2:~$ [W 15:31:44.879 LabApp] JupyterLab server extension not enabled, manually loading...
[I 15:31:44.905 LabApp] JupyterLab extension loaded from /usr/lib/python3.7/site-packages/jupyterlab
[I 15:31:44.906 LabApp] JupyterLab application directory is /usr/share/jupyter/lab
[I 15:31:44.927 LabApp] Serving notebooks from local directory: /opt/xilinx/share/notebooks
[I 15:31:44.928 LabApp] The Jupyter Notebook is running at:
[I 15:31:44.928 LabApp] http://xxx.xx.x.xxx:8888/?token=635db1d645eeccc6a72bf1bb9c125164b1d689696348c97f
[I 15:31:44.928 LabApp]  or http://127.0.0.1:8888/?token=635db1d645eeccc6a72bf1bb9c125164b1d689696348c97f
[I 15:31:44.928 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:31:44.952 LabApp]

To access the notebook, open this file in a browser:

file:///home/root/.local/share/jupyter/runtime/nbserver-1531-open.html

Or, copy and paste one of these URLs:

• http://xxx.xx.x.xxx:8888/?token=635db1d645eeccc6a72bf1bb9c125164b1d689696348c97f

• http://127.0.0.1:8888/?token=635db1d645eeccc6a72bf1bb9c125164b1d689696348c97f

If you have started Jupyter-lab by running another Application supported on KV260 Vision AI Starter kit, you may skip the preceding step and do not need to restart. If you close the Chrome browser and then need to find the URL for Jupyter lab, run the following command:

sudo jupyter notebook list

Output example:

Currently running servers:

http://ip:port/?token=xxxxxxxxxxxxxxxxxx :: /opt/xilinx/share/notebooks

A Jupyter notebook user can run cell by cell or run the defect detection full pipeline in Jupyter Notebook. To do this, go to defect-detect.ipynb, then from the menu bar, select Kernel, and then select Restart Kernel & Run All Cells. The notebook by default expects file source location to be /home/petalinux/input_video.y8 - you can make changes in the notebook if needed.

When using a file source, data is dumped in the rootfs (/home/petalinux). You will need to offload the output files to your PC and play using a YUV player.

Command line

Use the command line to set the resolution, media type, configuration file path and more, using the defect-detect application.

More combinations could be made based on the options provided by the defect-detect application.

Defect Detection Application Usage

 sudo defect-detect --help

Usage:

defect-detect [OPTION?] - Application to detect the defect of Mango on SOM board of Xilinx.

Help Options:

  -?, --help                        Show help options

  --help-all                        Show all help options

  --help-gst                        Show GStreamer Options

Application Options:

  -i, --infile=file path            Location of GRAY8 file as input

  -x, --rawout=file path            Location of GRAY8 file as raw output

  -y, --preprocessout=file path     Location of GRAY8 file as pre-processed output

  -z, --finalout=file path          Location of GRAY8 file as final stage output

  -w, --width=1280                  Resolution width of the input

  -h, --height=800                  Resolution height of the input

  -r, --framerate=60                Framerate of the input source

  -f, --inputtype                   For live playback value must be 0 otherwise 1, default is 0

  -d, --demomode                    For demo mode value must be 1 otherwise 0, default is 0

  -m, --mediatype                   Media node should be provided in live use case, default is /dev/media0

  -c, --cfgpath=config path         JSON file path

The application is targeted to run an input source that supports GRAY8 (Y8) format.

Xilinx assumes that the file to support is in GRAY8 (Y8) format with a resolution of 12800x800.

Defect Detect application supports File Source & File Sink.

Example Commands to Run the Defect Detection Pipeline in Various Modes

Examples of supported combinations sorted by input are as follows. If using the command line to invoke the defect-detect application, stop the process using CTRL-C prior to starting the next instance.

File-In and File-Out playback:

This pipeline uses the input Y8 (GRAY8) file as an input source and a file as output. It dumps the output of the three stages to the file. Therefore, there are three out file options.

It is mandatory to specify all the three output files, as otherwise the pipeline does not work.

The Y8 (GRAY8) file can be played back using a YUV player by specifying the proper size (width=1280, height=800), and color format (Y).

sudo defect-detect -w 1280 -h 800 -r 60 -f 1 -i input.y8 -x raw.y8 -y pre_pros.y8 -z final.y8

Note: You must update the command to the specific file desired as the input source.

File-In and Display-Out playback, run the following command.

sudo 01.file-defect-detect-display.sh ./input_video.y8

Note: For File-In and Display-Out playback, all three stage outputs will be displayed on DP/HDMI. The preceding command has a sample input file path. You need to change this path as per your requirement. The script 01.file-defect-detect-display.sh contains the Gstreamer pipeline command.

Once you are done with the Defect Detection application and to switch to another accelerator application, you can unload the currently loaded accelerator application firmware by running:

sudo xmutil unloadapp kv260-defect-detect

File Structure of the Application

To view details, click here.

The application is comprised of the following files:

• Binary File: => /opt/xilinx/bin

filename	description
defect-detect	main app

• Script File: => /opt/xilinx/bin/

filename	description
01.file-defect-detect-display.sh	Invoke GStreamer pipeline to detect the defects and display outputs on DP/HDMI.

• Configuration Files: => /opt/xilinx/share/ivas/defect-detect/

filename	description
otsu-accelarator.json	Configuration of OTSU accelerator
cca-accelerator.json	Configuration of CCA accelerator
preprocess-accelarator.json	Configuration of pre-process accelerator
text2overlay.json	Configuration of text2overlay accelerator

• Jupyter notebook file: => /opt/xilinx/share/notebooks/defect-detect

filename	description
defect-detect.ipynb	Jupyter notebook file to run the Defect Defection pipeline (File Source).

Next Steps

• Software Architecture

• Go back to the KV260 SOM Defect detect design start page

License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License.

You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright© 2021 Xilinx