Kria™ KV260 Vision AI Starter Kit Smart Camera Tutorial

Hardware Architecture of the Platform

Hardware Architecture of the Platform

Introduction

This section describes the design implemented in the programmable Logic (PL). The following figure shows the top-level hardware architecture of the reference design.

At a high-level, the design is comprised of four pipelines.

Capture/Input pipeline: This comprises of pipelines through which audio/video data are captured.

• USB capture pipeline (PS)

• MIPI CSI-2 capture pipeline (PL)

• I2S Audio receive pipeline (PL)

Display/Output pipeline: This comprises of pipelines through which audio/video data are outputted.

• DisplayPort pipeline (PS)

• Ethernet pipeline (PS)

• I2S Audio transmit pipeline (PL)

Video processing pipeline: This comprises of a video codec unit (VCU) encoder and decoder for compressing/decompressing video data.

Accelerator pipeline: This comprises of overlay accelerator functions integrated into the platform using AMD Vitis™.

• The Deep Learning Processing Unit (DPU) IP runs different neural network models (PL).

• The Pre-Processing block modifies the input data as required by the network (PL).

NOTE: The PS interconnects in the figure are conceptual.

Capture

Single Sensor MIPI Capture

A capture pipeline receives video frames from an external source and writes it into memory. The single sensor MIPI CSI-2 receiver capture pipeline is shown in the following figure.

This pipeline consists of four components, of which two are controlled by the application processing unit (APU) via an AXI-Lite based register interface; one is controlled by the APU via an I2C register interface, and one is configured statically.

• The On Semi AR1335 is a complementary metal–oxide semiconductor (CMOS) active pixel image sensor with a pixel array of 4208H x 3120V. It sends sensor data to the ISP through a four-lane MIPI interface. It is controlled and programmed by the ISP via a two-wire serial interface. The sensor is connected to the IAS 0 module connector on the ML carrier card. For more information, refer to the datasheet from https://www.onsemi.com/products/sensors/image-sensors-processors/image-sensors/ar1335.

• The On Semi AP1302 ISP is an image signal processor digital image sensor supporting image resolutions up to 4224x 3156V. AP1302 interfaces to CMOS imaging sensors and performs all the necessary operations required to capture video streams. It performs functions like auto white balancing (AWB), auto exposure (AE), and auto focus (AF) etc. The ISP is controlled via an I2C interface using an AXI I2C controller in the PL. It is mounted on a ML carrier card and has a MIPI output interface that is connected to the MIPI CSI-2 RX subsystem inside the PL. A general purpose I/O (GPIO) is used to reset the IP. For more information, refer to the datasheet from https://www.onsemi.com/products/sensors/image-sensors-processors/image-processors/ap1302.

• The MIPI CSI-2 receiver subsystem (CSI Rx) includes a MIPI D-PHY core that connects four data lanes and one clock lane to the ISP. It implements a CSI-2 receive interface according to the MIPI CSI-2 standard v2.0 with underlying MIPI D-PHY standard v1.2. The subsystem captures images from the ISP in YUV 4:2:0 format and outputs AXI4-Stream video data. At two pixels per clock (PPCS) and eight bits per pixel, the AXI4-Stream width is 32-bits. For more information, see the MIPI CSI-2 Receiver Subsystem Product Guide (PG232).

• The AXI subset converter, see the AXI4-Stream Infrastructure IP Suite LogiCORE IP Product Guide (PG085), is a statically-configured IP core that converts the 32-bit AXI4-Stream input data to a 48-bit AXI4-Stream output data by adding zeros to themost significant bit (MSB) of each data word. At two PPC, the AXI4-Stream width is 48-bits. The format on the AXI4-Stream is shown for the even and odd lines:

  

• The video frame buffer takes YUV 4:2:0 AXI4-Stream input data and converts it to AXI4-MM format which is written to memory in semi-planar YUV 4:2:0 format. The Y and UV are stored in separate planes as shown below. The AXI-MM interface is connected to the S_AXI port of the PS.

  

For each video frame transfer, an interrupt is generated. A GPIO is used to reset the IP between resolution changes. For more information, see Video Frame Buffer Read and Video Frame Buffer Write LogiCORE IP Product Guide (PG278).

All the IPs in this pipeline are configured to transport 2 PPC at 300 MHz, enabling up to 3840x2160 resolution at 60 frames per second (FPS). But the MIPI line rate of 896 Mbps does not support 60 fps. The maximum throughput requirement is 3840x2160 resolution at 30 FPS, so the set MIPI line rate should be adequate.

NOTE: The MIPI video clock requirement for NV12 (YUV420) at the line rate 896 Mbps is 221 MHz at 2ppc, so you could run all the IPs at 221 MHz at 2ppc. For more information, see the MIPI CSI-2 Receiver Subsystem Product Guide (PG232).

Display

The video output pipelines and video processing pipelines are not covered in this section as they are not PL components. For more information, refer to DisplayPort Controller and GEM Ethernet chapters in the Zynq UltraScale+ Device Technical Reference Manual (UG1085). For more information on Video Processing pipelines, refer to Video Codec Unit LogiCORE IP Product Guide (PG252.)

Audio Transmit and Receive

This platform also supports the capture and replay of audio. The I2S audio RX-to-TX pipeline is shown in the following figure. This pipeline consists of three components, all of which are controlled by the APU through an AXI4-Lite base register interface.

• The I2S Receiver/Transmitter IP can be connected to audio devices for receiving and transmitting PCM audio. The I2S IPs typically interface with the external analog-to-digital converter (ADC)/ digital-to-analog converter (DAC) which facilitates the playback of audio. The IPs are configured to have two audio channels. The IPs supports any sampling rate and can be configure registers via the AXI4-Lite Slave interface. For more information, refer to the LogiCORE IP Product Guide for I2S Transmitter and I2S Receiver (PG308). In this reference design, no acceleration function is run on the Aaudio data and it is in pass through mode.

• The audio formatter provides high-bandwidth direct memory access between the memory and AXI4-Stream target peripherals. The initialization, status, and management registers are accessed through an AXI4-Lite slave interface. It is configured with both read and write interfaces enabled for a maximum of two audio channels and an interleaved memory packing mode with memory data format configured asadvanced encryption standard (AES) to PCM. The IP receives audio input from the I2S RX IP and writes the data to memory. It reads the audio data from the memory and sends it out to the I2S TX IP, which forwards it to the output device. See the Audio Formatter Product Guide (PG330) for more information.

Clocks, Resets, and Interrupts

Clocks

The following table identifies the main clocks of the PL design, their source, their clock frequency, and their function.

Clock	Clock Source	Clock Frequency	Function
pl_clk0	PS	100 MHz	Clock source for clocking wizard (clk_wiz_0) generating clocks
clk_200M	Clocking wizard	200 MHz	MIPI D-PHY core clock
clk_100M*	Clocking wizard	100 MHz	AXI4-Lite clock to configure the different audio and video IPs in the design. It is also used as a AXI4-Stream clock for the stream connection between the I2S receiver IP and the Audio formatter IP
clk_300M*	Clocking wizard	300 MHz	AXI4 MM clock and AXI4-Stream clock used in the capture pipeline, display pipeline, and processing pipeline
clk_50M	Clocking wizard	50 MHz	PLL reference clock for the VCU IP
clk_600M*	Clocking wizard	600 MHz	Generated to be used by the accelerator
pl_clk1	PS	100 MHz	Clock source for clocking wizard (clk_wiz_audio) generating clocks mainly for the audioIPs
clk_out1	Clocking wizard	18.43 MHz	AXI MM clock and AXI4-Stream clock used in the audio pipeline. Also drives the mclk_out_rx on the 12-PIN PMOD interface via an ODDR
clk_out2	Clocking wizard	18.43 MHz	Drives the mclk_out_tx on the 12-PIN PMOD interface via an ODDR. Phase shifted by180 degrees wrt clk_out1
lrclk_out, sclk_out	I2S receiver	variable	Drives clocks on the 12-PIN PMOD interface
lrclk_out, sclk_out	I2S transmitter	variable	Drives clocks on the 12-PIN PMOD interface

*Clocks exposed as a platform interface and can be used by an accelerator.

Resets

The following table summarizes the resets used in this design.

Reset Source	Function
pl_resetn0	PL reset for proc_sys_reset modules and the clock_wizard generating clocks for the video domain
rst_processor_100MHz	Synchronous resets for clk_out100M clock domain
rst_processor_300MHz	Synchronous resets for clk_out300M clock domain
rst_processor_600MHz	Synchronous resets for clk_out600M clock domain
pl_resetn1	PL reset for proc_sys_reset modules and the clock_wizard generating clocks for the audio domain
rst_processor_18MHz	Synchronous resets for clk_out1 clock domain
GPIOs
emio_gpio_o[0]	Frame Buffer Write IP reset
emio_gpio_o[1]	AP1302 ISP Reset
emio_gpio_o[2]	VCU reset

Interrupts

The following table lists the PL-to-PS interrupts used in this design.

Interrupt ID	Instance
pl_ps_irq1[0]	MIPI RX Subsytem IP
pl_ps_irq1[1]	Frame Buffer Write IP
pl_ps_irq1[2]	VCU IP
pl_ps_irq1[3]	AXI I2C IP
pl_ps_irq1[4]	I2S receiver IP
pl_ps_irq1[5]	I2S transmitter IP
pl_ps_irq1[6]	Audio formatter IP - Stream to Memory Map
pl_ps_irq1[7]	Audio formatter IP - Memory Map to Stream
pl_ps_irq0	Exposed as a platform interface and can be used by an accelerator

Resource Utilization

The resource utilization numbers on this platform post implementation is reported in the following table.

Resource	Utilization	Avaialble	Utilization %
LUT	17312	117120	14.78
LUTRAM	1036	57600	1.80
FF	32031	234240	13.67
BRAM	19.5	144	13.54
URAM	8	64	12.59
DSP	1	1248	0.08
IO	23	186	12.37
BUFG	11	352	3.13
MMCM	2	4	50.0
PLL	1	8	12.5

Next Steps

• Hardware Architecture of the Accelerator

• Go back to the KV260 SOM Smart Camera Design Start Page

_{Copyright © 2021-2024 Advanced Micro Devices, Inc}

^{Terms and Conditions}