Section 6 - Larger Example Designs¶
There are a number of example designs available here, which further help explain many of the unique features of AI Engines and the NPU array in Ryzen™ AI. This section contains more complex application designs for both vision and machine learning use cases. In particular, we will describe a ResNet implementation on Ryzen™ AI.
Vision Kernels¶
| Design name | Data type | Description |
|---|---|---|
| Vision Passthrough | i8 | A simple pipeline with just one passThrough kernel. This pipeline mainly aims to test whether the data movement works correctly to copy a greyscale image. |
| Color Detect | i32 | This multi-kernel, multi-core pipeline detects colors in an RGBA image. |
| Edge Detect | i32 | A multi-kernel, multi-core pipeline that detects edges in an image and overlays the detection on the original image. |
| Color Threshold | i32 | A multi-core data-parallel implementation of color thresholding of a RGBA image. |
Machine Learning Designs¶
| Design name | Data type | Description |
|---|---|---|
| bottleneck | ui8 | A Bottleneck Residual Block is a variant of the residual block that utilizes three convolutions, using 1x1, 3x3, and 1x1 filter sizes, respectively. The implementation features fusing of multiple kernels and dataflow optimizations, highlighting the unique architectural capabilities of AI Engines |
| resnet | ui8 | ResNet with offloaded conv2_x layers. The implementation features depth-first implementation of multiple bottleneck blocks across multiple NPU columns. |
| MobileNet V3 | ui8 / i8 | End-to-end MobileNet V3 inference, depth-first across the 4x8 array of the Strix NPU2. The weights are a seeded random init (untrained), so accuracy here means every image of the batch bit-exact against mobilenet_numpy, within 3 of the brevitas golden. |
| YOLO26n | int8 | Stock Ultralytics YOLO26n object detection at 640x640, the whole graph on the NPU2, from a uint8 image to the (300, 6) detections. |
| Llama 3.2 1B | int8 | Llama 3.2 1B decode and prompt on the NPU2, all 16 layers, with on-chip token feedback and a growing KV cache. |
The MobileNet V3, YOLO26n and Llama 3.2 1B designs are measured every night; their current latency and throughput are on the dashboard.
Exercises¶
- In bottleneck design, how many different types of fused computations do you observe?
Show answer
- Convolution fused with ReLU and Convolution fused with element-wise addition. Please note that fusing adjacent convolution and batch norm layers is another inference-time optimization, which our implementation can handle.
Show answer
- This allows for the necessary neighborhood information required by the convolutional kernel to be available for processing.