Section 5 - Example Vector Designs¶
The programming examples are a number of sample designs that further help explain many of the unique features of AI Engines and the NPU array in Ryzen™ AI.
Simplest¶
Passthrough¶
The passthrough example is the simplest "getting started" example. It copies 4096 bytes from the input to output using vectorized loads and stores. The design example shows a typical project organization which is easy to reproduce with other examples. There are only two important source files here.
1. passthrough_kernel.py The IRON structural design plus the host-side test driver. Decorated with @iron.jit so the first call compiles the design and runs it on the NPU, then verifies the result against the input. Also shows a simple use of the ObjectFifos described in section 2.
1. passThrough.cc This is a C++ file which performs the vectorized copy operation.
The passthrough DMAs example shows an alternate method of performing a copy without involving the cores, and instead performing a loopback.
Basic¶
| Design name | Data type | Description |
|---|---|---|
| Vector Scalar Add | i32 | Adds 1 to every element in vector |
| Vector Scalar Mul | i16 | Returns a vector multiplied by a scale factor |
| Vector Vector Add | i32 | Returns a vector summed with another vector |
| Vector Vector Modulo | i32 | Returns vector % vector |
| Vector Vector Multiply | i32 | Returns a vector multiplied by a vector |
| Vector Reduce Add | bfloat16 | Returns the sum of all elements in a vector |
| Vector Reduce Max | bfloat16 | Returns the maximum of all elements in a vector |
| Vector Reduce Min | bfloat16 | Returns the minimum of all elements in a vector |
| Vector Exp | bfloat16 | Returns a vector representing ex of the inputs |
DMA Transpose (using --strategy=dma) |
i32 | Transposes a matrix with the Shim DMA using npu_dma_memcpy_nd |
| Matrix Scalar Add | i32 | Returns a matrix with a scalar added to each element |
| Single core GEMM | bfloat16 | A single core matrix-matrix multiply |
| Multi core GEMM | bfloat16 | A matrix-matrix multiply using 16 AIEs with operand broadcast. Uses a simple "accumulate in place" strategy |
| GEMV | bfloat16 | A vector-matrix multiply returning a vector |
Machine Learning Kernels¶
| Design name | Data type | Description |
|---|---|---|
| Eltwise (Add / Mul) | bfloat16 | Element-wise addition or multiplication of two vectors (op={add,mul} option). |
| Eltwise Unary (ReLU / SiLU / GELU) | bfloat16 | Element-wise ReLU, SiLU, or GELU activation on a vector (op={relu,silu,gelu} option). |
| Softmax | bfloat16 | Softmax operation on a matrix |
| Conv2D (optional fused ReLU) | i8 | 1x1 Conv2D for CNNs; fuse_relu=1 swaps the output to uint8 saturation, fusing ReLU at the vector register level. |
Exercises¶
-
Can you modify the passthrough design to copy more (or less) data?
Show answer
Pass--in1_sizetopassthrough_kernel.py; the output size follows the input size. -
Take a look at the host driver in our Vector Exp example vector_exp.py. Take note of the data type and the size of the test vector. What do you notice?
Show answer
We are testing 65536 values or 2^16, therefore testing all possiblebfloat16values through the approximation. -
What is the communication-to-computation ratio in Eltwise Unary (ReLU)?
Show answer
~6 as reported by the Trace. This is why it is a good candidate for kernel fusion with Conv2D or GEMMs for ML. -
HARD Which basic example is a component in Softmax?
Show answer
Vector Exp