L1 Test

All L1 primitive implementations have been tested against numpy functions. That is, a python based testing environment has been developed to generate random test inputs for each primitive and its corresponding function in numpy, compute the golden reference via the numpy function call, and finally compare the golden reference with the csim and cosim outputs of the primitive to verify the correctness of the implementation. To run the testing process of L1 primitives, please follow the steps below.

1. Set up Python environment

Please follow the instructions described in Python environment setup guide to install anaconda3 and setup xf_blas environment. All testing should be run under xf_blas environment. Please deactivate xf_blas environment after testing.

2. Set up Vivado_hls environment

Please navigate to directory L1/tests, and change the setting of environment variable TA_PATH to point to the installation path of your Vitis 2020.2, and run following command to set up Vivado_hls environment.

export XILINX_VITIS=${TA_PATH}/Vitis/2020.2
export XILINX_VIVADO=${TA_PATH}/Vivado/2020.2
source ${XILINX_VIVADO}/settings64.sh

3. Test L1 primitives

The L1 primitives can be tested individually or as a group. To launch the testing process, please navigate to the directory L1/tests, and enter the following command.

$ python ./run_test.py --operator amax amin asum axpy copy dot nrm2 scal swap gemv gbmv sbmvLo sbmvUp tbmvLo tbmvUp trmvLo trmvUp symvLo symvUp spmvUp spmvLo tpmvLo tpmvUp

The above command will test and verify all L1 primitives’ implementation in both csim and cosim modes. Hence, it can take a very long time. The following commands show examples for quickly testing some primitives in pure csim or cosim mode.

$ python ./run_test.py --operator amax amin --csim
$ python ./run_test.py --operator copy dot --cosim

By default, the testing process only runs in a single thread mode. To speed up the process, users can run the testing with multiple thread via –parallel option. For example,

$ python ./run_test --operator gemv gbmv --parallel 4

4. Test configuration

For each primitive, a test configuration file profile.json has been provided to specify the test inputs range, the size of the input vector or matrix, the template parameter value used for instantiating the primitive and the simulation mode (csim or cosim) used for testing. Users can find the profile.json file under directory L1/include/hw/primitive_name. For example, the profile.json file under L1/include/hw contains the following code.

 {
  "b_csim": true,
  "b_synth": true,
  "b_cosim": true,
  "dataTypes": [
    "float64",
    "uint16",
    "int32"
  ],
  "retTypes": [
    "int32",
    "int32",
    "int32"
  ],
  "op": "amax",
  "logParEntries": 2,
  "vectorDims": [
    1024,
    4096,
    8192
  ],
  "valueRange": [
    -1024,
    1024
  ],
  "numSimulation": 2
}

The configuration file will instruct our testing infrastructure to generate the tests.

5. Test outputs

At the end of the testing process, users will find a file called statistics.rpt that summarizes the test results, Passed or Failed for each primitive under test. For each primitive, there is also a file called report.rpt in the primitive’s folder under directory out_test/. This file summarizes the quality of the implementation, namely the resource usage and the efficiency of the implementation. Where efficiency is calculated by equation theoretical_cycles / measured_cosim_cycles. The higher the efficiency, the better the performance the implementation will provide.