mlir-aie maintainer dashboard

One page, utils/kernel_checks/index.html, with no build step: the nightly workflows copy it to gh-pages:dashboard/ each time they publish, and it reads the data they publish beside it.

Where the data is

kernel-checks/<npu>/

the kernel checks, per NPU, written by nightlyKernelChecks.yml through publishKernelResults.yml

component-checks/<check>/

the performance and compiler checks, per check: sa-placer and sa-placer-hw, written by nightlyComponentChecks.yml; allocator and router, written by modelBasedTests.yml

ci-health/

status.json, the critical test workflows' figures, merge time and the scheduled workflows' recent runs, and history.json, a point a day of those figures; written every six hours by publishCiHealth.yml (utils/kernel_checks/ci_health.py)

code-health/

status.json, last night's coverage, sanitizer findings, code scanning alerts and clang-tidy findings, and history.json, a point a night; written by nightlyCodeHealth.yml (utils/kernel_checks/code_health.py)

Both are written by utils/kernel_checks/publish.py in one format: runs/<id>.json (one run's record), runs.json (a summary per run), latest.json (the newest record that published numbers; the kernel checks' pull-request report compares against it) and history/<metric>.json (every run's value of one metric per case). A kernel NPU also has catalogue.json, every kernel factory and how its cases fared. Runs older than 90 days thin to weekly. Files carry a schema number, and the page sets aside a newer one rather than misread it.

Lower is better, except where a metric says higher is better: rates such as tokens or images per second, and verified counts. A change is colored by whether it is worse, not by its sign, and each chart's axis says which way is better. A change counts once it passes the metric's threshold in utils/kernel_checks/thresholds.json, which the pull-request report reads too; a noisy timing also has to pass a multiple of its run-to-run spread. Runs started by hand are charted but never compared against.

Working on the page

python3 utils/kernel_checks/preview.py --fetch                 # the published data, this page
python3 utils/kernel_checks/snapshot.py --fetch                  # the same as one self-contained file, no server or mlir_aie
python3 -m pytest test/test_kernel_checks_page.py                # the page's code, under node; its own failure states, against recorded data

To add a section: publish its checks to component-checks/<check>/ from a nightly job (add the check to COMPONENTS in publish.py), then add the section to SECTIONS and its checks to BANDS in the page.

publishCiHealth.yml runs every six hours: utils/kernel_checks/ci_health.py asks GitHub's API, with the workflow's own token, for each workflow's recent scheduled runs on main and, for those on the NPU and bench machines, the jobs of the latest run, and writes ci-health/status.json beside this page. The scheduled jobs are judged on their scheduled runs on main alone; the tests that gate a merge on their runs on main, and on how often a pull-request or merge-queue commit needed a re-run to pass.

Machines lists each self-hosted runner named in the jobs of those runs on main and of the scheduled jobs' latest runs, judged on its latest jobs rather than its two-week tally. Red wants a reboot: its last two jobs did not pass, or its last lost its runner mid-job. Amber wants a look: its latest job failed (often the test's fault; said when the other machines in the same run passed), or it has had no job for three days while the others did, which usually means it is offline. GitHub's API does not say which runners are online without an admin token, so silence is the sign. One-off cloud machines, named with a long id, are one row: each is gone after its job. A machine idle the whole window does not appear.

Every workflow is listed once, in one of two kinds. Tests that gate a merge are the workflows the merge queue runs, listed under critical in the file below. They also run on main, nightly and for some after each push; each is judged on its latest run there, names the jobs that failed in it, and is late when main has not run it for 36 hours. Main failing 2 of its last 5 runs, the latest passing, is intermittent: amber, worth watching. A hardware workflow's first failure after a passing run is also amber; software-only workflows fail red immediately. Its pass rate on main and how often a pull-request or merge-queue commit needed a re-run to pass, over two weeks, are its normal: shown under it, never judged, since a hardware runner that wedges now and then is usual.

Nightly and scheduled jobs are judged on their latest scheduled run on main, and are late when it started more than one and a half of their periods ago (36 hours for a nightly). A cancelled run has no result: the run before it stands. A job still queued after six hours almost always means no runner with its labels is online: red, since the nightly will not run until a machine is back.

Run time charts one workflow's finished runs, oldest first: how long each took from its first job's start, and how long its slowest job waited for a runner. A failed or timed-out run's point is red, one that lost its runner or found none amber; hover a point for its commit, click it for the run. A run on main that timed out counts as failed; one whose runner was lost, or never came, is the machines' and counts neither way.

Merging: the pull requests merged in the last two weeks and how long each was open, drafts included; and those open now.

To list another workflow, add it to utils/kernel_checks/ci_workflows.json.

What a card calls worse, here and under Compiler checks, is judged on two weeks of nightlies, not the one night before. A step is the latest nightly past the measure's threshold from the median of the week before it and outside that week's whole range; the card links the commits between the two nightlies. A creep (tagged with the date it is since) is the median of the last three nightlies past the threshold from the first three of the two weeks: a regression that came in several small steps. A move on one seed or batch of a measure the others held is a blip, folded away and never turning a card amber. Each line's bars are its two weeks. The tables' changes are still against the nightly before.

utils/component_checks/sa_placer_hw.py runs in nightlyComponentChecks.yml at 06:30 UTC on an npu2 bench runner (Strix or Krackan; each run names its part). The MobileNet design is placed with the SA placer at seeds 3 (the design's default), 2 and 7, then compiled once per seed and effort and run and verified at every power of two from 1 to 64 images per launch, every image checked. A publishing run refuses to time unless the NPU is in turbo mode.

utils/component_checks/llama_chain_hw.py runs in the same workflow, queued on the same NPU. The Llama 3.2 1B 16-layer persistent decode is compiled twice in one fresh cache, cold then warm, its 4 tokens per dispatch checked one by one against the numpy oracle, then timed.

utils/component_checks/yolo26n_hw.py runs after it, on the same NPU. The YOLO26n 640×640 detector is compiled twice in one fresh cache, cold then warm, run on 8 COCO128 images, each checked against the numpy oracle and, for information, onnxruntime, then timed at batch 1 and checked and timed again at batches of 2, 4, 8, 16, 32 and 64 images, all from one build (each batch an entry of the design with its own instruction stream).

What each run records

Dispatch and end-to-end per image are the fastest of 20 launches, over the images in the launch. Records from before one dispatch-time compile served every batch (each batch compiled on its own) are not charted beside them.

To compile one seed at one effort and run every batch as the nightly does, from programming_examples/ml, with E 0.05 for the design's default effort or 1.0 for full effort:

python -m mobilenet.aie2_mobilenet_iron --sa-seed N --sa-effort E --dynamic 1 2 4 8 16 32 64 -w 20 -i 20

and every seed and batch, as the nightly does:

python utils/component_checks/sa_placer_hw.py --out perf.json --meta meta.json

The Llama decode, from programming_examples/ml/llama_3_2_1b after gen_llama_data.py has built data/ from the model's weights:

python aie2_decode.py --p-max=4 --kset=8 --n-layers=16 --t=128 --full-elf-path build/chain_persist_grow_N16_PT4.elf && python test_decode.py --p-max=4 --kset=8 --n-layers=16 --t=128 --elf build/chain_persist_grow_N16_PT4.elf

and as the nightly does:

python utils/component_checks/llama_chain_hw.py --out perf.json --meta meta.json

The YOLO26n detector, from programming_examples/ml/yolo26n after gen_yolo_data.py has built data/ from the quantized model:

python yolo26n.py --images $IRON_EXAMPLE_WEIGHTS_DIR/yolo26n/coco128/images/train2017 --n 8

and as the nightly does:

python utils/component_checks/yolo26n_hw.py --out perf.json --meta meta.json

nightlyComponentChecks.yml runs at 06:30 UTC from main, on pull requests that touch the placer or these checks (the seed sweep only), and on demand; modelBasedTests.yml runs the router and allocator at 03:17 UTC from main and on demand, on a hosted runner with an assertions build. The nightly and an unfiltered dispatch on main publish; a check that fails still publishes what it measured, and its record lists the failed seeds and runs.

Seed sweep (no hardware)

utils/component_checks/sa_placer.py, on a hosted runner: aie-opt places two fixtures, the 4-core design in test/place-tiles/sa_placer/test_sa_effort.mlir and MobileNet before placement (utils/component_checks/fixtures/mobilenet.mlir), with seeds 1 to 20 at full effort. Per fixture it records how many seeds failed to place, the SA placer's final cost (mean and worst), and the CPU time and peak memory of a placement.

aie-opt '--aie-place-tiles=placer=sa_placer sa-seed=N sa-effort=1.0' \
    --mlir-pass-statistics utils/component_checks/fixtures/mobilenet.mlir

MobileNet placements on hardware (npu2)

utils/component_checks/sa_placer_hw.py: MobileNet is placed with seeds 3 (the design's default), 2 and 7, then compiled once per seed and effort and run and verified at every power of two from 1 to 64 images per launch, every image checked. For each seed it records the placement aiecc chose (a digest of the placed tiles) and its SA cost, replayed with aie-opt. What the placed design then costs to run is under Performance.

To rerun a whole check, run its script with --out perf.json --meta meta.json (the sweep also needs --aie-opt "$(which aie-opt)"; the hardware check takes --seeds and --batches). It prints a table with a row per seed, or per seed and batch.

Router model-based tests (no hardware)

router_properties.py: each night takes a new range of seeds, so the router meets designs it has not seen (the range is in the run's record, and a failure names the seeds to run again). It generates designs whose routing the model can construct or rule out, on npu1, npu2 and xcvc1902, routes each with hops on and off, and verifies the result: legal, deterministic, as short as the constructed routing, warned and rejected as the model predicts. Per device it records the properties that failed, the routings that failed or were illegal, the hops over the constructed routing, the arbiter msels per design and the time per design.

router_mutation.py changes routed designs in ways with a known answer (the router must route it, reject it, or route it the same), and corrupts routed output to measure how often the model's verifier notices. It records the cases that failed and the corruptions the verifier missed.

python test/create-packet-flows/nightly/router_properties.py --device npu2 --seeds 2000 --first-seed N
python utils/router_testing/router_mutation.py --device npu2 [--hops-off] --seeds 400 --first-seed N

Buffer allocator model-based tests (no hardware)

allocator_properties.py: aie-opt --aie-assign-buffer-addresses places 5,000 generated designs and 200,000 with per-bank reservations, each built around a layout known to fit, so a design that does not place is a bug. check-aie runs the same test over the first 200 and 12,000. It records the properties that failed, the designs left unplaced or placed illegally, the bank crossings a better placement avoids, the bank pairs cores share, and the run's wall time.

python test/assign-buffer-addresses/allocator_properties.py --seeds 5000 --bankres-seeds 200000

How to choose the placer and a seed for your own design is in section 1 of the programming guide.

nightlyCodeHealth.yml runs on main every night and utils/kernel_checks/code_health.py writes code-health/status.json and history.json beside this page. Nothing here gates a merge.

Coverage: check-aie built with clang's source-based coverage (codeCoverage.yml), counted over aie-opt, aie-translate and aiecc for lib/, include/, tools/ and python/. Amber when last night covered fewer lines than the night before, by points or more; the files that lost covered lines are named.

ASan and UBSan, TSan: check-aie (and, under TSan, check-aie-concurrency) on a build of this repository with -fsanitize=address,undefined or -fsanitize=thread. Every test's output is read, a passing one's too, since a test that expects a tool to fail passes when a sanitizer aborts it. A report is one finding per sanitizer, kind and the function in this repository's code it names first, however many tests reach it. Red for a finding new last night; amber for one still open, dated by the night it appeared. A test that failed with no report is listed apart: instrumented tools run several times slower, so that is usually a time-out. The prebuilt MLIR is not instrumented: a bug inside it goes unseen, and under TSan its atomics can read as races, so a TSan finding whose stack is mostly MLIR's wants a second look. Leak checking is off for now: the Python tests' interpreter leaks by design at exit.

Code scanning: the repository's open alerts, CodeQL's Python analysis (codeql.yml, on every push) and its C++ analysis (traced on the ASan build each night). Red for a new alert of high or critical severity, amber for a new one of any other.

clang-tidy over every C++ file in lib/ and tools/, with the repository's .clang-tidy. lintAndFormat.yml enforces it only on the files the clang-tidy hook in .pre-commit-config.yaml lists; a finding in one of those got past that gate and is red. The rest are counted, by check, and a file outside the list with none is ready to join it.

cmake ... -DCMAKE_CXX_FLAGS="-O1 -g -fsanitize=address,undefined -shared-libsan" ...   # as nightlyCodeHealth.yml
ASAN_OPTIONS=detect_leaks=0 lit -v build/test/<test>
MLIR_AIE_BUILD_DIR=build utils/run_clang_tidy.sh lib/<file>.cpp