How each area did on its latest run. Open one for what went wrong and how to reproduce it.
About this dashboard
One page, utils/kernel_checks/index.html, with no build step:
the nightly workflows copy it to gh-pages:dashboard/ each time they publish, and it reads the data they publish beside it.
Where the data is
- kernel-checks/<npu>/
- the kernel checks, per NPU, written by
nightlyKernelChecks.ymlthroughpublishKernelResults.yml - component-checks/<check>/
- the performance, routing and placement checks, per check:
sa-placerandsa-placer-hw, written bynightlyComponentChecks.yml;allocatorandrouter, written bymodelBasedTests.yml - ci-health/
status.json, the critical test workflows' figures, merge time and the scheduled workflows' recent runs, andhistory.json, a point a day of those figures; written every six hours bypublishCiHealth.yml(utils/kernel_checks/ci_health.py)
Both are written by utils/kernel_checks/publish.py in one format:
runs/<id>.json (one run's record), runs.json (a summary per run), latest.json (the newest record that published numbers;
the kernel checks' pull-request report compares against it) and history/<metric>.json (every run's value of one metric per case).
A kernel NPU also has catalogue.json, every kernel factory and how its cases fared. Runs older than 90 days thin to weekly.
Files carry a schema number, and the page sets aside a newer one rather than misread it.
Every metric is smaller-is-better. A change counts once it passes the metric's threshold in
utils/kernel_checks/thresholds.json,
which the pull-request report reads too; a noisy timing also has to pass a multiple of its run-to-run spread.
Runs started by hand are charted but never compared against.
Working on the page
python3 utils/kernel_checks/preview.py --fetch # the published data, this page python3 utils/kernel_checks/preview.py --scenario bad-night # a synthetic failing night on top python3 utils/kernel_checks/preview.py --site components --scenario bad-night python3 -m pytest test/test_kernel_checks_page.py # the page's code, under node
To add a section: publish its checks to component-checks/<check>/ from a nightly job
(add the check to COMPONENTS in publish.py), then add the section to SECTIONS and its checks to BANDS in the page.
CI health
The tests that gate a merge, the other scheduled jobs, and how long merging takes.
ActionsMerge queueDetails: every workflow, and how long its runs take
Nightly and scheduled jobs
Run time
How this section works
publishCiHealth.yml runs every six hours:
utils/kernel_checks/ci_health.py asks GitHub's API,
with the workflow's own token, for each workflow's recent scheduled runs on main and, for those on the NPU and bench machines, the jobs of the latest run,
and writes ci-health/status.json beside this page. Pull-request, merge-queue and hand-started runs are left out.
Every workflow is listed once, in one of two kinds. Tests that gate a merge are the workflows the merge queue runs, listed under
critical in the file below. They also run on main, after each push and nightly; each is judged on its latest run there,
names the jobs that failed in it, and is late when main has not run it for 36 hours. Over the last two weeks, main passing under 90%
of its runs, or one pull-request or merge-queue commit in ten needing a re-run to pass, also wants a look.
Nightly and scheduled jobs are judged on their latest scheduled run on main, and are late when it started more than one and a half of their periods ago (36 hours for a nightly). A job still queued after six hours almost always means no runner with its labels is online.
Run time charts one workflow's finished runs, oldest first: how long each took, from its start to its last update, and how long it waited to start. A failed run's point is red; hover a point for its commit, click it for the run.
Merging: the pull requests merged in the last two weeks and how long each was open, drafts included; and those open now.
To list another workflow, add it to
utils/kernel_checks/ci_workflows.json.
Kernels
Every kernel in the library, checked and timed nightly on npu1 and npu2.
Actions runs Kernel sourcesNeeds a look
Changed over recent nightlies
Reading the history…
Over recent nightlies
Cases
How these checks work
nightlyKernelChecks.yml runs at 06:00 UTC on an npu1 (AIE2) and an npu2 (AIE2P) bench runner,
on pull requests that bump the Peano pin, and on demand. Each run checks every case in
test/python/npu/kernel_cases.py for correctness
(test_kernels_e2e.py -m extensive: every case, every edge-data case, three random seeds), then times the cases that passed with
test_kernels_perf.py -m perf, and writes a catalogue of every factory and how its cases fared.
Only the nightly and an unfiltered dispatch on main publish, and only in the turbo power mode, which the bench runners set at boot: in any other mode
the timing step refuses rather than chart numbers taken at another clock. A failed preflight or passthrough sanity check publishes nothing
from that NPU; a failing case costs only its own series a point. A run that does not publish gets a report of its failures and of its changes against
the last nightly in its job summary (utils/kernel_checks/pr_report.py), kept as a comment on a Peano pull request.
What each case records
Cycles are the minimum over the traced calls of a kernel whose contract says Trace.whole_call(); their spread, and whether the trace
buffer held every call, are in the row's range. Cycles and object size are the numbers the checks gate on; host time moves with the machine,
so it only counts past three times its run-to-run spread. A dashed line on a chart marks a change of Peano, host, runtime or driver between nights.
The views
Summary: a verdict per NPU; what needs a look (failures, the cases that got slower or bigger last night, and any of the timing
harness's own tests that failed); and every case whose cycles or object size moved past its threshold between its first and latest nightly
of the last 30, as the two numbers and their dates.
All kernels: a line per kernel with, per NPU, its state last night, its cases' cycles now, and how many got slower or faster since
the first nightly shown; those that need a look first.
A kernel's own page (#view=kernel&kernel=<factory>) charts each of its cases' actual cycles (or size, or host time)
night by night, and tables each case's cycles now and then.
How to run the same checks locally, and to measure a kernel change against the code before it, is in the kernel library's testing section.
How these checks work
utils/component_checks/sa_placer_hw.py
runs in nightlyComponentChecks.yml at 06:30 UTC on an NPU2 bench runner. The
MobileNet design is placed with the SA placer at seeds 3
(the design's default), 2 and 7, then compiled, run and verified at batch 1, 4, 16 and 64, every image checked. A publishing run refuses to time
unless the NPU is in turbo mode.
What each seed and batch records
Per image and end-to-end are the fastest of 20 launches. Streaming, (T(N) − T(1)) ÷ (N − 1), is what one more image in a launch costs
with the launch's fixed cost taken out, so it is recorded above batch 1 only. A seed whose batches placed differently is flagged on its card.
To run one seed at one batch, from programming_examples/ml:
python -m mobilenet.aie2_mobilenet_iron --sa-seed N --sa-effort 1.0 --batch B
and every seed and batch, as the nightly does:
python utils/component_checks/sa_placer_hw.py --out perf.json --meta meta.json
Routing
The packet router against its model: does every design still route legally, and what routing costs.
Actions runs Router sourceTrends
How these checks work
modelBasedTests.yml runs at 03:17 UTC from main
and on demand, on a hosted runner with an assertions build. Each night takes a new range of seeds, so the router meets designs it has not seen;
the range is in the run's record, and a failure names the seeds to run again. The nightly and an unfiltered dispatch on main publish,
and a check that fails still publishes what it measured.
router_properties.py
generates designs whose routing the model can construct or rule out, on npu1, npu2 and xcvc1902, routes each with hops on and off,
and verifies the result: legal, deterministic, as short as the constructed routing, warned and rejected as the model predicts.
Per device it records the properties that failed, the routings that failed or were illegal, the hops over the constructed routing,
the arbiter msels per design and the time per design.
router_mutation.py
changes routed designs in ways with a known answer (the router must route it, reject it, or route it the same), and corrupts routed output
to measure how often the model's verifier notices. It records the cases that failed and the corruptions the verifier missed.
python test/create-packet-flows/nightly/router_properties.py --device npu2 --seeds 2000 --first-seed N python utils/router_testing/router_mutation.py --device npu2 [--hops-off] --seeds 400 --first-seed N
Placement
The SA tile placer and the buffer allocator: does every design still place, and how good the placements are.
Placer runs Placer source Allocator runs Allocator sourceTrends
How these checks work
nightlyComponentChecks.yml runs at 06:30 UTC from main,
on pull requests that touch the placer or these checks (the seed sweep only), and on demand. The nightly and an unfiltered dispatch on main publish;
a check that fails still publishes what it measured, and its record lists the failed seeds and runs.
Seed sweep (no hardware)
utils/component_checks/sa_placer.py, on a hosted runner:
aie-opt places two fixtures, the 4-core design in test/place-tiles/sa_placer/test_sa_effort.mlir and MobileNet before placement
(utils/component_checks/fixtures/mobilenet.mlir), with seeds 1 to 20 at full effort. Per fixture it records how many seeds failed to place,
the SA placer's final cost (mean and worst), and the CPU time and peak memory of a placement.
aie-opt '--aie-place-tiles=placer=sa_placer sa-seed=N sa-effort=1.0' \
--mlir-pass-statistics utils/component_checks/fixtures/mobilenet.mlir
MobileNet placements on hardware (NPU2)
utils/component_checks/sa_placer_hw.py:
MobileNet is placed with seeds 3 (the design's default), 2 and 7, then compiled, run and verified at batch 1, 4, 16 and 64, every image checked.
For each seed it records the placement aiecc chose (a digest of the placed tiles) and its SA cost, replayed with aie-opt.
What the placed design then costs to run is under Performance.
To rerun a whole check, run its script with --out perf.json --meta meta.json (the sweep also needs --aie-opt "$(which aie-opt)";
the hardware check takes --seeds and --batches). It prints a table with a row per seed, or per seed and batch.
Buffer allocator model-based tests (no hardware)
allocator_properties.py,
from modelBasedTests.yml at 03:17 UTC on a hosted runner:
aie-opt --aie-assign-buffer-addresses places 5,000 generated designs and 200,000 with per-bank reservations, each built around a layout
known to fit, so a design that does not place is a bug. check-aie runs the same test over the first 200 and 12,000. It records the properties
that failed, the designs left unplaced or placed illegally, the bank crossings a better placement avoids, the bank pairs cores share, and the run's wall time.
python test/assign-buffer-addresses/allocator_properties.py --seeds 5000 --bankres-seeds 200000
How to choose the placer and a seed for your own design is in section 1 of the programming guide.