Checklist - xilinx-u250-gen3x16-xdma-4.1¶

Table of content

Package information
System information
Requirement - Platform high level features
Calibration - Power CU - Results
Calibration - Memory CU power
Calibration - Memory bandwidth and latency
Checklist questionnaire
Pre-canned tests results

Package information ¶

xbtest information
Release Number	3522814
Description (what’s new)	Initial Release

Platform information

Development platform (XSA)

xilinx-u250-gen3x16-xdma-4.1-202210-1-dev-1-3512975

Deployment platform (shell, CMC, SC)

xilinx-cmc-u200-u250-1.2.23-3395909

xilinx-sc-fw-u200-u250-4.6.20-1.28f0c61

xilinx-u250-gen3x16-base-4-3494623

xilinx-u250-gen3x16-xdma-shell-4.1-3494623

xilinx-u250-gen3x16-xdma-validate-4.1-3512975

System information ¶

System information
OS	CentOS / 7.8.2003
Architecture	x86_64
Server type	PowerEdge R740
AUX cable	Yes
Card S/N	21330497P03F
XRT version	2.13.449
Measured download time	8.7 seconds

Requirement - Platform high level features ¶

Platform and card requirements¶

Platform and card requirements
FPGA part	xcu250-figd2104-2L-e
PCIe speed and width	gen3x16
DMA type	xdma
DDR: speed, size and quantity	4 DDR: 72 bits @ 2400 MT/s; 16GB
HBM: size, quantity and power rail	N/A
Host memory support	Yes
P2P support	Yes
GTs: type, quad, SLR location	GT[0]: GTY, Quad_X1Y11 (SLR2) GT[1]: GTY, Quad_X1Y10 (SLR2) Both using refclk_b.

Power rails requirements¶

Power rail	Card limit (W)	Note
`12v_pex`	69W	Permanent Connection
`12v_aux`	78W (AUX 4p) 156W (AUX 6p)	User Selectable
`vccint`	N/A	Fatal Limit/Card Shutdown @53.5A per phase (6 phases with full AUX power)

Calibration - Power CU - Results ¶

Description		Results
Power, temperature & toggle rate graph
Maximum toggle rate reached (%)		76
Power consumption balance achieved: Yes/no and justification (why are some rails not close to their critical current limit?)		No For the 12V_PEX to draw more power, multiple GTs with QSFP heater modules. Due to FPGA routing and resources, this is not possible.
`12v_pex` current (A)	Measurement	4.345
`12v_pex` current (A)	Critical limit	5.75
`12v_aux` current (A)	Measurement	12.828
`12v_aux` current (A)	Critical limit	13.0
`vccint` current (A)	Measurement	172.5
`vccint` current (A)	Critical limit	N/A

Calibration - Memory CU power ¶

Memory	Description	`simultaneous_wr_rd`
DDR	Is there enough power for memory?	Yes. The U250 card is capable of supplying full power to the DDR4 memories.
	Power graph
	BW graph
	Latency graphs

Calibration - Memory bandwidth and latency ¶

Note

N/A: not applicable
1

The outstanding write/read is the maximum number of outstanding reads just before the read pipeline is full, giving the best BW and latency results.

Memory	Description	`simultaneous_wr_rd`	`only_rd`	`only_wr`
DDR	QoS / power reduction rate	N/A	`N/A`	`N/A`
	Best write BW (MBps)	8487	N/A	15182
	Best write latency (ns)	1614	N/A	802
	Best read BW (MBps)	8487	17178	N/A
	Best read latency (ns)	1608	451	N/A
	Outstanding write/read 1	Write = 4 / read = 4	3	4
	BW/latency graphs (do one for read and one for write)

Checklist questionnaire ¶

Runtime¶

Question

Answer / justification / problem tracker reference number

Have you updated the default download time?

FYI:

u250 xclbin (90Mb) downloads in 3.9 sec (PCIe 3x16).

HBM requires calibration which may take time.

Yes

"download_time": 20

Sensors¶

Question

Answer / justification / problem tracker reference number

Do you need to monitor more mechanical sources than the default one (fpga_fan_1)?

Yes, see Fill platform definition JSON

No

Do you need to monitor more thermal sources than the default one (fpga0)?

Yes, see Fill platform definition JSON

No

Do you need to monitor more electrical sources than the default ones (power_consumption, 12v_pex, 12v_aux, 3v3_pex and vccint)?

Yes, see Fill platform definition JSON

No

GT¶

Question

Answer / justification / problem tracker reference number

GT

GT[0]

GT[1]

Have you updated GT default settings to maximize signal integrity?

Run all GT pre-canned tests.

"0": {
    "name": "top",
    "transceiver_settings": {
        "module": {
            "tx_differential_swing_control": 0,
            "tx_pre_emphasis": 0,
            "tx_post_emphasis": 0,
            "rx_equaliser": "DFE"
        },
        "cable": {
            "tx_differential_swing_control": 0,
            "tx_pre_emphasis": 0,
            "tx_post_emphasis": 0,
            "rx_equaliser": "DFE"
        }
    }
}

"1": {
    "name": "top",
    "transceiver_settings": {
        "module": {
            "tx_differential_swing_control": 0,
            "tx_pre_emphasis": 0,
            "tx_post_emphasis": 0,
            "rx_equaliser": "DFE"
        },
        "cable": {
            "tx_differential_swing_control": 0,
            "tx_pre_emphasis": 0,
            "tx_post_emphasis": 0,
            "rx_equaliser": "DFE"
        }
    }
}

Memory¶

Question	Answer / justification / problem tracker reference number
Memory	DDR	HOST
Have you defined specific DMA bandwidths (`dma_bw`)? This not applicable for host memory.	"dma_bw": { "write": { "low": 7500 } } It’s possible to increase DMA write performance by using a single DMA channel when concurrent buffers target a single DDR4 channel.	N/A
Have you defined optimum DMA settings (`buffer_size` and `total_size`) or are default settings enough? Run dma pre-canned test.	No	N/A
Have you defined specific P2P CARD bandwidths (`p2p_card_bw`)? This not applicable for host memory.	No	N/A
Have you defined optimum P2P CARD settings (`buffer_size` and `total_size`) or are default settings enough? Run p2p_card pre-canned test.	No	N/A
Have you defined specific P2P NVME bandwidths (`p2p_nvme_bw`)? This not applicable for host memory.	No	N/A
Have you defined optimum P2P NVME settings (`buffer_size` and `total_size`) or are default settings enough? Run p2p_nvme pre-canned test.	No	N/A
Have you reduced the CU rate to avoid over-power (`cu_rate`)?	No	No
Have you defined `cu_rate` for simultaneous write/read bandwidth QoS?	No	"cu_rate": { "only_wr": { "write": { "nominal": 50 } } }
Have you defined `cu_bw`? For each test mode: `only_wr`, `only_rd` and `simul_wr_rd` For host memory these values are depending on the PCIe speed.	"cu_bw": { "only_wr": { "write": { "average": 15182 } }, "only_rd": { "read": { "average": 17178 } }, "alt_wr_rd": { "write": { "average": 8500 }, "read": { "average": 8500 } }, "simul_wr_rd": { "write": { "average": 8487 }, "read": { "average": 8487 } } }	"cu_bw": { "only_wr": { "write": { "high": 16000, "low": 4000 } }, "only_rd": { "read": { "high": 16000, "low": 4000 } }, "simul_wr_rd": { "write": { "high": 16000, "low": 4000 }, "read": { "high": 16000, "low": 4000 } } }
Have you defined `cu_latency`? For each test mode: `only_wr`, `only_rd` and `simul_wr_rd`	"cu_latency": { "only_rd": { "read": { "high": 586, "low": 315 } }, "simul_wr_rd": { "write": { "high": 2098, "low": 1129 }, "read": { "high": 2098, "low": 1129 } }, "only_wr": { "write": { "high": 1042, "low": 561 } } }	"cu_latency": { "only_wr": { "write": { "low": 1, "high": 6000 } }, "only_rd": { "read": { "low": 1, "high": 6000 } }, "simul_wr_rd": { "write": { "low": 1, "high": 6000 }, "read": { "low": 1, "high": 6000 } } }
Have you defined `cu_outstanding`? For each test mode: `only_wr`, `only_rd` and `simul_wr_rd`	"cu_outstanding": { "only_wr": { "write": { "nominal": 4 } }, "only_rd": { "read": { "nominal": 3 } }, "simul_wr_rd": { "write": { "nominal": 4 }, "read": { "nominal": 4 } } }	No
Have you defined `cu_burst_size`? For each test mode: `only_wr`, `only_rd` and `simul_wr_rd`	No	No

Pre-canned tests¶

Question

Answer / justification / problem tracker reference number

Have you tested all provided pre-canned tests?

If not: list the skipped one and the reason?

Yes

Have you modified the pre-canned test template?

If yes:

Why?

Was it not possible to make a platform definition update (and thus, available for any test as default)?

Update next section with your modifications.

No

Have you saved all pre-canned results?

Update package information section of your checklist.

Yes

Issues tracking¶

Question

Answer / justification / problem tracker reference number

Did you modify the platform definition limits and range after running the pre-canned tests?

If yes:

List the problem tracker reference number

No

Pre-canned tests results ¶

Pre-canned test modifications¶

Test name	Modification from original template	Justifications
/	No modification	/

`dma` pre-canned test¶

Memory	Write BW (MBps)	Read BW (MBps)
DDR	8089	10393

`p2p_card` pre-canned test¶

Memory	Write BW (MBps)	Read BW (MBps)
DDR	6258	4762

p2p_card test was ran on R7525 server

`p2p_nvme` pre-canned test¶

Memory	Card mode	Write BW (MBps)	Read BW (MBps)
DDR	source	2451	5783
DDR	target	5972	2509

p2p_nvme test was ran on R7525 server

`memory` pre-canned test¶

Memory	Test mode	Write BW (MBps)	Read BW (MBps)	Write Latency (ns)	Read Latency (ns)
DDR	`alternate_wr_rd`	7853	8292	802	452
	`only_wr`	15182	N/A	802	N/A
	`only_rd`	N/A	17178	N/A	451
	`simultaneous_wr_rd`	8487	8487	1614	1608

`memory_host` pre-canned test¶

Memory	Test mode	Write BW (MBps)	Read BW (MBps)	Write Latency (ns)	Read Latency (ns)
HOST	`alternate_wr_rd`	5232	5217	881	3835
	`only_wr`	9155	N/A	881	N/A
	`only_rd`	N/A	12177	N/A	3835
	`simultaneous_wr_rd`	9155	11817	882	3952

Checklist - xilinx-u250-gen3x16-xdma-4.1¶

Platform and card requirements¶

Power rails requirements¶

Runtime¶

Sensors¶

GT¶

Memory¶

Pre-canned tests¶

Issues tracking¶

Pre-canned test modifications¶

dma pre-canned test¶

p2p_card pre-canned test¶

p2p_nvme pre-canned test¶

memory pre-canned test¶

memory_host pre-canned test¶

`dma` pre-canned test¶

`p2p_card` pre-canned test¶

`p2p_nvme` pre-canned test¶

`memory` pre-canned test¶

`memory_host` pre-canned test¶