.. include:: ../../../../shared/links.rst .. include:: ../links.rst ########################################################################## Checklist - xilinx-u250-gen3x16-xdma-4.1 ########################################################################## .. contents:: Table of content :depth: 1 :local: ******************************************************** Package information ******************************************************** .. table:: +--------------------------------------+------------------+ | xbtest information | +======================================+==================+ | Release Number | 3522814 | +--------------------------------------+------------------+ | Description (what's new) | Initial Release | +--------------------------------------+------------------+ .. table:: +--------------------------------------+-----------------------------------------------------+ | Platform information | +======================================+=====================================================+ | Development platform (XSA) | xilinx-u250-gen3x16-xdma-4.1-202210-1-dev-1-3512975 | +--------------------------------------+-----------------------------------------------------+ | Deployment platform (shell, CMC, SC) | xilinx-cmc-u200-u250-1.2.23-3395909 | | | | | | xilinx-sc-fw-u200-u250-4.6.20-1.28f0c61 | | | | | | xilinx-u250-gen3x16-base-4-3494623 | | | | | | xilinx-u250-gen3x16-xdma-shell-4.1-3494623 | | | | | | xilinx-u250-gen3x16-xdma-validate-4.1-3512975 | +--------------------------------------+-----------------------------------------------------+ ******************************************************** System information ******************************************************** .. table:: +------------------------+---------------------+ | System information | +========================+=====================+ | OS | CentOS / 7.8.2003 | +------------------------+---------------------+ | Architecture | x86_64 | +------------------------+---------------------+ | Server type | PowerEdge R740 | +------------------------+---------------------+ | AUX cable | Yes | +------------------------+---------------------+ | Card S/N | 21330497P03F | +------------------------+---------------------+ | XRT version | 2.13.449 | +------------------------+---------------------+ | Measured download time | 8.7 seconds | +------------------------+---------------------+ ******************************************************** Requirement - Platform high level features ******************************************************** ===================================================== Platform and card requirements ===================================================== .. table:: +--------------------------------------+-----------------------------------+ | Platform and card requirements | +======================================+===================================+ | FPGA part | xcu250-figd2104-2L-e | +--------------------------------------+-----------------------------------+ | PCIe speed and width | gen3x16 | +--------------------------------------+-----------------------------------+ | DMA type | xdma | +--------------------------------------+-----------------------------------+ | DDR: speed, size and quantity | 4 DDR: 72 bits @ 2400 MT/s; 16GB | +--------------------------------------+-----------------------------------+ | HBM: size, quantity and power rail | N/A | +--------------------------------------+-----------------------------------+ | Host memory support | Yes | +--------------------------------------+-----------------------------------+ | P2P support | Yes | +--------------------------------------+-----------------------------------+ | GTs: type, quad, SLR location | GT[0]: GTY, Quad_X1Y11 (SLR2) | | | | | | GT[1]: GTY, Quad_X1Y10 (SLR2) | | | | | | Both using refclk_b. | +--------------------------------------+-----------------------------------+ ===================================================== Power rails requirements ===================================================== .. table:: +-------------+-----------------+---------------------------------------------------------------------------+ | Power rail | Card limit (W) | Note | +=============+=================+===========================================================================+ | ``12v_pex`` | 69W | Permanent Connection | +-------------+-----------------+---------------------------------------------------------------------------+ | ``12v_aux`` | * 78W (AUX 4p) | User Selectable | | | * 156W (AUX 6p) | | +-------------+-----------------+---------------------------------------------------------------------------+ | ``vccint`` | N/A | Fatal Limit/Card Shutdown @53.5A per phase (6 phases with full AUX power) | +-------------+-----------------+---------------------------------------------------------------------------+ ******************************************************** Calibration - Power CU - Results ******************************************************** .. table:: +--------------------------+-------------------------------------------------------------------+----------------------------------------------------------------------------+ | Description | Results | +==========================+===================================================================+============================================================================+ | Power, temperature & toggle rate graph | .. figure:: ./attachments/power_calibration.png | | | :alt: | | | :align: center | +----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------+ | Maximum toggle rate reached (%) | 76 | +--------------------------+-------------------------------------------------------------------+----------------------------------------------------------------------------+ | Power consumption balance achieved: | No | | | For the 12V_PEX to draw more power, multiple GTs with QSFP heater modules. | | * Yes/no and justification (why are some rails not close to their critical current limit?) | Due to FPGA routing and resources, this is not possible. | +--------------------------+-------------------------------------------------------------------+----------------------------------------------------------------------------+ | | Measurement | 4.345 | | ``12v_pex`` current (A) +-------------------------------------------------------------------+----------------------------------------------------------------------------+ | | Critical limit | 5.75 | +--------------------------+-------------------------------------------------------------------+----------------------------------------------------------------------------+ | | Measurement | 12.828 | | ``12v_aux`` current (A) +-------------------------------------------------------------------+----------------------------------------------------------------------------+ | | Critical limit | 13.0 | +--------------------------+-------------------------------------------------------------------+----------------------------------------------------------------------------+ | | Measurement | 172.5 | | ``vccint`` current (A) +-------------------------------------------------------------------+----------------------------------------------------------------------------+ | | Critical limit | N/A | +--------------------------+-------------------------------------------------------------------+----------------------------------------------------------------------------+ ******************************************************** Calibration - Memory CU power ******************************************************** .. table:: +----------+-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | Memory | Description | ``simultaneous_wr_rd`` | ``only_rd`` | ``only_wr`` | +==========+=========================================+=======================================================================================================+=================================================================================+=================================================================================+ | DDR | Is there enough power for memory? | Yes. The U250 card is capable of supplying full power to the DDR4 memories. | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | Power graph | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_power_ddr.png | .. figure:: ./attachments/only_rd_rate_ramp_power_ddr.png | .. figure:: ./attachments/only_wr_rate_ramp_power_ddr.png | | | | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | BW graph | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_bw_ddr.png | .. figure:: ./attachments/only_rd_rate_ramp_bw_ddr.png | .. figure:: ./attachments/only_wr_rate_ramp_bw_ddr.png | | | | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | Latency graphs | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_latency_wr_ddr.png | .. figure:: ./attachments/only_rd_rate_ramp_latency_ddr.png | .. figure:: ./attachments/only_wr_rate_ramp_latency_ddr.png | | | | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | | | | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_latency_rd_ddr.png | | | | | | :alt: | | | | | | :align: center | | | | | | | | | +----------+-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ ******************************************************** Calibration - Memory bandwidth and latency ******************************************************** .. note:: * **N/A**: not applicable * .. [1] The outstanding write/read is the maximum number of outstanding reads just before the read pipeline is full, giving the best BW and latency results. .. table:: +-----------+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | Memory | Description | ``simultaneous_wr_rd`` | ``only_rd`` | ``only_wr`` | +===========+=========================================+=====================================================================================================================+===============================================================================================+===============================================================================================+ | DDR | QoS / power reduction rate | N/A | ``N/A`` | ``N/A`` | | | | | | | | | | .. figure:: ./attachments/simultaneous_wr_rd_qos_rate_ramp_bw_ddr.png | | | | | | :alt: | | | | | | :align: center | | | | | | | | | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best write BW (MBps) | 8487 | **N/A** | 15182 | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best write latency (ns) | 1614 | **N/A** | 802 | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best read BW (MBps) | 8487 | 17178 | **N/A** | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best read latency (ns) | 1608 | 451 | **N/A** | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Outstanding write/read [1]_ | Write = 4 / read = 4 | 3 | 4 | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | BW/latency graphs | .. figure:: ./attachments/simultaneous_wr_rd_outstanding_ramp_wr_ddr.png | .. figure:: ./attachments/only_rd_outstanding_ramp_ddr.png | .. figure:: ./attachments/only_wr_outstanding_ramp_ddr.png | | | (do one for read and one for write) | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | | | | .. figure:: ./attachments/simultaneous_wr_rd_outstanding_ramp_rd_ddr.png | | | | | | :alt: | | | | | | :align: center | | | | | | | | | +-----------+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ ******************************************************** Checklist questionnaire ******************************************************** ===================================================== Runtime ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+ | Have you updated the default download time? | Yes | | | | | * FYI: | .. code-block:: json | | | | | * u250 xclbin (90Mb) downloads in 3.9 sec (PCIe 3x16). | "download_time": 20 | | * HBM requires calibration which may take time. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ ===================================================== Sensors ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+ | Do you need to monitor more mechanical sources than the default one | No | | (``fpga_fan_1``)? | | | | | | * Yes, see |fill-platform-definition-json|_ | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Do you need to monitor more thermal sources than the default one (``fpga0``)? | No | | | | | * Yes, see |fill-platform-definition-json|_ | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Do you need to monitor more electrical sources than the default ones | No | | (``power_consumption``, ``12v_pex``, ``12v_aux``, ``3v3_pex`` and ``vccint``)? | | | | | | * Yes, see |fill-platform-definition-json|_ | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ ===================================================== GT ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+===========================================================+ | GT | GT[0] | GT[1] | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you updated GT default settings to maximize signal integrity? | .. code-block:: json | .. code-block:: json | | | | | | * Run all GT pre-canned tests. | "0": { | "1": { | | | "name": "top", | "name": "top", | | | "transceiver_settings": { | "transceiver_settings": { | | | "module": { | "module": { | | | "tx_differential_swing_control": 0, | "tx_differential_swing_control": 0, | | | "tx_pre_emphasis": 0, | "tx_pre_emphasis": 0, | | | "tx_post_emphasis": 0, | "tx_post_emphasis": 0, | | | "rx_equaliser": "DFE" | "rx_equaliser": "DFE" | | | }, | }, | | | "cable": { | "cable": { | | | "tx_differential_swing_control": 0, | "tx_differential_swing_control": 0, | | | "tx_pre_emphasis": 0, | "tx_pre_emphasis": 0, | | | "tx_post_emphasis": 0, | "tx_post_emphasis": 0, | | | "rx_equaliser": "DFE" | "rx_equaliser": "DFE" | | | } | } | | | } | } | | | } | } | | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ ===================================================== Memory ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+===========================================================+ | Memory | DDR | HOST | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined specific DMA bandwidths (``dma_bw``)? | .. code-block:: json | **N/A** | | | | | | * This not applicable for host memory. | "dma_bw": { | | | | "write": { | | | | "low": 7500 | | | | } | | | | } | | | | | | | | It's possible to increase DMA write performance | | | | by using a single DMA channel when concurrent buffers | | | | target a single DDR4 channel. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined optimum DMA settings (``buffer_size`` and ``total_size``) | No | **N/A** | | or are default settings enough? | | | | | | | | * Run dma pre-canned test. | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined specific P2P CARD bandwidths (``p2p_card_bw``)? | No | **N/A** | | | | | | * This not applicable for host memory. | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined optimum P2P CARD settings (``buffer_size`` and ``total_size``) | No | **N/A** | | or are default settings enough? | | | | | | | | * Run p2p_card pre-canned test. | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined specific P2P NVME bandwidths (``p2p_nvme_bw``)? | No | **N/A** | | | | | | * This not applicable for host memory. | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined optimum P2P NVME settings (``buffer_size`` and ``total_size``) | No | **N/A** | | or are default settings enough? | | | | | | | | * Run p2p_nvme pre-canned test. | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you reduced the CU rate to avoid over-power (``cu_rate``)? | No | No | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined ``cu_rate`` for simultaneous write/read bandwidth QoS? | No | .. code-block:: json | | | | | | | | "cu_rate": { | | | | "only_wr": { | | | | "write": { | | | | "nominal": 50 | | | | } | | | | } | | | | } | | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined ``cu_bw``? | .. code-block:: json | .. code-block:: json | | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | "cu_bw": { | "cu_bw": { | | | "only_wr": { | "only_wr": { | | For host memory these values are depending on the PCIe speed. | "write": { | "write": { | | | "average": 15182 | "high": 16000, | | | } | "low": 4000 | | | }, | } | | | "only_rd": { | }, | | | "read": { | "only_rd": { | | | "average": 17178 | "read": { | | | } | "high": 16000, | | | }, | "low": 4000 | | | "alt_wr_rd": { | } | | | "write": { | }, | | | "average": 8500 | "simul_wr_rd": { | | | }, | "write": { | | | "read": { | "high": 16000, | | | "average": 8500 | "low": 4000 | | | } | }, | | | }, | "read": { | | | "simul_wr_rd": { | "high": 16000, | | | "write": { | "low": 4000 | | | "average": 8487 | } | | | }, | } | | | "read": { | } | | | "average": 8487 | | | | } | | | | } | | | | } | | | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined ``cu_latency``? | .. code-block:: json | .. code-block:: json | | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | "cu_latency": { | "cu_latency": { | | | "only_rd": { | "only_wr": { | | | "read": { | "write": { | | | "high": 586, | "low": 1, | | | "low": 315 | "high": 6000 | | | } | } | | | }, | }, | | | "simul_wr_rd": { | "only_rd": { | | | "write": { | "read": { | | | "high": 2098, | "low": 1, | | | "low": 1129 | "high": 6000 | | | }, | } | | | "read": { | }, | | | "high": 2098, | "simul_wr_rd": { | | | "low": 1129 | "write": { | | | } | "low": 1, | | | }, | "high": 6000 | | | "only_wr": { | }, | | | "write": { | "read": { | | | "high": 1042, | "low": 1, | | | "low": 561 | "high": 6000 | | | } | } | | | } | } | | | } | } | | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined ``cu_outstanding``? | .. code-block:: json | No | | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | "cu_outstanding": { | | | | "only_wr": { | | | | "write": { | | | | "nominal": 4 | | | | } | | | | }, | | | | "only_rd": { | | | | "read": { | | | | "nominal": 3 | | | | } | | | | }, | | | | "simul_wr_rd": { | | | | "write": { | | | | "nominal": 4 | | | | }, | | | | "read": { | | | | "nominal": 4 | | | | } | | | | } | | | | } | | | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ | Have you defined ``cu_burst_size``? | No | No | | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+-----------------------------------------------------------+ ===================================================== Pre-canned tests ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+ | Have you tested all provided pre-canned tests? | Yes | | | | | * If not: list the skipped one and the reason? | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you modified the pre-canned test template? | No | | | | | * If yes: | | | | | | * Why?  | | | * Was it not possible to make a platform definition update | | | (and thus, available for any test as default)? | | | * Update next section with your modifications. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you saved all pre-canned results? | Yes | | | | | * Update package information section of your checklist. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ ===================================================== Issues tracking ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+ | Did you modify the platform definition limits and range after running the | No | | pre-canned tests? | | | | | | * If yes: | | | | | | * List the problem tracker reference number | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ ******************************************************** Pre-canned tests results ******************************************************** ===================================================== Pre-canned test modifications ===================================================== .. table:: +-----------+-------------------------------------+----------------+ | Test name | Modification from original template | Justifications | +===========+=====================================+================+ | / | No modification | / | +-----------+-------------------------------------+----------------+ ===================================================== ``dma`` pre-canned test ===================================================== .. table:: +-----------------+-----------------+-----------------+ | Memory | Write BW (MBps) | Read BW (MBps) | +=================+=================+=================+ | DDR | 8089 | 10393 | +-----------------+-----------------+-----------------+ ===================================================== ``p2p_card`` pre-canned test ===================================================== .. table:: +-----------------+-----------------+-----------------+ | Memory | Write BW (MBps) | Read BW (MBps) | +=================+=================+=================+ | DDR | 6258 | 4762 | +-----------------+-----------------+-----------------+ **p2p_card test was ran on R7525 server** ===================================================== ``p2p_nvme`` pre-canned test ===================================================== .. table:: +-----------------+-----------+-----------------+-----------------+ | Memory | Card mode | Write BW (MBps) | Read BW (MBps) | +=================+===========+=================+=================+ | DDR | source | 2451 | 5783 | +-----------------+-----------+-----------------+-----------------+ | DDR | target | 5972 | 2509 | +-----------------+-----------+-----------------+-----------------+ **p2p_nvme test was ran on R7525 server** ===================================================== ``memory`` pre-canned test ===================================================== .. table:: +-----------------+------------------------+-----------------+-----------------+--------------------+-------------------+ | Memory | Test mode | Write BW (MBps) | Read BW (MBps) | Write Latency (ns) | Read Latency (ns) | +=================+========================+=================+=================+====================+===================+ | | ``alternate_wr_rd`` | 7853 | 8292 | 802 | 452 | + +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``only_wr`` | 15182 | **N/A** | 802 | **N/A** | + DDR +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``only_rd`` | **N/A** | 17178 | **N/A** | 451 | + +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``simultaneous_wr_rd`` | 8487 | 8487 | 1614 | 1608 | +-----------------+------------------------+-----------------+-----------------+--------------------+-------------------+ ===================================================== ``memory_host`` pre-canned test ===================================================== .. table:: +-----------------+------------------------+-----------------+-----------------+--------------------+-------------------+ | Memory | Test mode | Write BW (MBps) | Read BW (MBps) | Write Latency (ns) | Read Latency (ns) | +=================+========================+=================+=================+====================+===================+ | | ``alternate_wr_rd`` | 5232 | 5217 | 881 | 3835 | + +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``only_wr`` | 9155 | **N/A** | 881 | **N/A** | + HOST +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``only_rd`` | **N/A** | 12177 | **N/A** | 3835 | + +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``simultaneous_wr_rd`` | 9155 | 11817 | 882 | 3952 | +-----------------+------------------------+-----------------+-----------------+--------------------+-------------------+