.. include:: ../../../../shared/links.rst .. include:: ../links.rst ########################################################################## Checklist - xilinx-u50lv-gen3x4-xdma-base-2 ########################################################################## .. contents:: Table of content :depth: 1 :local: ******************************************************** Package information ******************************************************** .. table:: +--------------------------------------+------------------+ | xbtest information | +======================================+==================+ | Release number | 3522863 | +--------------------------------------+------------------+ | Description (what's new) | Initial release | +--------------------------------------+------------------+ .. table:: +--------------------------------------+---------------------------------------------------+ | Platform information | +======================================+===================================================+ | Development platform (XSA) | xilinx-u50lv-gen3x4-xdma-2-202010-1-dev-1-2902115 | +--------------------------------------+---------------------------------------------------+ | Deployment platform (shell, CMC, SC) | xilinx-cmc-u50-1.0.40-3398385 | | | | | | xilinx-sc-fw-u50-5.2.18-1.bf9ba46 | | | | | | xilinx-u50lv-gen3x4-xdma-base-2-2902115 | | | | | | xilinx-u50lv-gen3x4-xdma-validate-2-2902115 | +--------------------------------------+---------------------------------------------------+ ******************************************************** System information ******************************************************** .. table:: +------------------------+-------------------------+ | System information | +========================+=========================+ | OS | RedHatEnterprise / 8.3 | +------------------------+-------------------------+ | Architecture | x86_64 | +------------------------+-------------------------+ | Server type | PowerEdge R740 | +------------------------+-------------------------+ | AUX cable | N/A | +------------------------+-------------------------+ | Card S/N | 501211101A11 | +------------------------+-------------------------+ | XRT version | 2.13.425 | +------------------------+-------------------------+ | Measured download time | 10.4 seconds | +------------------------+-------------------------+ ******************************************************** Requirement - Platform high level features ******************************************************** ===================================================== Platform and card requirements ===================================================== .. table:: +--------------------------------------+-------------------------------------+ | Platform and card requirements | +======================================+=====================================+ | FPGA part | xcu50-fsvh2104-2LV-e | +--------------------------------------+-------------------------------------+ | PCIe speed and width | gen3x4 | +--------------------------------------+-------------------------------------+ | DMA type | xdma | +--------------------------------------+-------------------------------------+ | DDR: speed, size and quantity | not available | +--------------------------------------+-------------------------------------+ | HBM: size, quantity and power rail | 32 Pseudo-Channels; 8192MB; 3v3_pex | +--------------------------------------+-------------------------------------+ | Host memory support | not available | +--------------------------------------+-------------------------------------+ | P2P support | not available | +--------------------------------------+-------------------------------------+ | GTs: type, quad, SLR location | GT[0]: GTY, Quad_X0Y7 (SLR1) | +--------------------------------------+-------------------------------------+ ===================================================== Power rails requirements ===================================================== .. table:: +-------------+----------------+-----------------------+ | Power rail | Card limit (W) | Note | +=============+================+=======================+ | ``12v_pex`` | 65W | | +-------------+----------------+-----------------------+ | ``3v3_pex`` | 11W | HBM only power source | +-------------+----------------+-----------------------+ ******************************************************** Calibration - Power CU - Results ******************************************************** .. table:: +--------------------------+-------------------------------------------------------------------+------------------------------------------------------------------------------------+ | Description | Results | +==========================+===================================================================+====================================================================================+ | Power, temperature & toggle rate graph | .. figure:: ./attachments/power_calibration.png | | | :alt: | | | :align: center | +--------------------------+-------------------------------------------------------------------+------------------------------------------------------------------------------------+ | Maximum toggle rate reached (%) | 83 | +--------------------------+-------------------------------------------------------------------+------------------------------------------------------------------------------------+ | Power consumption balance achieved: | No: | | | | | * Yes/no and justification (why are some rails not close to their critical current limit?) | * HBM is powered exclusively from the 3v3_pex and user logic is on the 12v_pex. | | | So the power CU has no effect on the 3v3_pex | | | and thus 3v3_pex current is not close to its critical current limit. | | | | +--------------------------+-------------------------------------------------------------------+------------------------------------------------------------------------------------+ | | Measurement | 0.637A | | ``3v3_pex`` current (A) +-------------------------------------------------------------------+------------------------------------------------------------------------------------+ | | Critical limit | 3.3A (11W) | +--------------------------+-------------------------------------------------------------------+------------------------------------------------------------------------------------+ | | Measurement | 5.148A | | ``12v_pex`` current (A) +-------------------------------------------------------------------+------------------------------------------------------------------------------------+ | | Critical limit | 5.4A (65W) | +--------------------------+-------------------------------------------------------------------+------------------------------------------------------------------------------------+ ******************************************************** Calibration - Memory CU power ******************************************************** .. table:: +----------+-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | Memory | Description | ``simultaneous_wr_rd`` | ``only_rd`` | ``only_wr`` | +==========+=========================================+=======================================================================================================+=================================================================================+=================================================================================+ | HBM | Is there enough power for memory? | No, HBM is powered from the 3v3_pex power rail. | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | Power rail name | 3v3_pex | 3v3_pex | 3v3_pex | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | Power throttle limit (W) | 10 | 10 | 10 | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | 20% below power throttle limit (W) | 8 | 8 | 8 | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | CU rate (%) @ 20% below limit | 26 | 50 | 57 | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | Power graph | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_power_hbm.png | .. figure:: ./attachments/only_rd_rate_ramp_power_hbm.png | .. figure:: ./attachments/only_wr_rate_ramp_power_hbm.png | | | | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | BW graph | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_bw_hbm.png | .. figure:: ./attachments/only_rd_rate_ramp_bw_hbm.png | .. figure:: ./attachments/only_wr_rate_ramp_bw_hbm.png | | | | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | Latency graphs | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_latency_wr_hbm.png | .. figure:: ./attachments/only_rd_rate_ramp_latency_hbm.png | .. figure:: ./attachments/only_wr_rate_ramp_latency_hbm.png | | | | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | | | | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_latency_rd_hbm.png | | | | | | :alt: | | | | | | :align: center | | | | | | | | | +----------+-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ ******************************************************** Calibration - Memory bandwidth and latency ******************************************************** .. note:: * **N/A**: not applicable * .. [1] The outstanding write/read is the maximum number of outstanding reads just before the read pipeline is full, giving the best BW and latency results. .. table:: +-----------+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | Memory | Description | ``simultaneous_wr_rd`` | ``only_rd`` | ``only_wr`` | +===========+=========================================+=====================================================================================================================+===============================================================================================+===============================================================================================+ | HBM | QoS / power reduction rate | N/A | N/A | N/A | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best write BW (MBps) | 2026 | **N/A** | 4410 | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best write latency (ns) | 80 | **N/A** | 80 | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best read BW (MBps) | 2026 | 3814 | **N/A** | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best read latency (ns) | 208 | 199 | **N/A** | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Outstanding write/read [1]_ | Write = 2 / read = 4 | 2 | 2 | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | BW/latency graphs | .. figure:: ./attachments/simultaneous_wr_rd_outstanding_ramp_wr_hbm.png | .. figure:: ./attachments/only_rd_outstanding_ramp_hbm.png | .. figure:: ./attachments/only_wr_outstanding_ramp_hbm.png | | | (do one for read and one for write) | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | | | | .. figure:: ./attachments/simultaneous_wr_rd_outstanding_ramp_rd_hbm.png | | | | | | :alt: | | | | | | :align: center | | | | | | | | | +-----------+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ ******************************************************** Checklist questionnaire ******************************************************** ===================================================== Runtime ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+ | Have you updated the default download time? | Yes, HBM requires calibration which may take time. | | | | | * FYI: | .. code-block:: json | | | | | * u250 xclbin (90Mb) downloads in 3.9 sec (PCIe 3x16). | "download_time": 20 | | * HBM requires calibration which may take time. | | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ ===================================================== Sensors ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+ | Do you need to monitor more mechanical sources than the default one | No | | (``fpga_fan_1``)? | | | | | | * Yes, see |fill-platform-definition-json|_ | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Do you need to monitor more thermal sources than the default one (``fpga0``)? | No | | | | | * Yes, see |fill-platform-definition-json|_ | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Do you need to monitor more electrical sources than the default ones | No | | (``power_consumption``, ``12v_pex``, ``12v_aux``, ``3v3_pex`` and ``vccint``)? | | | | | | * Yes, see |fill-platform-definition-json|_ | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ ===================================================== GT ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+ | GT | GT[0] | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you updated GT default settings to maximize signal integrity? | .. code-block:: json | | | | | * Run all GT pre-canned tests. | "0": { | | | "name": "top", | | | "transceiver_settings": { | | | "module": { | | | "tx_differential_swing_control": 11, | | | "tx_pre_emphasis": 0, | | | "tx_post_emphasis": 0, | | | "rx_equaliser": "DFE" | | | }, | | | "cable": { | | | "tx_differential_swing_control": 11, | | | "tx_pre_emphasis": 0, | | | "tx_post_emphasis": 0, | | | "rx_equaliser": "DFE" | | | } | | | } | | | } | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ ===================================================== Memory ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+ | Memory | HBM | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined specific DMA bandwidths (``dma_bw``)? | No | | | | | * This not applicable for host memory. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined optimum DMA settings (``buffer_size`` and ``total_size``) |.. code-block:: json | | or are default settings enough? | | | | "dma_config": { | | * Run dma pre-canned test. | "buffer_size": 128 | | | } | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined specific P2P CARD bandwidths (``p2p_card_bw``)? | N/A | | | | | * This not applicable for host memory. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined optimum P2P CARD settings (``buffer_size`` and ``total_size``) | N/A | | or are default settings enough? | | | | | | * Run p2p_card pre-canned test. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined specific P2P NVME bandwidths (``p2p_nvme_bw``)? | N/A | | | | | * This not applicable for host memory. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined optimum P2P NVME settings (``buffer_size`` and ``total_size``) | N/A | | or are default settings enough? | | | | | | * Run p2p_nvme pre-canned test. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you reduced the CU rate to avoid over-power (``cu_rate``)? | .. code-block:: json | | | | | | "cu_rate": { | | | "only_wr": { | | | "write": { | | | "nominal": 57 | | | } | | | }, | | | "only_rd": { | | | "read": { | | | "nominal": 50 | | | } | | | }, | | | "simul_wr_rd": { | | | "write": { | | | "nominal": 26 | | | }, | | | "read": { | | | "nominal": 26 | | | } | | | } | | | } | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined ``cu_rate`` for simultaneous write/read bandwidth QoS? | No | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined ``cu_bw``? | .. code-block:: json | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | "cu_bw": { | | | "only_wr": { | | For host memory these values are depending on the PCIe speed. | "write": { | | | "average": 4410 | | | } | | | }, | | | "only_rd": { | | | "read": { | | | "average": 3814 | | | } | | | }, | | | "simul_wr_rd": { | | | "write": { | | | "average": 2026 | | | }, | | | "read": { | | | "average": 2026 | | | } | | | } | | | } | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined ``cu_latency``? | .. code-block:: json | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | "cu_latency": { | | | "only_wr": { | | | "write": { | | | "high": 156, | | | "low": 56 | | | } | | | }, | | | "only_rd": { | | | "read": { | | | "high": 319, | | | "low": 138 | | | } | | | }, | | | "simul_wr_rd": { | | | "write": { | | | "high": 156, | | | "low": 56 | | | }, | | | "read": { | | | "high": 342, | | | "low": 138 | | | } | | | } | | | } | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined ``cu_outstanding``? | .. code-block:: json | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | "cu_outstanding": { | | | "only_wr": { | | | "write": { | | | "nominal": 2 | | | } | | | }, | | | "only_rd": { | | | "read": { | | | "nominal": 2 | | | } | | | }, | | | "simul_wr_rd": { | | | "write": { | | | "nominal": 2 | | | }, | | | "read": { | | | "nominal": 4 | | | } | | | } | | | } | | | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you defined ``cu_burst_size``? | No | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ ===================================================== Pre-canned tests ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+ | Have you tested all provided pre-canned tests? | Yes | | | | | * If not: list the skipped one and the reason? | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you modified the pre-canned test template? | No | | | | | * If yes: | | | | | | * Why?  | | | * Was it not possible to make a platform definition update | | | (and thus, available for any test as default)? | | | * Update next section with your modifications. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Have you saved all pre-canned results? | Yes | | | | | * Update package information section of your checklist. | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ ===================================================== Issues tracking ===================================================== .. table:: +---------------------------------------------------------------------------------+-----------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+===========================================================+ | Did you modify the platform definition limits and range after running the | No | | pre-canned tests? | | | | | | * If yes: | | | | | | * List the problem tracker reference number | | +---------------------------------------------------------------------------------+-----------------------------------------------------------+ ******************************************************** Pre-canned tests results ******************************************************** ===================================================== Pre-canned test modifications ===================================================== .. table:: +-----------+-------------------------------------+----------------+ | Test name | Modification from original template | Justifications | +===========+=====================================+================+ | / | No modification | / | +-----------+-------------------------------------+----------------+ ===================================================== ``dma`` pre-canned test ===================================================== .. table:: +-----------------+-----------------+-----------------+ | Memory | Write BW (MBps) | Read BW (MBps) | +=================+=================+=================+ | HBM | 3294 | 3305 | +-----------------+-----------------+-----------------+ ===================================================== ``p2p_nvme`` pre-canned test ===================================================== **not applicable** ===================================================== ``p2p_card`` pre-canned test ===================================================== **not applicable** ===================================================== ``memory`` pre-canned test ===================================================== .. table:: +-----------------+------------------------+-----------------+-----------------+--------------------+-------------------+ | Memory | Test mode | Write BW (MBps) | Read BW (MBps) | Write Latency (ns) | Read Latency (ns) | +=================+========================+=================+=================+====================+===================+ | | ``alternate_wr_rd`` | 65369 | 65369 | 81 | 200 | + +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``only_wr`` | 141145 | **N/A** | 81 | **N/A** | + HBM +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``only_rd`` | **N/A** | 122072 | **N/A** | 200 | + +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``simultaneous_wr_rd`` | 64851 | 64851 | 81 | 212 | +-----------------+------------------------+-----------------+-----------------+--------------------+-------------------+ ===================================================== ``memory_host`` pre-canned test ===================================================== **not applicable**