.. include:: ../../../../shared/links.rst .. include:: ../links.rst ########################################################################## Checklist - xilinx-u55c-gen3x16-xdma-3 ########################################################################## .. contents:: Table of content :depth: 1 :local: ******************************************************** Package information ******************************************************** .. table:: +--------------------------------------+-----------------+ | xbtest information | +======================================+=================+ | Release Number | 3522912 | +--------------------------------------+-----------------+ | Description (what's new) | Initial Release | +--------------------------------------+-----------------+ .. table:: +--------------------------------------+---------------------------------------------------+ | Platform information | +======================================+===================================================+ | Development platform (XSA) | xilinx-u55c-gen3x16-xdma-3-202210-1-dev-1-3514517 | +--------------------------------------+---------------------------------------------------+ | Deployment platform (shell, CMC, SC) | xilinx-cmc-u55-1.5.25-3395704 | | | | | | xilinx-sc-fw-u55-7.1.17-1.f7275be | | | | | | xilinx-u55c-gen3x16-xdma-base-3-3494559 | | | | | | xilinx-u55c-gen3x16-xdma-validate-3-3506150 | +--------------------------------------+---------------------------------------------------+ ******************************************************** System information ******************************************************** .. table:: +------------------------+--------------------+ | System information | +========================+====================+ | OS | CentOS / 7.8.2003 | +------------------------+--------------------+ | Architecture | x86_64 | +------------------------+--------------------+ | Server type | PowerEdge R740 | +------------------------+--------------------+ | AUX cable | yes | +------------------------+--------------------+ | Card S/N | XFL1BO3WZCH5 | +------------------------+--------------------+ | XRT version | 2.13.449 | +------------------------+--------------------+ | Measured download time | 10.5 seconds | +------------------------+--------------------+ ******************************************************** Requirement - Platform high level features ******************************************************** ===================================================== Platform and card requirements ===================================================== .. table:: +--------------------------------------+---------------------------------------+ | Platform and card requirements | +======================================+=======================================+ | FPGA part | xcu55c-fsvh2892-2L-e | +--------------------------------------+---------------------------------------+ | PCIe speed and width | gen3x16 | +--------------------------------------+---------------------------------------+ | DMA type | xdma | +--------------------------------------+---------------------------------------+ | DDR: speed, size and quantity | N/A | +--------------------------------------+---------------------------------------+ | HBM: size, quantity and power rail | 32 Pseudo-Channels; 16,384MB, 12v_pex | +--------------------------------------+---------------------------------------+ | Host memory support | yes | +--------------------------------------+---------------------------------------+ | P2P support | yes | +--------------------------------------+---------------------------------------+ | GTs: type, quad, SLR location | GT[0]: GTY, Quad_X0Y6 (SLR1) | | | | | | GT[1]: GTY, Quad_X0Y7 (SLR1) | +--------------------------------------+---------------------------------------+ ===================================================== Power rails requirements ===================================================== .. table:: +-------------+----------------+-------------------------------------------------------------------------+ | Power rail | Card limit (W) | Note | +=============+================+=========================================================================+ | ``12v_pex`` | 69W | Powers HBM | +-------------+----------------+-------------------------------------------------------------------------+ | ``3v3_pex`` | 10.4W | | +-------------+----------------+-------------------------------------------------------------------------+ | ``12v_aux`` | 78W | User Selectable | +-------------+----------------+-------------------------------------------------------------------------+ | ``vccint`` | N/A | Fatal Limit/Card Shutdown @60A per phase (8 phases with full AUX power) | +-------------+----------------+-------------------------------------------------------------------------+ ******************************************************** Calibration - Power CU - Results ******************************************************** .. table:: +--------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ | Description | Results | +==========================+===================================================================+=============================================================================================+ | Power, temperature & toggle rate graph | .. figure:: ./attachments/power_calibration.png | | | :alt: | | | :align: center | +--------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ | Maximum toggle rate reached (%) | 84 | | | | | | Board reboot due to 12v_aux threshold. Last measured power at the 12v_aux was :78.568776 W | +--------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ | Power consumption balance achieved: | No | | | | | * Yes/no and justification (why are some rails not close to their critical current limit?) | Historically, it is difficult to reach maximum 150W limit for u55c. | +--------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ | | Measurement | 1.577 | | ``3v3_pex`` current (A) +-------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ | | Critical limit | 3.15 | +--------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ | | Measurement | 4.264 | | ``12v_pex`` current (A) +-------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ | | Critical limit | 5.75 | +--------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ | | Measurement | 6.457 | | ``12v_aux`` current (A) +-------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ | | Critical limit | 6.5 | +--------------------------+-------------------------------------------------------------------+---------------------------------------------------------------------------------------------+ ******************************************************** Calibration - Memory CU power ******************************************************** .. table:: +----------+-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | Memory | Description | ``simultaneous_wr_rd`` | ``only_rd`` | ``only_wr`` | +==========+=========================================+=======================================================================================================+=================================================================================+=================================================================================+ | HBM |Is there enough power for memory? | Yes. The U55C card is capable of supplying full power to the HBM memories. | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | Power graph | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_power_hbm.png | .. figure:: ./attachments/only_rd_rate_ramp_power_hbm.png | .. figure:: ./attachments/only_wr_rate_ramp_power_hbm.png | | | | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | BW graph | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_bw_hbm.png | .. figure:: ./attachments/only_rd_rate_ramp_bw_hbm.png | .. figure:: ./attachments/only_wr_rate_ramp_bw_hbm.png | | | | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | + +-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ | | Latency graphs | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_latency_wr_hbm.png | .. figure:: ./attachments/only_rd_rate_ramp_latency_hbm.png | .. figure:: ./attachments/only_wr_rate_ramp_latency_hbm.png | | | | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | | | | .. figure:: ./attachments/simultaneous_wr_rd_rate_ramp_latency_rd_hbm.png | | | | | | :alt: | | | | | | :align: center | | | | | | | | | +----------+-----------------------------------------+-------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------+---------------------------------------------------------------------------------+ ******************************************************** Calibration - Memory bandwidth and latency ******************************************************** .. note:: * **N/A**: not applicable * .. [1] The outstanding write/read is the maximum number of outstanding reads just before the read pipeline is full, giving the best BW and latency results. .. table:: +-----------+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | Memory | Description | ``simultaneous_wr_rd`` | ``only_rd`` | ``only_wr`` | +===========+=========================================+=====================================================================================================================+===============================================================================================+===============================================================================================+ | HBM | QoS / power reduction rate | Write rate = 33 %; read rate = 33 % | ``N/A`` | ``N/A`` | | | | | | | | | | .. figure:: ./attachments/simultaneous_wr_rd_qos_rate_ramp_bw_hbm.png | | | | | | :alt: | | | | | | :align: center | | | | | | | | | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best write BW (MBps) | 6008 | **N/A** | 12255 | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best write latency (ns) | 143 | **N/A** | 203 | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best read BW (MBps) | 6008 | 12292 | **N/A** | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Best read latency (ns) | 681 | 952 | **N/A** | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | Outstanding write/read [1]_ | Write = 2 / read = 21 | 4 | 3 | + +-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ | | BW/latency graphs | .. figure:: ./attachments/simultaneous_wr_rd_outstanding_ramp_wr_hbm.png | .. figure:: ./attachments/only_rd_outstanding_ramp_hbm.png | .. figure:: ./attachments/only_wr_outstanding_ramp_hbm.png | | | (do one for read and one for write) | :alt: | :alt: | :alt: | | | | :align: center | :align: center | :align: center | | | | | | | | | | .. figure:: ./attachments/simultaneous_wr_rd_outstanding_ramp_rd_hbm.png | | | | | | :alt: | | | | | | :align: center | | | | | | | | | +-----------+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------+ ******************************************************** Checklist questionnaire ******************************************************** ===================================================== Runtime ===================================================== .. table:: +---------------------------------------------------------------------------------+------------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+============================================================+ | Have you updated the default download time? | Yes | | | | | * FYI: | .. code-block:: json | | | | | * u250 xclbin (90Mb) downloads in 3.9 sec (PCIe 3x16). | "download_time": 13 | | * HBM requires calibration which may take time. | | | | Measured download time is 10.5 sec. | | | Extra padding added to give time for HBM calibration. | +---------------------------------------------------------------------------------+------------------------------------------------------------+ ===================================================== Sensors ===================================================== .. table:: +---------------------------------------------------------------------------------+------------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+============================================================+ | Do you need to monitor more mechanical sources than the default one | No | | (``fpga_fan_1``)? | | | | | | * Yes, see |fill-platform-definition-json|_ | | +---------------------------------------------------------------------------------+------------------------------------------------------------+ | Do you need to monitor more thermal sources than the default one (``fpga0``)? | No | | | | | * Yes, see |fill-platform-definition-json|_ | | +---------------------------------------------------------------------------------+------------------------------------------------------------+ | Do you need to monitor more electrical sources than the default ones | No | | (``power_consumption``, ``12v_pex``, ``12v_aux``, ``3v3_pex`` and ``vccint``)? | | | | | | * Yes, see |fill-platform-definition-json|_ | | +---------------------------------------------------------------------------------+------------------------------------------------------------+ ===================================================== GT ===================================================== .. table:: +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+============================================================+============================================================+ | GT | GT[0] | GT[1] | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you updated GT default settings to maximize signal integrity? | .. code-block:: json | .. code-block:: json | | | | | | * Run all GT pre-canned tests. | "0": { | "1": { | | | "name": "top", | "name": "bottom", | | | "transceiver_settings": { | "transceiver_settings": { | | | "module": { | "module": { | | | "tx_differential_swing_control": 11, | "tx_differential_swing_control": 11, | | | "tx_pre_emphasis": 0, | "tx_pre_emphasis": 0, | | | "tx_post_emphasis": 0, | "tx_post_emphasis": 0, | | | "rx_equaliser": "DFE" | "rx_equaliser": "DFE" | | | }, | }, | | | "cable": { | "cable": { | | | "tx_differential_swing_control": 11, | "tx_differential_swing_control": 11, | | | "tx_pre_emphasis": 0, | "tx_pre_emphasis": 0, | | | "tx_post_emphasis": 0, | "tx_post_emphasis": 0, | | | "rx_equaliser": "DFE" | "rx_equaliser": "DFE" | | | } | } | | | } | } | | | } | } | | | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ ===================================================== Memory ===================================================== .. table:: +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+============================================================+============================================================+ | Memory | HBM | HOST | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined specific DMA bandwidths (``dma_bw``)? | No | **N/A** | | | | | | * This not applicable for host memory. | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined optimum DMA settings (``buffer_size`` and ``total_size``) | No | **N/A** | | or are default settings enough? | | | | | | | | * Run dma pre-canned test. | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined specific P2P CARD bandwidths (``p2p_card_bw``)? | No | **N/A** | | | | | | * This not applicable for host memory. | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined optimum P2P CARD settings (``buffer_size`` and ``total_size``) | No | **N/A** | | or are default settings enough? | | | | | | | | * Run p2p_card pre-canned test. | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined specific P2P NVME bandwidths (``p2p_nvme_bw``)? | No | **N/A** | | | | | | * This not applicable for host memory. | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined optimum P2P NVME settings (``buffer_size`` and ``total_size``) | No | **N/A** | | or are default settings enough? | | | | | | | | * Run p2p_nvme pre-canned test. | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you reduced the CU rate to avoid over-power (``cu_rate``)? | No | No | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined ``cu_rate`` for simultaneous write/read bandwidth QoS? | .. code-block:: json | .. code-block:: json | | | | | | | "cu_rate": { | "cu_rate": { | | | "simul_wr_rd": { | "only_wr": { | | | "write": { | "write": { | | | "nominal": 33 | "nominal": 50 | | | }, | } | | | "read": { | } | | | "nominal": 33 | } | | | } | | | | } | | | | } | | | | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined ``cu_bw``? | .. code-block:: json | .. code-block:: json | | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | "cu_bw": { | "cu_bw": { | | | "only_wr": { | "only_wr": { | | For host memory these values are depending on the PCIe speed. | "write": { | "write": { | | | "average": 12255 | "high": 16000, | | | } | "low": 4000 | | | }, | } | | | "only_rd": { | }, | | | "read": { | "only_rd": { | | | "average": 12292 | "read": { | | | } | "high": 16000, | | | }, | "low": 4000 | | | "simul_wr_rd": { | } | | | "write": { | }, | | | "average": 6008 | "simul_wr_rd": { | | | }, | "write": { | | | "read": { | "high": 16000, | | | "average": 6008 | "low": 4000 | | | } | }, | | | } | "read": { | | | } | "high": 16000, | | | | "low": 4000 | | | | } | | | | } | | | | } | | | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined ``cu_latency``? | .. code-block:: json | .. code-block:: json | | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | "cu_latency": { | "cu_latency": { | | | "only_wr": { | "only_wr": { | | | "write": { | "write": { | | | "high": 508, | "high": 6000, | | | "low": 141 | "low": 1 | | | } | } | | | }, | }, | | | "only_rd": { | "only_rd": { | | | "read": { | "read": { | | | "high": 1239, | "high": 6000, | | | "low": 666 | "low": 1 | | | } | } | | | }, | }, | | | "simul_wr_rd": { | "simul_wr_rd": { | | | "write": { | "write": { | | | "high": 232, | "high": 6000, | | | "low": 100 | "low": 1 | | | }, | }, | | | "read": { | "read": { | | | "high": 1007, | "high": 6000, | | | "low": 466 | "low": 1 | | | } | } | | | } | } | | | } | } | | | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined ``cu_outstanding``? | .. code-block:: json | No | | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | "cu_outstanding": { | | | | "only_wr": { | | | | "write": { | | | | "nominal": 3 | | | | } | | | | }, | | | | "only_rd": { | | | | "read": { | | | | "nominal": 4 | | | | } | | | | }, | | | | "simul_wr_rd": { | | | | "write": { | | | | "nominal": 2 | | | | }, | | | | "read": { | | | | "nominal": 21 | | | | } | | | | } | | | | } | | | | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ | Have you defined ``cu_burst_size``? | No | No | | | | | | * For each test mode: ``only_wr``, ``only_rd`` and ``simul_wr_rd`` | | | +---------------------------------------------------------------------------------+------------------------------------------------------------+------------------------------------------------------------+ ===================================================== Pre-canned tests ===================================================== .. table:: +---------------------------------------------------------------------------------+------------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+============================================================+ | Have you tested all provided pre-canned tests? | Yes | | | | | * If not: list the skipped one and the reason? | | +---------------------------------------------------------------------------------+------------------------------------------------------------+ | Have you modified the pre-canned test template? | No | | | | | * If yes: | | | | | | * Why?  | | | * Was it not possible to make a platform definition update | | | (and thus, available for any test as default)? | | | * Update next section with your modifications. | | +---------------------------------------------------------------------------------+------------------------------------------------------------+ | Have you saved all pre-canned results? | Yes | | | | | * Update package information section of your checklist. | | +---------------------------------------------------------------------------------+------------------------------------------------------------+ ===================================================== Issues tracking ===================================================== .. table:: +---------------------------------------------------------------------------------+------------------------------------------------------------+ | Question | Answer / justification / problem tracker reference number | +=================================================================================+============================================================+ | Did you modify the platform definition limits and range after running the | No | | pre-canned tests? | | | | | | * If yes: | | | | | | * List the problem tracker reference number | | +---------------------------------------------------------------------------------+------------------------------------------------------------+ ******************************************************** Pre-canned tests results ******************************************************** ===================================================== Pre-canned test modifications ===================================================== .. table:: +-----------+-------------------------------------+----------------+ | Test name | Modification from original template | Justifications | +===========+=====================================+================+ | / | No modification | / | +-----------+-------------------------------------+----------------+ ===================================================== ``dma`` pre-canned test ===================================================== .. table:: +-----------------+-----------------+-----------------+ | Memory | Write BW (MBps) | Read BW (MBps) | +=================+=================+=================+ | HBM | 11932 | 12083 | +-----------------+-----------------+-----------------+ ===================================================== ``p2p_card`` pre-canned test ===================================================== .. table:: +-----------------+-----------------+-----------------+ | Memory | Write BW (MBps) | Read BW (MBps) | +=================+=================+=================+ | HBM | 3723 | 1832 | +-----------------+-----------------+-----------------+ ===================================================== ``p2p_nvme`` pre-canned test ===================================================== .. table:: +-----------------+-----------+-----------------+-----------------+ | Memory | Card mode | Write BW (MBps) | Read BW (MBps) | +=================+===========+=================+=================+ | HBM | source | 2181 | 2855 | +-----------------+-----------+-----------------+-----------------+ | HBM | target | 2968 | 1950 | +-----------------+-----------+-----------------+-----------------+ ===================================================== ``memory`` pre-canned test ===================================================== .. table:: +-----------------+------------------------+-----------------+-----------------+--------------------+-------------------+ | Memory | Test mode | Write BW (MBps) | Read BW (MBps) | Write Latency (ns) | Read Latency (ns) | +=================+========================+=================+=================+====================+===================+ | | ``alternate_wr_rd`` | 196608 | 196160 | 208 | 952 | + +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``only_wr`` | 392172 | **N/A** | 208 | **N/A** | + HBM +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``only_rd`` | **N/A** | 393369 | **N/A** | 952 | + +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``simultaneous_wr_rd`` | 192263 | 192260 | 144 | 733 | +-----------------+------------------------+-----------------+-----------------+--------------------+-------------------+ ===================================================== ``memory_host`` pre-canned test ===================================================== .. table:: +-----------------+------------------------+-----------------+-----------------+--------------------+-------------------+ | Memory | Test mode | Write BW (MBps) | Read BW (MBps) | Write Latency (ns) | Read Latency (ns) | +=================+========================+=================+=================+====================+===================+ | | ``alternate_wr_rd`` | 5229 | 5222 | 829 | 3835 | + +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``only_wr`` | 9155 | **N/A** | 829 | **N/A** | + HOST +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``only_rd`` | **N/A** | 12178 | **N/A** | 3835 | + +------------------------+-----------------+-----------------+--------------------+-------------------+ | | ``simultaneous_wr_rd`` | 9155 | 12006 | 829 | 3890 | +-----------------+------------------------+-----------------+-----------------+--------------------+-------------------+