AVED Debug Techniques

Debug Methods

The debug techniques are listed here in approximate increasing order of complexity and effort.

Programming

If an issue occurs during a cfgmem_fpt/cfgmem_program (AVED Management Interface userguide (ami_tool)#overview) please go straight to AVED Updating FPT Image in Flash to recover the card.

Card status

Once an error has occurred, it can be useful to check the card status and health to see what is still working and what is not. Some useful checks are:

For AMI specific commands see - AVED Management Interface userguide (ami_tool).

  • lspci device check

    • Issue ‘sudo lspci -vvs <BDF>’

    • 21:00.0 Processing accelerators: Xilinx Corporation Device 50b4
              Subsystem: Xilinx Corporation Device 000e
              Physical Slot: 2-1
              Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
              Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
              Latency: 0
              NUMA node: 2
              Region 0: Memory at 10400000000 (64-bit, prefetchable) [size=256M]
              Capabilities: [40] Power Management version 3
                      Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold+)
                      Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
              Capabilities: [70] Express (v2) Endpoint, MSI 00
                      DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                              ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
                      DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
                              RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                              MaxPayload 512 bytes, MaxReadReq 512 bytes
                      DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                      LnkCap: Port #0, Speed 32GT/s, Width x8, ASPM not supported
                              ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                      LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                              ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                      LnkSta: Speed 16GT/s (downgraded), Width x8 (ok)
                              TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                      DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR+
                               10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt- EETLPPrefix-
                               EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                               FRS- TPHComp- ExtTPHComp-
                               AtomicOpsCap: 32bit+ 64bit+ 128bitCAS+
                      DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- LTR- OBFF Disabled,
                               AtomicOpsCtl: ReqEn-
                      LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                      LnkCtl2: Target Link Speed: 32GT/s, EnterCompliance- SpeedDis-
                               Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                               Compliance De-emphasis: -6dB
                      LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                               EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                               Retimer- 2Retimers- CrosslinkRes: unsupported
              Capabilities: [100 v1] Advanced Error Reporting
                      UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                      UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                      UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                      CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                      CEMsk:  RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
                      AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                              MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                      HeaderLog: 00000000 00000000 00000000 00000000
              Capabilities: [188 v1] Alternative Routing-ID Interpretation (ARI)
                      ARICap: MFVC- ACS-, Next Function: 1
                      ARICtl: MFVC- ACS-, Function Group: 0
              Capabilities: [1c0 v1] Secondary PCI Express
                      LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                      LaneErrStat: 0
              Capabilities: [3a0 v1] Data Link Feature <?>
              Capabilities: [3b0 v1] Physical Layer 16.0 GT/s <?>
              Capabilities: [400 v1] Lane Margining at the Receiver <?>
              Capabilities: [460 v1] Extended Capability ID 0x2a
              Capabilities: [600 v1] Vendor Specific Information: ID=0020 Rev=0 Len=010 <?>
              Kernel driver in use: ami
              Kernel modules: ami
      
    • Check the PF has expected

      • device ID

      • number of BARs

      • BAR sizes

      • link speed and width in LnkCap

      • link speed and width in LnkSta

      • Vendor Specific capability present

      • kernel driver in use

        • PF0 AMI

  • ami_tool link check

    • Issue ‘ami_tool pcieinfo -d <BDF>’

    • For each expected PF, check the following entries for consistency with lspci link status

      • PCIe Link

      • NUMA NODE

      • CPU Affinity

  • ami_tool overview check

    • Issue ‘ami_tool overview’

    • check the AMI version is as expected

      • AMI version should match AMI driver version

      • AMI major version should match AMC major version

    • For each BDF check

      • ‘Devices State’ is READY

        • if the device state is not READY, please cold reboot the system

      • Expected design name and UUID

  • ami_tool mfg_info check

    • Issue ‘ami_tool mfg_info -d <BDF>’

    • Check

      • eeprom version

      • product name

      • board revision

      • serial no

      • mac address

      • mfg date

      • uuid

      • board part number

      • mfg part number

  • xbtest verify check

    • issue ‘xbtest -c verify -d <BDF> -F’

    • When completed, for each BDF check

      • “RESULT: ALL TESTS PASSED” present

      • “ERROR” not present

  • sensor check

    • Issue “ami_tool sensors -d <BDF> -f json -o <FILE>’

    • Check produced json format is correct

    • For each expected sensor

xbtest logs and CSV files

For errors that occurred while xbtest was running, check the xbest logs for context and error reporting.

xbtest writes sensor data into several CSV files. These can be analyzed as text or imported into Microsoft Excel to analyze the data graphically.

xbtest can be useful to test the card capabilities and to push it to its limits. This is more often helpful for exposing or reproducing an issue, rather than triaging or debugging an issue. However, xbtest does provide an easy method for continuously reading card sensors such as power and temperature, which can be useful in some debug situations.

For xbtest documentation, see AVED Deployment / xbtest Userguide.

Server logs and sensors

  • BMC (Baseboard Management Controller)

    • This provides remote configuration and power control for the Dell servers that V80 cards have been verified against.

    • There is an BMC for each server. It stores logs of server activity. These can help to diagnose physical server issues (e.g. someone pulled out a power cord, or shut down the wrong server). Use the log timestamps to correlate this with other logs.

    • The BMC stores server sensor readings such as temperature. This historical data can be viewed in logs and charts and can help to diagnose issues with power or temperature. For example, if card sensors are not working properly or are not being reported correctly then xbtest would not report the correct temperature, but the BMC would.

  • Crash logs

    • To help with debug, when a server crash occurs, the server loads a crash kernel, which freezes the state of the server and writes this state and the current messages into a crash directory.

    • This is stored in /var/crash/ where a new directory is created for each server crash, named with a timestamp.

dmesg

dmesg is an OS tool for reporting messages from the kernel, including from drivers such as AMI. Useful commands are:

dmesg | less          # show full dmesg output, piped to a pager
dmesg -wT             # show live dmesg output, auto-updates when new messages are sent. -T gives wall clock timestamps (default is seconds since server reboot)

Consult OS documentation for full usage details.

dmesg is particularly useful for information about PCIe® connections at server boot and for AMI debug information.

For AMI messages:

  • messages are preceded by ami: and usually include the card BDF.

  • heartbeat message for AMI/AMC comms will appear if there’s a failure or breakdown in comms.

    • No response - “Failed to get the heartbeat msg!” and “AMC Heartbeat expired event received”

    • Fatal failure (communications have been shut down) - “Heartbeat fail count above threshold! Raising fatal event…” and “AMC Heartbeat fatal event received, stopping GCQ…”

    • Incorrect Response - “Heartbeat validation failed!” and “AMC Heartbeat validation event received”

dmesg content is also written into log files at /var/log/ - the exact directory and file names here differ between RedHat/CentOS and Ubuntu (e.g. on RedHat/CentOS the full dmesg content is written to file messages , on Ubuntu this file is named kern.log ). Use sudo to read these log files.


The AMI driver also populates dmesg with log messages received from AMC; these are always additionally prefixed with the string “AMC OUTPUT:”. By default, the only AMC messages printed in dmesg are those with the level “LOG”.

Changing message verbosity

The verbosity of the AMI messages can be changed by changing the value in /sys/bus/pci/drivers/ami/ami_debug_enabled (ensure the file has write-permissions).

  • If the value is 1, additional debug messages are added to dmesg.

  • If the value is 0, only errors and important information messages are added to dmesg.

The verbosity of the AMC messages can be updated with the debug_verbosity command; for example, to enable AMC debug messages you can run the command ami_tool debug_verbosity -d <bdf> -l debug - this will cause all AMC debug messages to appear in dmesg.

Resets and reboots

Performing a reset after an issue has occurred can help to understand the impact and scope of the issue. For example, if the issue is cleared by a hot reset, the issue is not wide-ranging, but if a full power cycle is required, the issue is more significant.

Several types of reset are possible for aved design:

  • power cycle : shut down server using BMC, power cycle system, boot up server using BMC

  • server cold reboot : shut down server using BMC, boot up server using BMC

  • server warm reboot reboot server using BMC

  • Host PCI reset (removes the PCI device from the host and forces a bus rescan; this does not remove power from the device): ami_tool reload -t pci -d <bdf>

  • PCIe hot reset (aka in-band reset - reset the FPGA but does not reprogram it from flash) : ami_tool reload -t sbr -d <bdf>

  • AMI driver reload (put driver into clean state without reprogramming): ami_tool reload -t driver -d <bdf>

Revert to provided exdes

For issues seen in custom designs, if applicable, it may be useful to revert back to the released AVED image to confirm they are/are not seen also.

sysfs

sysfs is a Linux pseudo file system that provides information about hardware devices in the server system. sysfs is mounted at /sys/ and can be accessed like any other Linux file system (cd, ls, cat, etc.).

/sys/bus/pci/devices/ shows the PCIe connected devices. Each directory here shows a mapping from the BDF to the associated device within sysfs. Each directory contains many files and subdirectories that show device information available to the OS.

Some examples include:

  • amc_version : version number for alveo management controller (amc)

  • logic_uuid: Build UUID

Hardware read/write

Use ami_tool bar_rd and bar_wr to read or write from memory-mapped registers or memories within the platform

bar_rd

% ami_tool bar_rd -d <BDF> -b 2 -a 0x0000ffff -l 4
0000ffff DEAD BEEF DEAD BEEF DEAD BEEF DEAD BEEF
INFO: 4 words read successfully

% ami_tool bar_rd -d <BDF> -b 2 -a 0x0000ffff -l 256 -o outfile.bin
INFO: 256 words read successfully

bar_wr

% ami_tool bar_wr -d <BDF> -b 2 -a 0x0000ffff -i 0xACDCACDC
INFO: 1 words written successfully

% ami_tool bar_wr -d <BDF> -b 2 -a 0x0000ffff -I infile.bin
INFO: 512 words written successfully

Swap card slots

If an issue occurs on one card but not on another, try swapping the cards over or move the card into an empty slot in the server and see where the issue occurs.

This can help to isolate an issue to a faulty card, faulty slot, or incorrect slot configuration.

There may be other factors involved. For example if the test only fails intermittently, or there may be some server race or load condition required to create the conditions that triggers the issue.

Swap cards or servers

Try running the test on a different card and/or in a different server, and see where the issue occurs.

This can help to isolate an issue to a faulty card or faulty server.

There may be other factors involved. For example, if the test only fails intermittently, or there may be some server race or load condition required to create the conditions that triggers the issue.


Issue Reporting

When reporting an issue against the AVED solution, provide the following information to aid with triage.

Required

  • Details of issue and any debug attempted.

    • OS and Kernel versions of the host server.

    • Consistent or intermittent issue?

    • Single or multi card?

    • Recoverable via reset/reboot?

    • steps to reproduce

  • logs

    • ami_tool overview

    • ami_tool pcieinfo -d <bdf>

    • ami_tool mfg_info -d <bdf>

    • ami_tool sensors -d <bdf>

    • ami_tool cfgmem_info -d <bdf> -t <type>

    • lspci -vvs <bdf>

    • dmesg

If applicable

  • xbtest logs:

    • Provide the full set of log files: zip the folder “BDF_<date>_<time>” of the falling run.

    • By default, xbtest stores log files under the folder “xbtest_logs”. With this folder each run is stored within its own folder titled BDF_<date>_<time>.

      • If you store your file differently (-l option), provide the folder accordingly.

  • server logs

Page Revision: v. 35