ib trace (ibtracert)

ibtracert uses SMPs to trace the path from a source GID/LID to a destination GID/LID. Each hop along the path is displayed until the destination is reached or a hop does not respond. By using
the -m option, multicast path tracing can be performed between source and destination nodes.

In this example, we are using mlx5_1 device and lid 92 to reach two different lids, one (lid 265) over single switch hop and the other (lid 144) over 3 switches.

$ ibstat mlx5_1
CA 'mlx5_1'
        CA type: MT4123
        Number of ports: 1
        Firmware version: 20.26.4012
        Hardware version: 0
        Node GUID: 0x98039b0300855747
        System image GUID: 0x98039b0300855746
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 100
                Base lid: 92
                LMC: 0
                SM lid: 1
                Capability mask: 0x2651e848
                Port GUID: 0x98039b0300855747
                Link layer: InfiniBand

$ sudo ibtracert 92 265
From ca {0x98039b0300855747} portnum 1 lid 92-92 "thor001 HCA-2"
[1] -> switch port {0xb8599f0300fcca4c}[33] lid 105-105 "MF0;thor-qm8700-3:MQM8700/U1"
[34] -> ca port {0x98039b030085576f}[1] lid 265-265 "thor002 HCA-2"
To ca {0x98039b030085576f} portnum 1 lid 265-265 "thor002 HCA-2"

$ sudo ibtracert 92 114
From ca {0x98039b0300855747} portnum 1 lid 92-92 "thor001 HCA-2"
[1] -> switch port {0xb8599f0300fcca4c}[33] lid 105-105 "MF0;thor-qm8700-3:MQM8700/U1"
[31] -> switch port {0xb8599f0300fcca6c}[16] lid 118-118 "MF0;thor-qm8700-4:MQM8700/U1"
[19] -> switch port {0xb8599f0300fccaac}[29] lid 136-136 "MF0;thor-qm8700-2:MQM8700/U1"
[35] -> ca port {0x98039b0300855757}[1] lid 114-114 "thor017 HCA-2"
To ca {0x98039b0300855757} portnum 1 lid 114-114 "thor017 HCA-2"

Here is another example to trace the node to the SM node.

$ ibstat | grep lid
		Base lid: 13
		SM lid: 1

$ sudo ibtracert 13 1 -C mlx5_0
From ca {0x98039b0300855746} portnum 1 lid 13-13 "thor001 HCA-1"
[1] -> switch port {0x98039b030048fdac}[56] lid 6-6 "MF0;thor-qm8700:MQM8700/U1"
[1] -> switch port {0xec0d9a03001c7060}[36] lid 2-2 "MF0;helios-sb7800:MSB7800/U1"
[32] -> switch port {0x98039b03007ab858}[77] lid 33-33 "MF0;helios-qm8700:MQM8700/U1"
[79] -> ca port {0x98039b03009fd0ae}[1] lid 1-1 "sm02 HCA-1"
To ca {0x98039b03009fd0ae} portnum 1 lid 1-1 "sm02 HCA-1"