In some clusters that InfiniBand switches doesn’t have CPU (unmanaged or externally managed), in order to force different port speed, use ibportstate command.

ibportstate

For that we will need to change the speed and espeed (extended speed) parameter.

# Force PortInfo:LinkSpeedEnabled on switch ports
# If 0, don't modify PortInfo:LinkSpeedEnabled on switch port
# Otherwise, use value for PortInfo:LinkSpeedEnabled on switch port
# Values are (IB Spec 1.2.1, 14.2.5.6 Table 146 "PortInfo")
#    1: 2.5 Gbps
#    3: 2.5 or 5.0 Gbps
#    5: 2.5 or 10.0 Gbps
#    7: 2.5 or 5.0 or 10.0 Gbps
#    2,4,6,8-14 Reserved
#    Default 15: set to PortInfo:LinkSpeedSupported
force_link_speed 15

# Force PortInfo:LinkSpeedExtEnabled on ports
# If 0, don't modify PortInfo:LinkSpeedExtEnabled on port
# Otherwise, use value for PortInfo:LinkSpeedExtEnabled on port
# Values are (MgtWG RefIDs #4722 and #9366)
#    1: 14.0625 Gbps
#    2: 25.78125 Gbps
#    3: 14.0625 Gbps or 25.78125 Gbps
#    4: 53.125 Gbps
#    5: 14.0625 Gbps or 53.125 Gbps
#    6: 25.78125 Gbps or 53.125 Gbps
#    7: 14.0625 Gbps, 25.78125 Gbps or 53.125 Gbps
#    30: Disable extended link speeds
#    Default 31: set to PortInfo:LinkSpeedExtSupported
force_link_speed_ext 31

For example:

This link is set to HDR (rate is 200Gb/s), LID=360.

$ ibstat
CA 'mlx5_5'
        CA type: MT4123
        Number of ports: 1
        Firmware version: 20.29.2002
        Hardware version: 0
        Node GUID: 0xb8599f0300568101
        System image GUID: 0xb8599f0300568100
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 360
                LMC: 0
                SM lid: 9
                Capability mask: 0x2651e848
                Port GUID: 0xb8599f0300568101
                Link layer: InfiniBand

To force the port to EDR, run the following command:

$ sudo ibportstate -L 360 1 espeed 2
…

$ sudo ibportstate -L 360 1 reset
…

use ibstate to check that the rate is 100.

To force the port to FDR for example, use espeed=1.

use ibstate to check that the rate is 56.

To revert back to normal speed use espeed=31 (default):

For HDR100 speed you will need to force the port width to 2x instead of the 4x default. To do that use the width option on the ibportstate.

Note: 16 is the value of x2 (two lanes), other options as available, see the list below. The default value is 255.

For example:

Script Example: changing Rome 8 servers to HDR100

mlxlink

Another option is to do it via mlxlink, as follows:

HPC-Works

How to Change port speed for InfiniBand Unmanaged Switches

ibportstate

mlxlink