In some clusters that InfiniBand switches doesn’t have CPU (unmanaged or externally managed), in order to force different port speed, use ibportstate command.

ibportstate

For that we will need to change the speed and espeed (extended speed) parameter.

# Force PortInfo:LinkSpeedEnabled on switch ports
# If 0, don't modify PortInfo:LinkSpeedEnabled on switch port
# Otherwise, use value for PortInfo:LinkSpeedEnabled on switch port
# Values are (IB Spec 1.2.1, 14.2.5.6 Table 146 "PortInfo")
#    1: 2.5 Gbps
#    3: 2.5 or 5.0 Gbps
#    5: 2.5 or 10.0 Gbps
#    7: 2.5 or 5.0 or 10.0 Gbps
#    2,4,6,8-14 Reserved
#    Default 15: set to PortInfo:LinkSpeedSupported
force_link_speed 15

# Force PortInfo:LinkSpeedExtEnabled on ports
# If 0, don't modify PortInfo:LinkSpeedExtEnabled on port
# Otherwise, use value for PortInfo:LinkSpeedExtEnabled on port
# Values are (MgtWG RefIDs #4722 and #9366)
#    1: 14.0625 Gbps
#    2: 25.78125 Gbps
#    3: 14.0625 Gbps or 25.78125 Gbps
#    4: 53.125 Gbps
#    5: 14.0625 Gbps or 53.125 Gbps
#    6: 25.78125 Gbps or 53.125 Gbps
#    7: 14.0625 Gbps, 25.78125 Gbps or 53.125 Gbps
#    30: Disable extended link speeds
#    Default 31: set to PortInfo:LinkSpeedExtSupported
force_link_speed_ext 31

For example:

This link is set to HDR (rate is 200Gb/s), LID=360.

$ ibstat
CA 'mlx5_5'
        CA type: MT4123
        Number of ports: 1
        Firmware version: 20.29.2002
        Hardware version: 0
        Node GUID: 0xb8599f0300568101
        System image GUID: 0xb8599f0300568100
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 200
                Base lid: 360
                LMC: 0
                SM lid: 9
                Capability mask: 0x2651e848
                Port GUID: 0xb8599f0300568101
                Link layer: InfiniBand

To force the port to EDR, run the following command:

$ sudo ibportstate -L 360 1 espeed 2
…

$ sudo ibportstate -L 360 1 reset
…

use ibstate to check that the rate is 100.

$ ibstat

CA 'mlx5_5'
        CA type: MT4123
        Number of ports: 1
        Firmware version: 20.29.2002
        Hardware version: 0
        Node GUID: 0xb8599f0300568101
        System image GUID: 0xb8599f0300568100
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 100
                Base lid: 360
                LMC: 0
                SM lid: 9
                Capability mask: 0x2651e848
                Port GUID: 0xb8599f0300568101
                Link layer: InfiniBand

To force the port to FDR for example, use espeed=1.

$ sudo ibportstate -L 360 1 espeed 1
…

$ sudo ibportstate -L 360 1 reset
…

use ibstate to check that the rate is 56.

$ ibstat

CA 'mlx5_5'
        CA type: MT4123
        Number of ports: 1
        Firmware version: 20.29.2002
        Hardware version: 0
        Node GUID: 0xb8599f0300568101
        System image GUID: 0xb8599f0300568100
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 56
                Base lid: 360
                LMC: 0
                SM lid: 9
                Capability mask: 0x2651e848
                Port GUID: 0xb8599f0300568101
                Link layer: InfiniBand

To revert back to normal speed use espeed=31 (default):

$ sudo ibportstate -L 360 1 espeed 31
…

$ sudo ibportstate -L 360 1 reset
…

For HDR100 speed you will need to force the port width to 2x instead of the 4x default. To do that use the width option on the ibportstate.

Note: 16 is the value of x2 (two lanes), other options as available, see the list below. The default value is 255.

# Force PortInfo:LinkWidthEnabled on switch ports
# If 0, don't modify PortInfo:LinkWidthEnabled on switch port
# Otherwise, use value for PortInfo:LinkWidthEnabled on switch port
# Values are (IB Spec 1.2.1, 14.2.5.6 Table 146 "PortInfo"
# augmented by MgtWG RefIDs #9306-9309)
# 1: 1x
# 2: 4x
# 3: 1x or 4x
# 4: 8x
# 5: 1x or 8x
# 6: 4x or 8x
# 7: 1x or 4x or 8x
# 8: 12x
# 9: 1x or 12x
# 10: 4x or 12x
# 11: 1x or 4x or 12x
# 12: 8x or 12x
# 13: 1x or 8x or 12x
# 14: 4x or 8x or 12x
# 15: 1x or 4x or 8x or 12x
# 16: 2x
# 17: 1x or 2x
# 18: 2x or 4x
# 19: 1x or 2x or 4x
# 20: 2x or 8x
# 21: 1x or 2x or 8x
# 22: 2x or 4x or 8x
# 23: 1x or 2x or 4x or 8x
# 24: 2x or 12x
# 25: 1x or 2x or 12x
# 26: 2x or 4x or 12x
# 27: 1x or 2x or 4x or 12x
# 28: 2x or 8x or 12x
# 29: 1x or 2x or 8x or 12x
# 30: 2x or 4x or 8x or 12x
# 31: 1x or 2x or 4x or 8x or 12x
# 32-254 Reserved
# Default 255: set to PortInfo:LinkWidthSupported
force_link_width 255

For example:

$sudo ibportstate -G <GUID> 1 width 16

$sudo ibportstate -G <GUID> 1 reset

Script Example: changing Rome 8 servers to HDR100

for i in {1..8}; do
        ssh rome00$i 'sudo su -c "/usr/sbin/ibportstate -G $(/sbin/ibstat mlx5_4 -p) 1 width 16"'
        ssh rome00$i 'sudo su -c "/usr/sbin/ibportstate -G $(/sbin/ibstat mlx5_4 -p) 1 reset"'
done

mlxlink

Another option is to do it via mlxlink, as follows:

# mlxlink -d lid-<lid from switch> -p <port-number> -s HDR, EDR, FDR, FDR10, QDR,DDR,SDR