How to Split HDR port to two HDR100 ports on Mellanox HDR Switch

Owned by Ophir Maor
Last updated: Feb 27, 2020
6 min read

This post walk-through the procedure to split HDR switch port to two HDR100 ports on Mellanox HDR Quantum switch (QM8700). 

Note: Nothing is needed to be done on the hosts, it is only switch configuration.


References 

Notes:

  • Make sure to use HDR split-cables connecting the servers to the switch
  • In case you use regular HDR cable, only two lanes will be used to reach HDR100 speed, other 2 lanes will not be used.

Configuration 

1. Connect to hosts with HDR-to-HDR100 Y cable to an HDR switch. Make sure to connect the HDR side to the switch.


2. Login to the switch and check the system profile. Split Ready should be yes

switch (config) # show system profile

Profile : ib
Number of SWIDs : 1
Adaptive Routing : yes
Adaptive Routing Groups: 1792
Split Ready : yes
IB Routing : no


Note that by default, Split Ready is no. To enable it, run the following command to change the system profile. It requires reboot.

switch (config) # system profile ib split-ready


This command will clear the switch configuration (besides the management IP), so it is recommended to save your current configuration, and apply again after reboot.

To save the configuration run:

switch (config) # configuration write to <filename>


3. After reboot, verify that the system profile was changed, and split the relevant ports using this command:

switch (config)  # interface ib 1/10 module-type qsfp-split-2 force


4. Make sure that SM is enabled and check the port status, see that the split port 1/10 is now two ports 1/10/1 and 1/10/2 , see that the speed is 100Gb/s HDR and the width is 2X.


# show interfaces ib 1/10/1
IB1/10/1 state:
        Logical port state          : Active
        Physical port state         : LinkUp
        Current line rate           : 100.0 Gbps
        Supported speeds            : sdr, edr, hdr
        Speed                       : hdr
        Supported widths            : 1X, 2X
        Width                       : 2X
        Max supported MTUs          : 4096
        MTU                         : 4096
        VL capabilities             : VL0 - VL7
        Operational VLs             : VL0 - VL3
        Description                 : 
        IB Subnet                   : infiniband-default
        Phy-profile                 : high-speed-ber
        Width reduction mode        : Not supported
        Telemetry sampling          : Disabled
        Telemetry threshold         : Disabled
        Telemetry record            : Disabled
        Telemetry threshold level   : N/A bytes

        RX bytes                    : 186135451664
        RX packets                  : 55767477
        RX errors                   : 0
        Symbol errors               : 0
        VL15 dropped packets        : 0

        TX bytes                    : 441369932
        TX packets                  : 13916715
        TX wait                     : 0
        TX discarded packets        : 0


5. On the Server you should use ibstatus and check the rate parameter.


In case the link is HDR100 you would see that rate is equal to 100 Gb/sec (2X HDR)

$ ibstatus mlx5_2
Infiniband device 'mlx5_2' port 1 status:
        default gid:     fe80:0000:0000:0000:b859:9f03:0056:8100
        base lid:        0xcf
        sm lid:          0x1
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            100 Gb/sec (2X HDR)
        link_layer:      InfiniBand


As a reference, in case the link is HDR you would see  that rate is 200 Gb/sec (4X HDR)

$ ibstatus mlx5_2
Infiniband device 'mlx5_2' port 1 status:
        default gid:     fe80:0000:0000:0000:b859:9f03:0056:8100
        base lid:        0xcf
        sm lid:          0x1
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (4X HDR)
        link_layer:      InfiniBand


6. Script example to change 40 switch ports from HDR to HDR100, need to make sure that the profile is set to split-ready as mentioned above. Change the <switch-name> and <switch-password> to yours.

#!/usr/bin/expect

set timeout 20

#spawn "./addition.pl"
spawn ssh admin@switch-name

expect "Password:" { send "switch-password\r" }
expect "> " { send "en\r" }
expect "# " { send "conf term\r"

puts ""
for {set port 1} {$port<=40} {incr port} {
        send "interface ib 1/$port module-type qsfp-split-2 force \r"

        puts "PORT=$port is splitted."
}

send "configuration write      \r"
send "exit                     \r"
send "exit                     \r"
}
interact


Un-Split HDR100 to HDR

1. To unsplit the port on the switch use the following command 

switch (config) # interface ib 1/<port>/<1-2> module-type qsfp force



All trademarks are property of their respective owners. All information is provided “As-Is” without any kind of warranty.   The HPC Advisory Council makes no representation to the accuracy and completeness of the information contained herein.  HPC Advisory Council undertakes no duty and assumes no obligation to update or correct any information presented herein