HDR InfiniBand and Dual rail support for HPC-X/UCX

Owned by Ophir Maor
Last updated: Jul 19, 2019
1 min read

HDR InfiniBand can come with both single rail using PCI gen4 x16 or Dual rail using  2x PCI gen3 x16.

In case dual rail is used (2x PCI gen3 x16), HPC-X OpenMPI or UCX should receive the interfaces to be used.


HDR InfiniBand lspci example and ibdev2netdev 

$ lspci | grep Mel
3b:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
3b:00.1 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
d8:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
d8:00.1 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]

$ ibdev2netdev 
mlx5_0 port 1 ==> ib0_mlx5 (Up)
mlx5_1 port 1 ==> ib0 (Down)
mlx5_2 port 1 ==> ib2_mlx5 (Up)
mlx5_3 port 1 ==> ib1 (Down)

$ sudo ibdev2netdev -v
0000:3b:00.0 mlx5_0 (MT4123 - MCX654106A-HCAT) ConnectX-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE, dual-port QSFP56, Socket Direct -Prime fw 20.25.6000 port 1 (ACTIVE) ==> ib0_mlx5 (Up)
0000:3b:00.1 mlx5_1 (MT4123 - MCX654106A-HCAT) ConnectX-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE, dual-port QSFP56, Socket Direct -Prime fw 20.25.6000 port 1 (DOWN  ) ==> ib0 (Down)
0000:d8:00.0 mlx5_2 (MT4123 - MCX654106A-HCAT) ConnectX-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE, dual-port QSFP56, Socket Direct -Aux[1] fw 20.25.6000 port 1 (ACTIVE) ==> ib2_mlx5 (Up)
0000:d8:00.1 mlx5_3 (MT4123 - MCX654106A-HCAT) ConnectX-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE, dual-port QSFP56, Socket Direct -Aux[1] fw 20.25.6000 port 1 (DOWN  ) ==> ib1 (Down)


When using UCX/HPC-X in case of dual rail, you need to pass the relevant device information to the UCX, in this example mlx5_0:1,mlx5_2:1

-x UCX_NET_DEVICES=mlx5_0:1,mlx5_2:1


Note: Since HPC-X 2.4, dual rail is supported by default.


More info can be found here.