HowTo Set Specific Network Interface when using Intel MPI

Owned by Ophir Maor
Apr 24, 2019
1 min read

In cases when the cluster is connected to more than one fabric, you may want to use specific network interface on your MPI job.


For DAPL: 

1.  You need to add the device name you want to use to /etc/dat.conf file for Intel MPI. it includes only mlx5_0 and mlx5_1 by default.


2. Copy /etc/dat.conf to a shared folder like $HOME/dat.conf. use the -genv DAT_OVERRIDE $HOME/dat.conf flag in your script.


3. Add similar devices as the below example in $HOME/dat.conf

ofa-v2-mlx5_0-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_0 1" ""
ofa-v2-mlx5_0-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_0 2" ""
ofa-v2-mlx5_1-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_1 1" ""
ofa-v2-mlx5_1-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_1 2" ""
ofa-v2-mlx5_2-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_2 1" ""
ofa-v2-mlx5_2-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_2 2" ""


For OFA:

1. you need just to specify the device name, using I_MPI_OFA_ADAPTER_NAME flag.


Example:

Here is a script for Intel MPI using mlx5_2 device for both OFA and DAPL:

---
TRANSPORT=DAPL
# TRANSPORT=OFA

HCA=mlx5_2

MPIFLAGS=""
MPIFLAGS+="-genv DAT_OVERRIDE $HOME/dat.conf "
MPIFLAGS+="-genv I_MPI_DAT_LIBRARY /usr/lib64/libdat2.so "

if [[ "$TRANSPORT" == "DAPL" ]]; then
MPIFLAGS+="-DAPL "
MPIFLAGS+="-genv I_MPI_FABRICS shm:dapl "
MPIFLAGS+="-genv I_MPI_DAPL_UD off "
MPIFLAGS+="-genv I_MPI_DAPL_PROVIDER ofa-v2-$HCA-1u "
MPIFLAGS+="-genv DAPL_MAX_INLINE 256 "
MPIFLAGS+="-genv I_MPI_DAPL_RDMA_RNDV_WRITE on "
MPIFLAGS+="-genv DAPL_IB_MTU 4096 "

elif [[ "$TRANSPORT" == "OFA" ]]; then
MPIFLAGS+="-IB "
MPIFLAGS+="-genv MV2_USE_APM 0 "
MPIFLAGS+="-genv I_MPI_FABRICS shm:ofa "
MPIFLAGS+="-genv I_MPI_OFA_USE_XRC 1 "
MPIFLAGS+="-genv I_MPI_OFA_NUM_ADAPTERS 1 "
MPIFLAGS+="-genv I_MPI_OFA_ADAPTER_NAME $HCA "
MPIFLAGS+="-genv I_MPI_OFA_NUM_PORTS 1 "
fi
MPIFLAGS+="-genv I_MPI_PIN on "
MPIFLAGS+="-genv I_MPI_DEBUG 4 "

mpirun -np <NPROC> $MPIFLAGS <exe>