/
HowTo Set Specific Network Interface when using Intel MPI

HowTo Set Specific Network Interface when using Intel MPI

Owned by Ophir Maor
Apr 24, 2019

In cases when the cluster is connected to more than one fabric, you may want to use specific network interface on your MPI job.


For DAPL: 

1.  You need to add the device name you want to use to /etc/dat.conf file for Intel MPI. it includes only mlx5_0 and mlx5_1 by default.


2. Copy /etc/dat.conf to a shared folder like $HOME/dat.conf. use the -genv DAT_OVERRIDE $HOME/dat.conf flag in your script.


3. Add similar devices as the below example in $HOME/dat.conf

ofa-v2-mlx5_0-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_0 1" ""
ofa-v2-mlx5_0-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_0 2" ""
ofa-v2-mlx5_1-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_1 1" ""
ofa-v2-mlx5_1-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_1 2" ""
ofa-v2-mlx5_2-1u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_2 1" ""
ofa-v2-mlx5_2-2u u2.0 nonthreadsafe default libdaploucm.so.2 dapl.2.0 "mlx5_2 2" ""


For OFA:

1. you need just to specify the device name, using I_MPI_OFA_ADAPTER_NAME flag.


Example:

Here is a script for Intel MPI using mlx5_2 device for both OFA and DAPL:

---
TRANSPORT=DAPL
# TRANSPORT=OFA

HCA=mlx5_2

MPIFLAGS=""
MPIFLAGS+="-genv DAT_OVERRIDE $HOME/dat.conf "
MPIFLAGS+="-genv I_MPI_DAT_LIBRARY /usr/lib64/libdat2.so "

if [[ "$TRANSPORT" == "DAPL" ]]; then
MPIFLAGS+="-DAPL "
MPIFLAGS+="-genv I_MPI_FABRICS shm:dapl "
MPIFLAGS+="-genv I_MPI_DAPL_UD off "
MPIFLAGS+="-genv I_MPI_DAPL_PROVIDER ofa-v2-$HCA-1u "
MPIFLAGS+="-genv DAPL_MAX_INLINE 256 "
MPIFLAGS+="-genv I_MPI_DAPL_RDMA_RNDV_WRITE on "
MPIFLAGS+="-genv DAPL_IB_MTU 4096 "

elif [[ "$TRANSPORT" == "OFA" ]]; then
MPIFLAGS+="-IB "
MPIFLAGS+="-genv MV2_USE_APM 0 "
MPIFLAGS+="-genv I_MPI_FABRICS shm:ofa "
MPIFLAGS+="-genv I_MPI_OFA_USE_XRC 1 "
MPIFLAGS+="-genv I_MPI_OFA_NUM_ADAPTERS 1 "
MPIFLAGS+="-genv I_MPI_OFA_ADAPTER_NAME $HCA "
MPIFLAGS+="-genv I_MPI_OFA_NUM_PORTS 1 "
fi
MPIFLAGS+="-genv I_MPI_PIN on "
MPIFLAGS+="-genv I_MPI_DEBUG 4 "

mpirun -np <NPROC> $MPIFLAGS <exe>



Related content