...
| Code Block |
|---|
$ module load osu/5.4-impi-2018.1.163-gcc-4.8.5 $ mpirun -n 2 -ppn 1 -hosts jupiter002,jupiter003 -genv I_MPI_DEBUG 4 -genv I_MPI_FALLBACK 0 -genv I_MPI_FABRICS shm:dapl -genv DAT_OVERRIDE ./dat.conf -genv I_MPI_DAT_LIBRARY /usr/lib64/libdat2.so -genv I_MPI_DAPL_PROVIDER=ofa-v2-cma-roe-enp5s0f1 osu_bw |
Note: it is important to use -genv I_MPI_FABRICS shm:dapl and -genv I_MPI_FALLBACK 0 and not just -dapl, this will guarantee that no fabric fallback will happen. If they simply use -dapl this allows the fabric to fallback to other DAPL capable device.
5. To confirm traffic is going through RoCEv2 please follow the guidance in Ref. 2.
...