MPI Profiling example

This example assumes that you have ipm installed as a module in your cluster.

1. Add the IPM profiler to your slurm script

Use this example to add the profiler to your slurm script

#!/bin/bash
#
#SBATCH --job-name=test
#
#SBATCH --time=10:00
#SBATCH --partition helios
#SBATCH --nodes=32
#SBATCH --ntasks-per-node=2
#SBATCH --threads-per-core=1
#SBATCH --output=test-%j.out

.
.
.

PROFILE=1
if [[ $PROFILE == 1 ]]; then
        module load ipm/2.0.6-hpcx-2.1.0   # in this example we use this ipm module.
        export IPM_DIR=$HPCX_IPM_DIR
        IPM_KEYFILE=$IPM_DIR/etc/ipm_key_mpi
        export IPM_LOG=full
        export IPM_REPORT=full
        export IPM_STATS=all
        PROFILE_FLAGS="-x LD_PRELOAD=$IPM_DIR/lib/libipm.so "

        BARRIER=1  # Barrier is important to have a clean picture of the MPI API commands
        if [[ "$BARRIER" == "1" ]]; then
                export IPM_ADD_BARRIER_TO_REDUCE=1
                export IPM_ADD_BARRIER_TO_ALLREDUCE=1
                export IPM_ADD_BARRIER_TO_GATHER=1
                export IPM_ADD_BARRIER_TO_ALL_GATHER=1
                export IPM_ADD_BARRIER_TO_ALLTOALL=1
                export IPM_ADD_BARRIER_TO_ALLTOALLV=1
                export IPM_ADD_BARRIER_TO_BROADCAST=1
                export IPM_ADD_BARRIER_TO_SCATTER=1
                export IPM_ADD_BARRIER_TO_SCATTERV=1
                export IPM_ADD_BARRIER_TO_GATHERV=1
                export IPM_ADD_BARRIER_TO_ALLGATHERV=1
        fi
fi

mpirun $FLAGS $PROFILE_FLAGS app.exe

2. Run the sbatch of the slurm script, and look for xml output in the benchmarking directory

$ ls 
...
ophirm.1536187829.459772.ipm.xml

3. Load the ploticus module

$ module load ploticus

4. Run the IPM parser (ipm_parse) to generate the html folder.

$ /global/software/centos-7/modules/intel/2018.1.163/ipm/2.0.6-hpcx-2.1.0/bin/ipm_parse -html ophirm.1536187829.459772.ipm.xml
# data_acquire = 0 sec
# data_workup = 0 sec
# mpi_pie = 0 sec
# task_data = 0 sec
# load_bal = 1 sec
# time_stack = 0 sec
# mpi_stack = 0 sec
# mpi_buff = 1 sec
# switch+mem = 0 sec
# topo_tables = 0 sec
# topo_data = 1 sec
# topo_time = 0 sec
# html_all = 3 sec
# html_regions = 0 sec
# html_nonregion = 0 sec

5. Open the index.html in a browser. Copy the whole directory to Windows server if needed, and save as PDF.

Browser not supported