MPI Profiling is very important for benchmarking an HPC application.
There are several ways to profile an application, here is an example using ipm.
MPI Profiling example
This example assumes that you have ipm installed as a module in your cluster.
1. Add the IPM profiler to your slurm script
Use this example to add the profiler to your slurm script
Code Block |
---|
#!/bin/bash
#
#SBATCH --job-name=test
#
#SBATCH --time=10:00
#SBATCH --partition helios
#SBATCH --nodes=32
#SBATCH --ntasks-per-node=2
#SBATCH --threads-per-core=1
#SBATCH --output=test-%j.out
.
.
.
PROFILE=1
if [[ $PROFILE == 1 ]]; then
module load ipm/2.0.6-hpcx-2.1.0 # in this example we use this ipm module.
export IPM_DIR=$HPCX_IPM_DIR
IPM_KEYFILE=$IPM_DIR/etc/ipm_key_mpi
export IPM_LOG=full
export IPM_REPORT=full
export IPM_STATS=all
PROFILE_FLAGS="-x LD_PRELOAD=$IPM_DIR/lib/libipm.so "
BARRIER=1 # Barrier is important to have a clean picture of the MPI API commands
if [[ "$BARRIER" == "1" ]]; then
export IPM_ADD_BARRIER_TO_REDUCE=1
export IPM_ADD_BARRIER_TO_ALLREDUCE=1
export IPM_ADD_BARRIER_TO_GATHER=1
export IPM_ADD_BARRIER_TO_ALL_GATHER=1
export IPM_ADD_BARRIER_TO_ALLTOALL=1
export IPM_ADD_BARRIER_TO_ALLTOALLV=1
export IPM_ADD_BARRIER_TO_BROADCAST=1
export IPM_ADD_BARRIER_TO_SCATTER=1
export IPM_ADD_BARRIER_TO_SCATTERV=1
export IPM_ADD_BARRIER_TO_GATHERV=1
export IPM_ADD_BARRIER_TO_ALLGATHERV=1
fi
fi
mpirun $FLAGS $PROFILE_FLAGS app.exe |
2. Run the sbatch of the slurm script, and look for xml output in the benchmarking directory
Code Block |
---|
$ ls
...
ophirm.1536187829.459772.ipm.xml |
3. Load the ploticus module
Code Block |
---|
$ module load ploticus |
4. Run the IPM parser (ipm_parse) to generate the html folder.
Code Block |
---|
$ /global/software/centos-7/modules/intel/2018.1.163/ipm/2.0.6-hpcx-2.1.0/bin/ipm_parse -html ophirm.1536187829.459772.ipm.xml
# data_acquire = 0 sec
# data_workup = 0 sec
# mpi_pie = 0 sec
# task_data = 0 sec
# load_bal = 1 sec
# time_stack = 0 sec
# mpi_stack = 0 sec
# mpi_buff = 1 sec
# switch+mem = 0 sec
# topo_tables = 0 sec
# topo_data = 1 sec
# topo_time = 0 sec
# html_all = 3 sec
# html_regions = 0 sec
# html_nonregion = 0 sec |
5. Open the index.html in a browser. Copy the whole directory to Windows server if needed, and save as PDF.