Profile an HPC Application

Owned by Ophir Maor
Last updated: Sept 05, 2018

MPI Profiling is very important for benchmarking an HPC application. 

There are several ways to profile an application, here is an example using ipm.

MPI Profiling example 

This example assumes that you have ipm installed as a module in your cluster.


1. Add the IPM profiler to your slurm script 

Use this example to add the profiler to your slurm script


#!/bin/bash
#
#SBATCH --job-name=test
#
#SBATCH --time=10:00
#SBATCH --partition helios
#SBATCH --nodes=32
#SBATCH --ntasks-per-node=2
#SBATCH --threads-per-core=1
#SBATCH --output=test-%j.out

.
.
.

PROFILE=1
if [[ $PROFILE == 1 ]]; then
        module load ipm/2.0.6-hpcx-2.1.0   # in this example we use this ipm module.
        export IPM_DIR=$HPCX_IPM_DIR
        IPM_KEYFILE=$IPM_DIR/etc/ipm_key_mpi
        export IPM_LOG=full
        export IPM_REPORT=full
        export IPM_STATS=all
        PROFILE_FLAGS="-x LD_PRELOAD=$IPM_DIR/lib/libipm.so "

        BARRIER=1  # Barrier is important to have a clean picture of the MPI API commands
        if [[ "$BARRIER" == "1" ]]; then
                export IPM_ADD_BARRIER_TO_REDUCE=1
                export IPM_ADD_BARRIER_TO_ALLREDUCE=1
                export IPM_ADD_BARRIER_TO_GATHER=1
                export IPM_ADD_BARRIER_TO_ALL_GATHER=1
                export IPM_ADD_BARRIER_TO_ALLTOALL=1
                export IPM_ADD_BARRIER_TO_ALLTOALLV=1
                export IPM_ADD_BARRIER_TO_BROADCAST=1
                export IPM_ADD_BARRIER_TO_SCATTER=1
                export IPM_ADD_BARRIER_TO_SCATTERV=1
                export IPM_ADD_BARRIER_TO_GATHERV=1
                export IPM_ADD_BARRIER_TO_ALLGATHERV=1
        fi
fi

mpirun $FLAGS $PROFILE_FLAGS app.exe


2. Run the sbatch of the slurm script, and look for xml output in the benchmarking directory

$ ls 
...
ophirm.1536187829.459772.ipm.xml


3. Load the ploticus module 

$ module load ploticus


4. Run the IPM parser (ipm_parse) to generate the html folder.

$ /global/software/centos-7/modules/intel/2018.1.163/ipm/2.0.6-hpcx-2.1.0/bin/ipm_parse -html ophirm.1536187829.459772.ipm.xml
# data_acquire = 0 sec
# data_workup = 0 sec
# mpi_pie = 0 sec
# task_data = 0 sec
# load_bal = 1 sec
# time_stack = 0 sec
# mpi_stack = 0 sec
# mpi_buff = 1 sec
# switch+mem = 0 sec
# topo_tables = 0 sec
# topo_data = 1 sec
# topo_time = 0 sec
# html_all = 3 sec
# html_regions = 0 sec
# html_nonregion = 0 sec


5. Open the index.html in a browser. Copy the whole directory to Windows server if needed, and save as PDF.