Profile an HPC Application
MPI Profiling is very important for benchmarking an HPC application.Â
There are several ways to profile an application, here is an example using ipm.
MPI Profiling exampleÂ
This example assumes that you have ipm installed as a module in your cluster.
1. Add the IPM profiler to your slurm scriptÂ
Use this example to add the profiler to your slurm script
#!/bin/bash # #SBATCH --job-name=test # #SBATCH --time=10:00 #SBATCH --partition helios #SBATCH --nodes=32 #SBATCH --ntasks-per-node=2 #SBATCH --threads-per-core=1 #SBATCH --output=test-%j.out . . . PROFILE=1 if [[ $PROFILE == 1 ]]; then module load ipm/2.0.6-hpcx-2.1.0 # in this example we use this ipm module. export IPM_DIR=$HPCX_IPM_DIR IPM_KEYFILE=$IPM_DIR/etc/ipm_key_mpi export IPM_LOG=full export IPM_REPORT=full export IPM_STATS=all PROFILE_FLAGS="-x LD_PRELOAD=$IPM_DIR/lib/libipm.so " BARRIER=1 # Barrier is important to have a clean picture of the MPI API commands if [[ "$BARRIER" == "1" ]]; then export IPM_ADD_BARRIER_TO_REDUCE=1 export IPM_ADD_BARRIER_TO_ALLREDUCE=1 export IPM_ADD_BARRIER_TO_GATHER=1 export IPM_ADD_BARRIER_TO_ALL_GATHER=1 export IPM_ADD_BARRIER_TO_ALLTOALL=1 export IPM_ADD_BARRIER_TO_ALLTOALLV=1 export IPM_ADD_BARRIER_TO_BROADCAST=1 export IPM_ADD_BARRIER_TO_SCATTER=1 export IPM_ADD_BARRIER_TO_SCATTERV=1 export IPM_ADD_BARRIER_TO_GATHERV=1 export IPM_ADD_BARRIER_TO_ALLGATHERV=1 fi fi mpirun $FLAGS $PROFILE_FLAGS app.exe
2. Run the sbatch of the slurm script, and look for xml output in the benchmarking directory
$ ls ... ophirm.1536187829.459772.ipm.xml
3. Load the ploticus moduleÂ
$ module load ploticus
4. Run the IPM parser (ipm_parse) to generate the html folder.
$ /global/software/centos-7/modules/intel/2018.1.163/ipm/2.0.6-hpcx-2.1.0/bin/ipm_parse -html ophirm.1536187829.459772.ipm.xml # data_acquire = 0 sec # data_workup = 0 sec # mpi_pie = 0 sec # task_data = 0 sec # load_bal = 1 sec # time_stack = 0 sec # mpi_stack = 0 sec # mpi_buff = 1 sec # switch+mem = 0 sec # topo_tables = 0 sec # topo_data = 1 sec # topo_time = 0 sec # html_all = 3 sec # html_regions = 0 sec # html_nonregion = 0 sec
5. Open the index.html in a browser. Copy the whole directory to Windows server if needed, and save as PDF.