Overview

HemeLB is a 3D blood flow simulation code based on the lattice Boltzmann method (LBM). Optimised to solve for macroscopic flows in sparse domains characteristic of arteries and veins, it has been used to simulate flows from retinal vessels, to aneurysms to human-scale arterial trees. It is an open-source code built using C++ and MPI. Recent work has been undertaken to enable the HemeLB code to run on multiple GPU architectures. Both versions have demonstrated excellent strong scaling performance on some of the largest and fastest supercomputers on the planet.

Presentation:

In this event, we will be using the HemePure variant of HemeLB. This versions has been streamlined for large-scale performance on HPC systems. To get started, follow the links below for the repositories for the code.

CPU - https://github.com/UCL-CCS/HemePure

GPU - https://github.com/UCL-CCS/HemePure-GPU

Some further tips on using HemeLB can also be found here: https://hemelb-dev.github.io/HemeLB-Carpentries/index.html

HemePure has previously been built in Unix-style environments, with the current code Python3 will needed in addition to MPI and a C++ compiler to successfully build the dependencies/code.

Reasonably modern versions of these supporting software should suffice for building HemePure:

Python3 - v3.10+
C++ - GCC 11+. Earlier versions are also likely to work.
MPI - OpenMPI v4.1; MPICH v4.3 should also work if available on your system.

Build for CPU nodes

Compilation of HemePure is a two-stage process. Firstly the dependencies (packaged with the code) are built; once successful this doesn’t need to be repeated for future builds of the code (unless there is particular changes to supporting system software - e.g. operating system, MPI, Python3, etc). The second stage is building the source code. Solver parameters such as collision kernel and boundary conditions are determined at compile time. If different settings are desired for these then a separate executable will need to be compiled from source with the appropriate flags set.

Instructions for compilation are provided in the README files of the CPU code. In particular, the FullBuild.sh script contains the steps needed for running the dependency and source compilations. Adjust this file to ensure modules/compilers are loaded correctly for your system.

Modules for Intel compilers on Iris

MODULES(){
module load cmake
module load python/3.12.2

module load intel/2024.2
module load compiler mkl
module load hpcx/2.25.1

export CC=mpicc
export CXX=mpicxx
export OMPI_CC=icx
export OMPI_CXX=icpx
}

Modules for GNU compilers on Iris

MODULES(){
module load cmake
module load python/3.12.2

module load gcc
module load hpcx/2.25.1

export CC=mpicc
export CXX=mpicxx
}

Build for GPU nodes

HemePure can be compiled to run on Nvidia, AMD or Intel GPU architectures. The basic compilation process is the same as the CPU version but please refer to the README pages within the GPU repository for specific details on compilation for your desired hardware.

Sample build script on Romeo

#!/bin/bash
## Compilation/build script for HEMELB
MODULES(){

module use /gpfs/projet/r250119/tools/nvhpc/modulefiles
module load nvhpc-hpcx-cuda12/26.1
export OMPI_CC=gcc
export OMPI_CXX=g++
export CC=mpicc
export CXX=mpicxx
export BASE=$PWD

}

DEPbuild(){
cd dep
rm -rf build
mkdir build
cd build
cmake -DCMAKE_C_COMPILER=${CC} -DCMAKE_CXX_COMPILER=${CXX} ..
make -j  && echo "Done HemeLB Dependencies"

cd ../..
}

SRCbuild_Benchmark(){
cd src
FOLDER=build_PP_Benchmark

rm -rf $FOLDER
mkdir $FOLDER
cd $FOLDER

cmake -DCMAKE_C_COMPILER=${CC} -DCMAKE_CXX_COMPILER=${CXX} -DCMAKE_CXX_FLAGS="-std=c++11 -g -Wno-narrowing" -DCMAKE_EXE_LINKER_FLAGS="-L${BASE}/dep/install/lib -ltirpc" -DCMAKE_SHARED_LINKER_FLAGS="-L${BASE}/dep/install/lib -ltirpc" -DHEMELB_CUDA_AWARE_MPI=ON -DCMAKE_CUDA_ARCHITECTURES="90" -DCMAKE_CUDA_FLAGS="-I${BASE}/dep/install/include/tirpc" -DHEMELB_USE_GMYPLUS=OFF -DHEMELB_USE_MPI_WIN=OFF -DHEMELB_USE_VELOCITY_WEIGHTS_FILE=OFF -DHEMELB_INLET_BOUNDARY=NASHZEROTHORDERPRESSUREIOLET -DHEMELB_WALL_INLET_BOUNDARY=NASHZEROTHORDERPRESSURESBB -DHEMELB_OUTLET_BOUNDARY=NASHZEROTHORDERPRESSUREIOLET -DHEMELB_WALL_OUTLET_BOUNDARY=NASHZEROTHORDERPRESSURESBB ..

make -j && echo "Done HemeLB Source"

cd ../..
}

MODULES
DEPbuild
SRCbuild_Benchmark

Running simulations

For both versions of HemePure, the basic format of the execution step is the same:

mpirun -np N <path/to/executable> -in input.xml -out results

This will run the executable located at the specified path on N MPI ranks. Typically for the CPU code, one rank will correspond to one CPU core, whilst for the GPU code the number of ranks corresponds to the number of GPU cards being deployed. For the CPU code, N>3 is required. The executable will read the input.xml file and run the simulation described within. Output from the simulation is written to the results directory. Note that HemePure will abort a simulation if a directory named results already exists in the specified location. Deleting the existing file or choosing a new output directory name will be required for a subsequent job to write output in the same location.

System requirements of your local machine may require modification of the execution step: e.g. swapping mpirun with mpiexec or srun. Provision of extra execution flags to the execution instruction may also be required to support this e.g. for controlling CPU-GPU binding.

You can practice using the input files contained in this tarball: .

Analysing your results

Information about the simulation performance can be extracted from the data written to screen/file during a simulation. Other details are also summarised in the report.txt file written to the simulation output folder.

HemePure stores output data from the simulation domain in a compressed *.dat file format. To make this human-readable use the hemeXtract tool provided at GitHub - UCL-CCS/hemeXtract . There are multiple options for how files can be converted but the most straightforward is:

hemeXtract -X output.dat > readable-output.txt

Where output.dat is the address of the file you wish to convert and readable-output.txt is the file to which it is converted too.

The readable-output.txt file contains all the data from the simulation. To observe transient evolution of a model, or results at a specific step, it can be easier to split this into separate files for each output step. This can be achieved, for example, with the script https://github.com/UCL-CCS/HemePure/blob/master/cases/UnevenArms/toParaview.sh. This splits the readable-output.txt file into sequentially numbered text files.

These text files can be visualised using Paraview. Due to the format of the files, a couple of extra steps are required to successfully render the files.

1. Import paraview-file-nameXX.txt files: File > Open > Navigate to folder > double-click on paraview-file-name..txt (type GROUP)

2. Click green apply button on LHS

After importing files

3. Change default comma (,) in Field Delimeter Characters to a space ( )

4. Click green apply button on LHS (this is a recurring theme in Paraview)

Correcting for file format

5. Filters > Alphabetical > Table To Points

6. Change Y Columns to 'gridY' and Z columns to 'gridZ'

7. Click green apply button on LHS

Converting to points

8. Ensure RenderView panel is active and click on eye next to TableToPoints1 in Pipeline

9. Change rendering option from 'Solid Colour' to 'velZ'

10. Use Play and rescale options to watch time evolution (if multiple files loaded).

Viewing a domain

From this basic process of loading HemePure output data, more advanced analysis of the data can be conducted. Converting the scalar input data for the velocity components into a vector using a calculator is a common requirement.

Tasks and Submissions

Build HemePure with pressure boundary conditions being applied at both inlets and outlets. The SRCbuild_Benchmark option in the FullBuild.sh located in the CPU version indicates a cmake command that will apply these conditions.

Use a small input to try your execution and analysis process. There is no need to submit the results from this test case, it just for practice and to get familiar with the steps for executing and post-processing HemePure simulations.

For scalabilty, benchmarking may generally focus on the time taken to conduct the simulation itself and not include any pre- or post-processing time. Determine how to extract this information from the HemePure output.

A commonly used metric to evaluate the performance of LBM codes is the number of lattice site updates per second, typically expressed as MLUPS = Millions of Lattice Updates Per Second. Investigate how this metric changes per-core, and in total, and between hardware types. For reference, site counts for a domain are printed in the information written as part of simulation output, and in the report.txt file.

Run the HemePure application with the Aneurysm-VIRTUAL on 1, 2, 4 Iris cluster:
- Analyse the scalability of the results.
- Investigate how the number of MPI ranks used changes the time taken to complete a simulation.
- Examine how simulation stability monitoring impacts your results by uncommenting the <monitoring> section of the input.xml file.
- Investigate the impact of using intrinsics in CPU codes - recompile the CPU code with both -DHEMELB_USE_SSE3=OFF and -DHEMELB_USE_AVX2=OFF set and compare to your original results; repeat with just -DHEMELB_USE_SSE3=ON.
- Submit the results to the teams folder.
Run the HemePure applications with on 1, 2, 4 Romeo GPU nodes (each node has 4 Grace-Hopper CPU-GPU):
- Analyse the scalability of your results.
- Investigate the impact of writing data to file for GPU based simulations - how does the performance change when data is written to file 10x more/less frequently (modify the period variable for each <propertyoutput>), or not at all (comment out the entirety of the <properties> section of the input.xml file).
- Submit the results to the teams folder.
Use an MPI profiling tool (any profiler) to analyze the performance of HemePure application on Romeo and Iris clusters on 4 nodes. Illustrate your the results in the teams presentation, particularly addressing the following questions: What are the 3 most used MPI calls? Are there differences between the two clusters? Do your results change when running on 2 nodes?
Visualize the results using any tool, create a short video or several photos.

Note for Task 3, HemePure uses MPI rank 0 to coordinate the simulation execution. Looking at other ranks will show some more interesting behavior.