Get Started with HPC-X
- Ophir Maor
HPC-X is a comprehensive software package that includes MPI and SHMEM communications libraries. HPC-X also includes various acceleration packages to improve both the performance and scalability of applications running on top of these libraries, including UCX (Unified Communication X), which accelerate the underlying point to point. It also includes HCOLL which accelerates the underlying collective operations used by the MPI/PGAS languages. In addition, HPC-X contains SHARP library. For more information about HPC-X see here.
HPC-X is installed by default as a part of Mellanox OFED, or can be installed by itself.
HPC-X installation as a part of Mellanox OFED
1. Download and Mellanox OFED on your server(s), untar and install.
./mlnxofedinstall [options]See here for example.
2. Export libraries for OpenMPI and OSU benchmarks (if needed).
For example:
$ export PATH=/usr/mpi/gcc/openmpi-4.0.2rc3/bin:/usr/mpi/gcc/openmpi-4.0.2rc3/tests/osu-micro-benchmarks-5.3.2:$PATH
$ export LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi-4.0.2rc3/lib:$LD_LIBRARY_PATH
3. Check that you can use mpirun and osu:
$ which mpirun
/usr/mpi/gcc/openmpi-4.0.2rc3/bin/mpirun
$ which osu_latency
/usr/mpi/gcc/openmpi-4.0.2rc3/tests/osu-micro-benchmarks-5.3.2/osu_latency
4. Check HPC-X versions per package
$ ofed_info -s
MLNX_OFED_LINUX-4.7-3.2.9.0:
$ rpm -qa |grep hcoll
hcoll-4.4.2938-1.47329.x86_64
$ rpm -qa |grep ucx
ucx-knem-1.7.0-1.47329.x86_64
ucx-1.7.0-1.47329.x86_64
ucx-ib-1.7.0-1.47329.x86_64
ucx-devel-1.7.0-1.47329.x86_64
ucx-ib-cm-1.7.0-1.47329.x86_64
ucx-rdmacm-1.7.0-1.47329.x86_64
ucx-cma-1.7.0-1.47329.x86_64
$ rpm -qa |grep sharp
sharp-2.0.0.MLNX20190922.a9ebf22-1.47329.x86_64
4. Allocate nodes and test
$ mpirun -np 28 osu_alltoall
# OSU MPI All-to-All Personalized Exchange Latency Test v5.3.2
# Size Avg Latency(us)
1 4.10
2 4.89
4 4.95
8 5.58
16 7.00
...HPC-X installation as a separate package
Download and Mellanox OFED on your server(s), untar and install.
./mlnxofedinstall [options]See here for example.
2. Download and untar HPC-X package , for example, HPC-X 2.7 for RHEL 7.6
$wget https://www.mellanox.com/downloads/hpc/hpc-x/v2.7/hpcx-v2.7.0-gcc-MLNX_OFED_LINUX-5.1-0.6.6.0-redhat7.6-x86_64.tbz .
$tar xf hpcx-v2.7.0-gcc-MLNX_OFED_LINUX-5.1-0.6.6.0-redhat7.6-x86_64.tbz
3. Install environment modules
$ yum -y install environment-modules
...
4. exit and ssh again to the session.
5. Use hpcx module
$ cd hpcx-v2.7.0-gcc-MLNX_OFED_LINUX-5.1-0.6.6.0-redhat7.6-x86_64/modulefiles/
$ module use $PWD
$ module load hpcx
6. Run basic OSU test
mpirun -np 2 --allow-run-as-root $HPCX_OSU_DIR/osu_latency
...
For more info refer to HPC-X documentation here.