This post describes how the performance of GRID benchmark built with HPC-X 2.0 was enhanced by enabling UCX.
...
- Grid Testing and Profiling
- Grid: data parallel library for QCD
- Grid: A next generation data parallel C++ QCD library
- Grid on Github
- Understanding Tag Matching Offload on ConnectX-5 Adapters
Overview
What is GRID
Grid is a C++ library, Developed by Peter Boyle (U. of Edinburgh) et al, for Lattice Quantum Chromodynamics (Lattice QCD) calculations that is available on github and is designed to exploit parallelism at all levels:
...
Building the library is straightforward. A number of tests and benchmarks are built along with it. While building the library it also compiles a set of tests and benchmarks, one of which exercises the library thoroughly and prints out performance information.
HPC- X
HPC-X is the Mellanox application accelerator software package that includes the communications libraries you need to get the most out of your Mellanox InfiniBand interconnect. The package may be downloaded here. Building and running applications with OpenMPI and other support libraries that are provided in HPC-X is easy, as the following write-up shows.
- UCX introduced two new accelerated transports, dc_x and rc_x, that use enhanced verbs.
- The best transports for GRID turned out to be rc_x and rc
- In the mpirun example, we enabled hardware tag matching (UCX_RC_VERBS_TM_ENABLE=y) that perform the tag matching lookup in hardware instead of software.
- The selection of a pml (point-to-point management layer): UCX pml and the MXM pml are mutually exclusive; thus, MXM was disabled
...
Configuration
Obtaining Grid
1. Download Grid from Github. You can either download the zip file or clone the software directly onto your system. For the DiRAC benchmark, a slightly different tarball was provided.
...