OpenFOAM (Open-source Field Operation And Manipulation) is a C++ toolbox for the development of customized numerical solvers, and pre-/post-processing utilities for the solution of continuum mechanics problems, including computational fluid dynamics (CFD). OpenFOAM

Presentation:

Build and Compile (from source)

This task uses SPUMA, a fork of OpenFOAM that builds and runs on CPUs, NVIDIA GPUs, and AMD GPUs from a single code base. There is one setup for everyone: the same case, mesh, fvSchemes, fvSolution, and simpleFoam solver. Each team chooses whether to build and run on CPUs or on GPUs — only the build backend and the launch options differ.

Build constraints

• Base version: SPUMA 0.2-v2412 (OpenFOAM-v2412 fork).

• Precision: WM_PRECISION_OPTION=DP (SP/SPDP not accepted).

• Label size: WM_LABEL_SIZE=32

• Stock solver/models: the SPUMA source tree, simpleFoam, the mesh, and fvSchemes/fvSolution must remain unchanged. The only fields you may edit in system/include/caseDefinition are nCores, decompositionMethod, and nHierarchical. Compile-flag changes under wmake/rules/ are allowed and must be submitted as patch files.

0. Prerequisites (adapt to your cluster modules)

• CPU-only: a C++14 compiler (GCC) and an MPI library. Nothing GPU-specific.

• NVIDIA: NVIDIA HPC SDK ≥ 22.3 (nvc++; tested 24.3) + CUDA ≥ 11.6 (bundled) + a GPU-aware MPI. Compute capability: A100 = 80, H100 = 90, L40S = 89.

• AMD: ROCm ≥ 6.0.3 (amdclang/hipcc) + a ROCm-aware MPI. Architecture: MI200/MI250X = gfx90a, MI300A/MI300X = gfx942.

• Common: cmake, flex, bison, make, git, zlib.

1. Build SPUMA — choose ONE backend

No separate vanilla-OpenFOAM or ThirdParty build is required for this case. SPUMA is a self-contained OpenFOAM-v2412 fork.

Create etc/prefs.sh with the options for your chosen backend, then source the environment (etc/bashrc) and build, as described below.
Note: FOAM_SIGFPE=false is required for the GPU backends because floating-point-exception trapping breaks GPU kernels.

Option A — CPU-only (etc/prefs.sh)

Option B — NVIDIA (CUDA) (etc/prefs.sh)

Option C — AMD (HIP) (etc/prefs.sh)

Verify the solver built:

Notes: compilation needs the compiler/toolkit but not a live GPU (you can build on a CPU-only node). With no ThirdParty present the build simply skips it (as above). If you do provide a ThirdParty directory (e.g. for scotch/kahip), the build will also compile it and ADIOS2/CGAL may emit non-fatal errors — the core libraries and simpleFoam still build. -k keeps the build going past the optional boxTurb/randomProcesses app (needs FFTW, unused here).

2. Get the case from the SPUMA GitLab

The complete case ships in the SPUMA repository at tutorials/incompressible/simpleFoam/occDrivAerStaticMesh (already present after the recursive clone in Step 1). To fetch only the case, use a sparse checkout from the SPUMA GitLab:

The 65M mesh is too large for git; download it from the link in the case README (Zenodo) and unpack it into constant/:

Set the run length to 1000 iterations in system/controlDict (tutorial default is 4000); the drag-coefficient validation uses the [800, 1000] window:

3. Decomposition

Edit system/include/caseDefinition: set nCores to the number of MPI ranks and make nHierarchical multiply to it. Map one rank per CPU core for CPU runs, or one rank per GPU for GPU runs.

For 65M cells, 8 GPUs gives ~8.2M cells/GPU — the efficient regime reported in the SPUMA paper.

4. Run

SPUMA executables accept two extra flags, -pool <type> -poolSize <GiB-per-rank>, already wired into the provided Allrun via the poolType/poolSize variables at the top of the script. The pool size is per rank.

• GPU runs: always use fixedSizeMemoryPool; the default dummyMemoryPool is ~4× slower. Size it to fit one rank on one GPU — the 65M/8-GPU run used ~10 GiB, so poolSize=32 is safe on 64–80 GB GPUs.

• CPU runs: use dummyMemoryPool (or fixedSizeMemoryPool with a small per-rank size, e.g. 1–2 GiB), because dozens of ranks share the node's host RAM — a large per-rank pool will exhaust memory.

Set the launcher and pool variables at the top of Allrun:

The Allrun pipeline is: decomposePar → restore0Dir → renumberMesh → potentialFoam → applyBoundaryLayer → simpleFoam.

GPU binding (GPU runs only). Use one rank per GPU; SPUMA binds device = rank % nDevices automatically when all node GPUs are visible to each rank. With Slurm, srun --gpus-per-task=1 gives a clean 1-rank-per-GPU mapping. If you launch with mpirun and ranks pile onto GPU 0, add a small wrapper that sets the visible device to the local rank:

and launch mpirun -np N ./gpuwrap.sh simpleFoam ....

Example Slurm batch (GPU, 1 node, 8 GPUs):

Tip: decomposePar of the 65M mesh is serial and takes a few minutes. To iterate quickly, reuse an existing decomposition: after the first run, comment out decomposePar/renumberMesh in Allrun and restart from potentialFoam.

5. Validation and headline metric

• Headline: the last ClockTime = X s after the Time = 1000 block in logFiles/50_simpleFoam.*.log (setup phases — decomposePar, renumberMesh, potentialFoam, applyBoundaryLayer — are not counted). SPUMA also prints Average wall-clock time per time step via the FOwallClockTimeStatistics function object; report that too.

• Cd validation: the mean Cd over iterations [800, 1000] from postProcessing/forceCoeffsAll/*/coefficient.dat (column 2 = Cd) must be within ±1.5% of the reference value 0.2699. This reference was established on this exact setup (SPUMA GAMG, 65M mesh, 8× H100 GPUs, 1000 iterations); CPU-only and GPU SPUMA builds use identical dictionaries and agree to <1% on the mean Cd, so the same reference and tolerance apply to both. (For context the instantaneous Cd oscillates with a standard deviation of ~2.3% over the window, which is why the validation is on the mean.)

• Submit: your build script, run script, 50_simpleFoam.*.log, the edited caseDefinition, and a copy of coefficient.dat. Submit a single result; no binaries.

6. Troubleshooting (issues encountered during bring-up)