/
Getting Started with Secret App for ISC26 SCC (In-Person Part)

Getting Started with Secret App for ISC26 SCC (In-Person Part)

Introduction

SOD2D (Spectral high-Order coDe 2 solve partial Differential equations) will be our secret app for ISC26 SCC.

Wiki: https://ceec-coe.eu/sod2d/

Git: https://gitlab.com/bsc_sod2d/sod2d_gitlab

The code is designed to simulate scale-resolving simulations of turbulent flows, both at compressible and incompressible regimes. Such simulations can quickly reach hundreds of millions of DoFs, and therefore require that the code be able to scale across a vast number of computational resources.

With that in mind, the implemented algorithms were implemented with multi-GPU performance in mind: the code is written in a combination of Fortran+OpenACC+MPI that allows for excellent performance on EuroHPC systems that support such a combination.

Relevant to the competition, key aspects of the incompressible algorithm are the following:

  • Semi-implicit time-integration, requiring a linear solver for NS equation diffusion terms

  • Fractional step method: closure for the pressure problem requires solving a Poisson system

Currently, SOD2D only offers a Flexible Preconditioned Conjugate Gradient (FPCG) solver option for both scenarios, with an internal fixed preconditioner.

Building SOD2D

  1. Install the requirements:

    • CMake 3.15+

    • NVHPC

    • MPI

    • HDF5

      • For HDF5, we suggest either a SPACK installation or a manual build of HDF5/2.0.0 using CMake. Remember to enable parallel support and Fortran bindings when building HDF5!

  2. Download the source:

    git clone --recursive https://gitlab.com/bsc_sod2d/sod2d_gitlab.git

  1. Building partitioner and sod2d using nvhpc

    module load nvhpc-hpcx-<version> module load hdf5-<version> mkdir build_p8 && cd build_p8 cmake -DUSE_GPU=ON -DTOOL_MESHPART=ON -DUSE_PORDER=8 .. make -j 16

  2. Partitioning the 8th order case

  • Change "num_partitions":4 in part.json to # GPUs you want to run with and partition the input. We use 4 GPUs in this example.

    mpirun -np 1 ./build_p8/tool_meshConversorPar/tool_meshconversorPar part.json

  1. Run SOD2D

    mpirun -np 4 $MPI_FLAGS ./build_p8/src/app_sod2d/sod2d TGVSolverIncomp.json

  2. Sample output (sod2d_tgv_p8_44M.log)

    --| Full commit ID: a92afd70897fd1eb5abe40d685088028eb1c9203 --| Branch: master --| Commit date: 2026-05-26 09:39:15 +0000 --| Cmake conf date: 2026-06-05 16:43:41 --| Current case settings: -------------------------------------------- # Constants: rp: 4 rp_vtk: 4 rp_avg: 8 porder: 8 flag_real_diff: 1 flag_diff_suth: 0 ... step 5271 time: 19.85394 s (dt 6.6645248E-03 s) --|[pres] CG, iters: 21 tol 1.0948631853291204E-002 --|[veloc] CG, iters: 5 tol 4.6438450396965132E-003 --| time EK eps_S eps_D eps_T max(Ma) 19.92273712 0.02118932 0.00194766 0.00000067 0.00194834 0.00000000 step 5281 time: 19.92274 s (dt 6.8797367E-03 s) --|[pres] CG, iters: 9 tol 9.9546566369443333E-003 --|[veloc] CG, iters: 4 tol 8.5791771352320707E-003 --| time EK eps_S eps_D eps_T max(Ma) 19.99145508 0.02105641 0.00192472 0.00000065 0.00192536 0.00000000 step 5291 time: 19.99146 s (dt 7.3038689E-03 s) --| Time integration finished at step: 5293 | time: 20.00620

Task and Submissions

  1. Run the application with the given input.

    1. For SCC, the task will be to execute the given inputs with SOD2D’s incompressible model. Apart from the non-partitioned meshes, the following JSON scripts will be provided:

      • Partition tool configuration

      • Incompressible TGV case configuration

  2. Submit your best performance (stdout), sod2d_tgv_p8_44M-xxx.log, timer_tgv_p8_44M-xxx.dat, TGVSolverIncomp.json, build script and run script. Do not submit multiple results nor binary files.


NOTES:

  • Depending on your cmake system, you may need to add “-DCMAKE_POLICY_VERSION_MINIMUM=3.5" related to JSON-Fortran

  • For the 8th order case, “-DUSE_PORDER=8”

  • For multi-node runs using IB and NVIDIA GPUs, create “mpi_bind.sh” as follows:

#!/bin/bash case ${OMPI_COMM_WORLD_LOCAL_RANK} in 0) export CUDA_VISIBLE_DEVICES=0 export OMPI_MCA_btl_openib_if_include=mlx5_0:1 export UCX_NET_DEVICES=mlx5_0:1 numactl --membind=0 "$@" ;; 1) export CUDA_VISIBLE_DEVICES=1 export UCX_NET_DEVICES=mlx5_1:1 export OMPI_MCA_btl_openib_if_include=mlx5_1:1 numactl --membind=1 "$@" ;; 2) export CUDA_VISIBLE_DEVICES=2 export UCX_NET_DEVICES=mlx5_4:1 export OMPI_MCA_btl_openib_if_include=mlx5_4:1 numactl --membind=2 "$@" ;; 3) export CUDA_VISIBLE_DEVICES=3 export UCX_NET_DEVICES=mlx5_5:1 export OMPI_MCA_btl_openib_if_include=mlx5_5:1 numactl --membind=3 "$@" ;; esac

then run:

mpirun -np N --map-by ppr:4:node:PE=20 ./mn5_bind.sh ./sod2d TGVSolverIncomp.json

References

https://www.sciencedirect.com/science/article/pii/S0010465523004125