...

SeisSol is a software package for simulating wave propagation and dynamic rupture based on the arbitrary high-order accurate derivative discontinuous Galerkin method (ADER-DG).

Website: https://seissol.org/

Documentation: https://seissol.readthedocs.io/en/latest/

Build

...

Building SeisSol

The list of necessary dependencies to install and compile can be found under https://seissol.readthedocs.io/en/latest/compiling-seissol.html (TODO: The installation in the home directory, as described in there, is outdated). In particular, you will need to install:

• Python

• MPI

• Hdf5

• Netcdf

• yaml-cpp

• easi https://github.com/SeisSol/easi

  • ASAGI https://github.com/TUM-I5/ASAGI

  • Lua

• Eigen

• ParMETIS

• libxsmm

• PSpaMM https://github.com/SeisSol/PSpaMM

For GPUs, you will also need:

• AdaptiveCpp

• gemmforge https://github.com/SeisSol/gemmforge

• chainforge https://github.com/SeisSol/chainforge

Alternatively, we provide a Spack environment under https://github.com/SeisSol/seissol-spack-aid .

For SeisSol itself, you will need to set the architecture you are going to use—since our matrix kernel generators usually require that information. That can be done via the HOST_ARCH parameter. On x86 architectures without AVX512 support, choose either hsw , rome , or milan as host architecture. With AVX512 support, choose skx or bergamo . On ARM machines, we have the dummy targets neon , sve128 , sve256 , and sve512 , respectively. Setting these will enable the instruction generation for the respective SIMD architecture instructions. You should also set your CPU by the -mcpu parameter.

For GPUs, you will also need to set the DEVICE_ARCH and DEVICE_BACKEND parameters.

Testing Your Installation

Once the build is finished, you will obtain two binaries:

• SeisSol_Release_ARCH

• SeisSol_proxy_Release_ARCH

To test your installation, you can do so within two steps:

• Run SeisSol_proxy_Release_ARCH 100000 100 all . This command will give you an idealized performance figure, and also run through all kernels once.

• Run SeisSol_Release_ARCH with a parameter file. We provide reference values for all scenarios in the https://github.com/SeisSol/precomputed-seissol repository which you could use to test your installation.

Pinning and Performance Considerations

For optimal performance, we recommend using one process per NUMA domain. If you use the GPU version, you will need one process per GPU.

Furthermore, we recommend keeping one CPU core per process free, to be used for the so-called communication thread. It will be used for advancing MPI communication and IO instead. To do so, you will need to set some environment variables, similar to the following snippet:

THREADS_PER_TASK=16 # TODO: set
CPU_HYPERTHREADING=2 #TODO: set
NUM_CORES=$(expr $CPUS_PER_TASK / $CPU_HYPERTHREADING)
NUM_COMPUTE_CORES=$(expr $NUM_CORES - 1)
export OMP_NUM_THREADS="$(expr $NUM_COMPUTE_CORES \* $CPU_HYPERTHREADING)"
export OMP_PLACES="cores($NUM_COMPUTE_CORES)"
export PROC_BIND=spread
export MP_SINGLE_THREAD=no
unset KMP_AFFINITY

Alternatively, you can also disable the communication thread, by setting SEISSOL_COMMTHREAD=0 .

Tasks and Submissions

1. Run the application on 4 CPU nodes and submit the results

...