Getting started with Neko for ISC24 SCC

Owned by Ophir Maor
Last updated: Mar 08, 2024
3 min read

Overview

Neko is a portable framework for high-order spectral element flow simulations. Written in modern Fortran, Neko adopts an object-oriented approach, allowing multi-tier abstractions of the solver stack and facilitating various hardware backends ranging from general-purpose processors, CUDA and HIP enabled accelerators to SX-Aurora vector processors. Neko has its roots in the spectral element code Nek5000 from UChicago/ANL, from where many of the namings, code structure and numerical methods are adopted.

For more information, please visit https://neko.cfd.

Note: The page may be changed until the competition stats, maybe sure to follow up until the opening ceremony.

 

Neko presentation to the teams:

 

Presentation file:

Building and Running example

Download Neko v0.6.1, https://github.com/ExtremeFLOW/neko/archive/refs/tags/v0.6.1.tar.gz.

Sample build script on PSC:

#!/bin/bash source /jet/packages/oneapi/v2023.2.0/compiler/2023.2.1/env/vars.sh source /jet/packages/oneapi/v2023.2.0//mpi/2021.10.0/env/vars.sh export MPIFC=mpiifort export CC=mpiicc export FC=$MPIFC ./regen.sh ./configure --prefix=<path> make make install

 

Note: Intel MPI is recommended, OpenMPI may not be stable, but you are welcome to try.

 

 Sample run script:

#!/bin/bash #SBATCH -p RM #SBATCH --nodes=4 #SBATCH --ntasks-per-node=128 #SBATCH -J neko #SBATCH --time=4:00:00 #SBATCH --exclusive module purge source /jet/packages/oneapi/v2023.2.0/compiler/2023.2.1/env/vars.sh source /jet/packages/oneapi/v2023.2.0/mkl/2023.2.0/env/vars.sh HCA=mlx5_0:1 source /jet/packages/oneapi/v2023.2.0//mpi/2021.10.0/env/vars.sh USE_UCX=1 MPIFLAGS="" if [ $USE_UCX -ne 0 ]; then MPIFLAGS+="-genv USE_UCX=$USE_UCX " MPIFLAGS+="-genv UCX_NET_DEVICES ${HCA} " MPIFLAGS+="-genv FI_PROVIDER=mlx " export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path to ucx>/lib else MPIFLAGS+="-genv FI_PROVIDER=^mlx " fi cd tgv <path to neko>/bin/makeneko ${TEST}.f90 mpirun -np <# procs> $MPIFLAGS ./neko tgv_Re1600.case

 

 Sample output:

_ __ ____ __ __ ____ / |/ / / __/ / //_/ / __ \ / / / _/ / ,< / /_/ / /_/|_/ /___/ /_/|_| \____/ (version: 0.6.1) (build: 2023-09-14 on x86_64-pc-linux-gnu using intel) -------Job Information-------- Start time: 18:44 / 2023-09-18 Running on: 128 MPI ranks CPU type : Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz Bcknd type: CPU Real type : double precision -------------Case------------- Reading case file tgv.case ... --------Postprocessing-------- --------Writer output--------- File name: field.fld Output number: 2 Writing at time: 2.000500 Output time (s): 2.501281 Normal end.

 

GPU support:

To compile Neko with GPU support, please follow the generic CUDA instructions at https://github.com/ExtremeFLOW/neko/discussions/540

Note: A Neko installation can only support one backend. Thus, to run both CPU and GPU experiments, two different builds and installations are necessary.

Tasks & Submissions

  1. Run Neko with the given input (neko tgv_Re1600.case) on 4 CPU nodes and submit the results to the team’s folder (standard output, field0.f00000, field0.f00001 and field0.nek5000)
    Note: the small input is for you to play around.

  2. Run Neko with the given input on 4 GPUs (V100) and submit the results to the team’s folder (see instructions for GPU support above, and the additional information provided with the input file)

  3. Run IPM profile or any other MPI profile on 4 nodes, and find the 3 most used MPI calls, show your work in the team interview.

  4. Create a visualization short video with Paraview and show it on your team interview

    1. Change nsamples to e.g. 100 in tgv_Re1600.case

    2. Download all field0.* files to a machine with Paraview (e.g. your laptop)

    3. Open field0.nek5000 in Paraview, don’t forget to select all “Point Arrays” before pressing “Apply”

    4. Try various filters and representations, for example, iso-surfaces, glyphs or volume rendering

      1. The flow field will show more interesting features if the simulation is allowed to run longer (controlled by T_end in the case file).

    5. To save the final animation, follow the instructions at https://docs.paraview.org/en/latest/UsersGuide/savingResults.html#saving-animation