...
For more information, please visit https://neko.cfd.
Note: The page may be changed until the competition stats, maybe sure to follow up until the opening ceremony.
Neko presentation to the teams:
...
To build Neko, you will need a Fortran compiler supporting the Fortran-08 standard, autotools, pkg-config, a working MPI installation supporting the Fortran 2008 bindings (mpi_f08), BLAS/LAPACK and JSON-Fortran. Detailed installation instructions can be found in the Neko manual.
Note: For the in-person event, a Fortran compiler supporting the entire Fortran 2008 standard is required; compilers with partial support are not supported. It is not allowed to modify src/common/signal.f90, with the exception of provided patches located in patches/.
Sample build script on PSC:
Build json-fortran:
| Code Block |
|---|
#!/bin/bash
source /jet/packages/oneapi/v2023.2.0/compiler/2023.2.1/env/vars.sh
source /jet/packages/oneapi/v2023.2.0//mpi/2021.10.0/env/vars.sh
export MPIFC=mpiifort
export CC=mpiicc
export FC=$MPIFC
./regen.sh
./configure --prefix=<path>
make
make install |
Note: Intel MPI is recommended, OpenMPI may not be stable, but you are welcome to try.
...
git clone --depth=1 https://github.com/jacobwilliams/json-fortran.git
cd json-fortran && mkdir b && cd b
cmake -DCMAKE_INSTALL_PREFIX=/path/to/installation -DUSE_GNU_INSTALL_CONVENTION=ON ..
make install |
Build Neko:
| Code Block |
|---|
#!/bin/bash #SBATCH# -pLoad RMIntel #SBATCH --nodes=4 #SBATCH --ntasks-per-node=128 #SBATCH -J neko #SBATCH --time=4:00:00 #SBATCH --exclusive module purge source /jet/packages/oneapi/v2023.2.0/compiler/2023.2.1/env/vars.sh source /jet/packages/oneapi/v2023.2.0/mkl/2023.2.0/env/vars.sh HCA=mlx5_0:1 source /jet/packages/oneapi/v2023.2.0//mpi/2021.10.0/env/vars.sh USE_UCX=1 MPIFLAGS="" if [ $USE_UCX -ne 0 ]; then MPIFLAGS+="-genv USE_UCX=$USE_UCX " MPIFLAGS+="-genv UCX_NET_DEVICES ${HCA} " MPIFLAGS+="-genv FI_PROVIDER=mlx " Compilers and MPI libraries. export MPIFC=mpiifort export CC=mpiicc export FC=$MPIFC export PKG_CONFIG_PATH=/path/to/jsonfortran/lib64/pkgconfig:${PKG_CONFIG_PATH} ./regen.sh ./configure --prefix=<path> make make install |
Sample run script:
| Code Block |
|---|
# Untar tgv.zip cd tgv # Load Intel Compilers and MPI libraries. export MPIFC=mpiifort export FC=$MPIFC export LD_LIBRARY_PATH=<path>/json-fortran/lib64:$LD_LIBRARY_PATH:<path to ucx>/lib else MPIFLAGS+="-genv FI_PROVIDER=^mlx " fi cd tgv <path to neko><path>/neko/bin/makeneko ${TEST}.f90 mpirun -np <# procs> $MPIFLAGS <NPROC> -genv USE_UCX=1 -genv UCX_NET_DEVICES mlx5_0:1 -genv FI_PROVIDER=mlx ./neko tgv_Re1600.case |
Sample Sample output: (Update)
| Code Block |
|---|
_ __ ____ __ __ ____ / |/ / / __/ / //_/ / __ \ / / / _/ / ,< / /_/ / /_/|_/ /___/ /_/|_| \____/ (version: 0.68.10-rc1) (build: 20232024-0904-1410 on x86_64-pc-linux-gnu using intelcray) -------Job Information-------- Start time: 1810:4441 / 20232024-0904-1810 Running on: 128256 MPI ranks CPU type : Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHzAMD EPYC 7742 64-Core Processor Bcknd type: CPU Real type : double precision -------------Case------------- Reading case file tgv_Re1600.case -------------Mesh------------- Reading a binary Neko file 32768.nmsh ... -----Material properties------ Read non-dimensional values: ... -----Starting simulation------ T : [ 0.0000000E+00, 0.2000000E+01) dt : 0.5000000E-03 ... ---------------------------------------------------------------- t = 0.0000000E+00 [ 0.00% ] ---------------------------------------------------------------- Time-step: 1 CFL: 0.3970812E-01 dt: 0.5000000E-03 ... ----------------------Postprocessing------------------------------------------ t = 0.2683500E+01 [ 26.84% ] ---------------------------------------------------------------- Time-step: 5368 CFL: 0.4915199E--Writer output01 dt: 0.5000000E-03 ------------Fluid------------- Projection Pressure Proj. vec.: Orig. residual: 2 0.4020491E-05 Pressure Iterations: Start residual: Final residual: 1 0.2061696E-06 0.8489571E-07 X-Velocity File name: field.fld Output number: 2Iterations: Start residual: Final residual: 2 0.8948043E-03 0.1355196E-08 Y-Velocity Iterations: Start residual: Final residual: 2 0.8948043E-03 Writing at time0.1355196E-08 Z-Velocity Iterations: Start residual: Final residual: 2.000500 Output 0.8240080E-03 0.1270342E-08 Elapsed time (s): 0.8980917E+03 Step time: 2.5012810.1604918E+00 --------Postprocessing-------- ! stop at job limit >>> ! saving checkpoint >>> Normal end. |
GPU support:
To compile Neko with GPU support, please follow either the instructions in the manual (Compiling Neko for NVIDIA GPUs and Compiling Neko for AMD GPUs).
Note: A Neko installation can only support one backend. Thus, to run both CPU and GPU experiments, two different builds and installations are necessary.
...
Task & Submission
The aim of this task is to compute as many time steps as possible for the given flow case within a time limit of maximum 15 minutes.
Run Neko with the given input
(View file name tgv.zip neko tgv_Re1600.case) on 4 CPU nodes and submit the results to the team’s folder (standard output,field0.f00000,field0.f00001andfield0.nek5000)
Note: the small input is for you to play around.Run Neko with the given input on 4 GPUs (V100) and submit the results to the team’s folder (see instructions for GPU support above, and the additional information provided with the input file) either CPUs or GPUs.Submit your best result (standard output). Do not submit binary output files nor multiple results.
Note: You are allowed to experiment with different linear solvers in Neko (see the manual) to achieve the fastest runtime. However, all of them are not guaranteed to work for the given case or support all hardware backends in Neko. Furthermore, it is allowed to experiment with different configuration options when building Neko, e.g., enabling device-aware MPI. However, it is not allowed to use single precision, changing the time-step size or any other parameters related to the case.