Versions Compared

Version Old Version 1 New Version Current
Changes made by

Ophir Maor

Niclas Jansson (Unlicensed)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

For more information, please visit https://neko.cfd.

Note: The page may be changed until the competition stats, maybe sure to follow up until the opening ceremony.

Neko presentation to the teams:

...

Building and Running example

Download Neko v0.68.10-rc1, https://github.com/ExtremeFLOW/neko/archive/refs/tags/v0.68.10-rc1.tar.gz.

To build Neko, you will need a Fortran compiler supporting the Fortran-08 standard, autotools, pkg-config, a working MPI installation supporting the Fortran 2008 bindings (mpi_f08), BLAS/LAPACK and JSON-Fortran. Detailed installation instructions can be found in the Neko manual.

Note: For the in-person event, a Fortran compiler supporting the entire Fortran 2008 standard is required; compilers with partial support are not supported. It is not allowed to modify src/common/signal.f90, with the exception of provided patches located in patches/.

Sample build script on PSC:

Build json-fortran:

Code Block
#!/bin/bash
source /jet/packages/oneapi/v2023.2.0/compiler/2023.2.1/env/vars.sh
source /jet/packages/oneapi/v2023.2.0//mpi/2021.10.0/env/vars.sh
export MPIFC=mpiifort
export CC=mpiicc
export FC=$MPIFC

./regen.sh
./configure --prefix=<path>
make
make install

Note: Intel MPI is recommended, OpenMPI may not be stable, but you are welcome to try.

 Sample run script:

git clone --depth=1 https://github.com/jacobwilliams/json-fortran.git
cd json-fortran && mkdir b && cd b
cmake -DCMAKE_INSTALL_PREFIX=/path/to/installation -DUSE_GNU_INSTALL_CONVENTION=ON ..
make install

Build Neko:

Code Block
#!/bin/bash
#SBATCH# -pLoad RMIntel #SBATCH --nodes=4
#SBATCH --ntasks-per-node=128
#SBATCH -J neko
#SBATCH --time=4:00:00
#SBATCH --exclusive
module purge
source /jet/packages/oneapi/v2023.2.0/compiler/2023.2.1/env/vars.sh
source /jet/packages/oneapi/v2023.2.0/mkl/2023.2.0/env/vars.sh

HCA=mlx5_0:1

source /jet/packages/oneapi/v2023.2.0//mpi/2021.10.0/env/vars.sh
USE_UCX=1
MPIFLAGS=""
if [ $USE_UCX -ne 0 ]; then
        MPIFLAGS+="-genv USE_UCX=$USE_UCX "
        MPIFLAGS+="-genv UCX_NET_DEVICES ${HCA} "
        MPIFLAGS+="-genv FI_PROVIDER=mlx "
       Compilers and MPI libraries.
export MPIFC=mpiifort
export CC=mpiicc
export FC=$MPIFC
export PKG_CONFIG_PATH=/path/to/jsonfortran/lib64/pkgconfig:${PKG_CONFIG_PATH}

./regen.sh
./configure --prefix=<path>
make
make install

Sample run script:

Code Block
# Untar tgv.zip
cd tgv
# Load Intel Compilers and MPI libraries.
export MPIFC=mpiifort
export FC=$MPIFC

export LD_LIBRARY_PATH=<path>/json-fortran/lib64:$LD_LIBRARY_PATH:<path
to ucx>/lib
else
        MPIFLAGS+="-genv FI_PROVIDER=^mlx "
fi

cd tgv
<path to neko><path>/neko/bin/makeneko  ${TEST}.f90
mpirun -np <# procs> $MPIFLAGS <NPROC> -genv USE_UCX=1 -genv UCX_NET_DEVICES mlx5_0:1 -genv FI_PROVIDER=mlx ./neko tgv_Re1600.case

 Sample Sample output:

Code Block

    _  __  ____  __ __  ____
   / |/ / / __/ / //_/ / __ \
  /    / / _/  / ,<   / /_/ /
 /_/|_/ /___/ /_/|_|  \____/

 (version: 0.68.10-rc1)
 (build: 20232024-0904-1410 on x86_64-pc-linux-gnu using intelcray)


 -------Job Information--------
 Start time: 1810:4441 / 20232024-0904-1810
 Running on: 128256 MPI ranks
 CPU type  : Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz
 AMD EPYC 7742 64-Core Processor
 Bcknd type: CPU
 Real type : double precision

 -------------Case-------------
 Reading case file tgv_Re1600.case

   -------------Mesh-------------
   Reading a binary Neko file 32768.nmsh
...
   -----Material properties------
   Read non-dimensional values:
...
   -----Starting simulation------
   T  : [  0.0000000E+00,  0.2000000E+01)
   dt :    0.5000000E-03
...
   ----------------------------------------------------------------
   t =   0.0000000E+00                                  [   0.00% ]
   ----------------------------------------------------------------
   Time-step:      1
    CFL:  0.3970812E-01 dt:  0.5000000E-03

...

   ----------------------------------Postprocessing------------------------------
   t =   0.2683500E+01                                  [  26.84% ]
   --------------------------------------------------Writer output------------------------
   Time-step:   5368
    CFL:  0.4915199E-01 dt:  0.5000000E-03

       ------------Fluid-------------
       Projection Pressure
       Proj. vec.:   Orig. residual:
                 2     0.4020491E-05
       Pressure
       Iterations:   Start residual:     Final residual:
                 1     0.2061696E-06       0.8489571E-07
       X-Velocity
       File name: field.fld
  Iterations:   Start residual:     Final residual:
                 2     0.8948043E-03       0.1355196E-08
       Y-Velocity
       Iterations:   Start residual:     OutputFinal numberresidual:
                 2     0.8948043E-03     Writing at time:   2.000500 Output  0.1355196E-08
       Z-Velocity
       Iterations:   Start residual:     Final residual:
                 2     0.8240080E-03       0.1270342E-08
       Elapsed time (s):  2.5012810.8980917E+03 Step time:  0.1604918E+00

       --------Postprocessing--------
   ! stop at job limit >>>
   ! saving checkpoint >>>
   Normal end.

GPU support:

To compile Neko with GPU support, please follow either the generic CUDA instructions at https://github.com/ExtremeFLOW/neko/discussions/540 instructions in the manual (Compiling Neko for NVIDIA GPUs and Compiling Neko for AMD GPUs).

Note: A Neko installation can only support one backend. Thus, to run both CPU and GPU experiments, two different builds and installations are necessary.

...

Task & Submission

The aim of this task is to compute as many time steps as possible for the given flow case within a time limit of maximum 15 minutes.

  1. Run Neko with the given input

    View file
    nametgv.zip
    (neko tgv_Re1600.case) on 4 CPU nodes and submit the results to the team’s folder either CPUs or GPUs.

  2. Submit your best result (standard output, field0.f00000, field0.f00001 and field0.nek5000)
    Note: the small input is for you to play around.

  3. Run Neko with the given input on 4 GPUs (V100) and submit the results to the team’s folder (see instructions for GPU support above, and the additional information provided with the input file)

  4. Run IPM profile or any other MPI profile on 4 nodes, and find the 3 most used MPI calls, show your work in the team interview.

  5. Create a visualization short video with Paraview and show it on your team interview

  6. Change nsamples to e.g. 100 in tgv_Re1600.case

  7. Download all field0.* files to a machine with Paraview (e.g. your laptop)

  8. Open field0.nek5000 in Paraview, don’t forget to select all “Point Arrays” before pressing “Apply”

  9. Try various filters and representations, for example, iso-surfaces, glyphs or volume rendering

    1. The flow field will show more interesting features if the simulation is allowed to run longer (controlled by T_end in the case file).

    10. To save the final animation, follow the instructions at https://docs.paraview.org/en/latest/UsersGuide/savingResults.html#saving-animation

    ). Do not submit binary output files nor multiple results.

Note: You are allowed to experiment with different linear solvers in Neko (see the manual) to achieve the fastest runtime. However, all of them are not guaranteed to work for the given case or support all hardware backends in Neko. Furthermore, it is allowed to experiment with different configuration options when building Neko, e.g., enabling device-aware MPI. However, it is not allowed to use single precision, changing the time-step size or any other parameters related to the case.