This page is an overview, configuration and tasks for ISC22 Student Cluster Competition teams.

References

• https://www.incompact3d.com/

• GitHub - incompressible solver for the Navier-Stokes equations

• Paper describing Xcompac3d:

Open

Overview

Xcompact3d is a Fortran90 MPI based, finite difference high-performance framework for solving the Navier-Stokes equations and associated scalar transport equations. Dedicated to Direct and Large Eddy Simulations (DNS/LES) for which the largest turbulent scales are simulated, it can combine the versatility of industrial codes with the accuracy of spectral codes. Its user-friendliness, simplicity, versatility, accuracy, scalability, portability and efficiency makes it an attractive tool for the Computational Fluid Dynamics community.

XCompact3d is currently able to solve the incompressible and low-Mach number variable density Navier-Stokes equations using sixth-order compact finite-difference schemes with a spectral-like accuracy on a monobloc Cartesian mesh.  It was initially designed in France in the mid-90's for serial processors and later converted to HPC systems. It can now be used efficiently on hundreds of thousands CPU cores to investigate turbulence and heat transfer problems thanks to the open-source library 2DECOMP&FFT (a Fortran-based 2D pencil decomposition framework to support building large-scale parallel applications on distributed memory systems using MPI; the library has a Fast Fourier Transform module).
When dealing with incompressible flows, the fractional step method used to advance the simulation in time requires to solve a Poisson equation. This equation is fully solved in spectral space via the use of relevant 3D Fast Fourier transforms (FFTs), allowing the use of any kind of boundary conditions for the velocity field. Using the concept of the modified wavenumber (to allow for operations in the spectral space to have the same accuracy as if they were performed in the physical space), the divergence free condition is ensured up to machine accuracy. The pressure field is staggered from the velocity field by half a mesh to avoid spurious oscillations created by the implicit finite-difference schemes. The modelling of a fixed or moving solid body inside the computational domain is performed with a customised Immersed Boundary Method. It is based on a direct forcing term in the Navier-Stokes equations to ensure a no-slip boundary condition at the wall of the solid body while imposing non-zero velocities inside the solid body to avoid discontinuities on the velocity field. This customised IBM, fully compatible with the 2D domain decomposition and with a possible mesh refinement at the wall, is based on a 1D expansion of the velocity field from fluid regions into solid regions using Lagrange polynomials or spline reconstructions. In order to reach high velocities in a context of LES, it is possible to customise the coefficients of the second derivative schemes (used for the viscous term) to add extra numerical dissipation in the simulation as a substitute of the missing dissipation from the small turbulent scales that are not resolved. 

For more details about the numerical methods and parallelisation strategy used in Xcompact3d:

Wind Turbine Simulations

Modern large–scale offshore wind farms consist of multiple turbines clustered together usually in
well–structured formations. Such a clustering exhibits a number of drawbacks during the operation
of the wind farm, as some of the downstream turbines will inevitably have to operate within the wake
of the upstream ones. A wind turbine operating within a wake field is an issue for two reasons. First,
the apparent reduction of its power output due to the wind speed de-acceleration and second an
increase of the fatigue loads due to experiencing the upstream wake-laden turbulence. Power
losses due to wake effects were recently reported to be in the order of 10–25 % while the fatigue–
related failures were reported to be around the same levels owing to a limited understanding of the
offshore turbulence. To summarise, currently installed wind farms do not produce as much power as
expected.

Open

Photograph of the Horns Rev 2 offshore wind farm on 25 January 2016 at 12:45 UTC. From Hasager, C. B., Nygaard, N. G., Volker, P. J., Karagali, I., Andersen, S. J., & Badger, J. (2017). Wind farm wake: The 2016 Horns Rev photo case. Energies, 10(3), 317.

In order to optimise the power output of wind farms, there is a clear need for reliable physics-based
simulation methods that can faithfully replicate realistic scenarios during operational conditions, using wind farm simulators (WFS) such as Winc3d.

Introduction to XCompact3D

The slides:

Open

Configuration Example

First, get the source code by cloning the git repository (you want to work with version 4.0):

Load gcc or icc and you may use your choice of MPI like Intel MPI, Open MPI or HPC-X.

To compile the code, you need to set-up the Makefile accordingly:

Set below variables in Makefile for the Intel compiler (and adjust the variable FC accordingly, it is the wrapper command for the Fortran compiler)

Set below variables in Makefile for the GNU compiler (and adjust the variable FC accordingly, it is the wrapper command for the Fortran compiler)

Then

On successful completion you will have an executable called xcompact3d.

Running the wind turbine case

The bulk of the tasks is based on a wind turbine case for which two generic wind turbines are aligned and subject to an incoming uniform wind. The files required for this case are provided in the Tasks section.

The setup is similar to the following experiment:

Open

From Bartl, J., & Sætran, L. (2017). Blind test comparison of the performance and wake flow between two in-line wind turbines exposed to different turbulent inflow conditions. Wind energy science, 2(1), 55-76.

Tunables Parameters

For the competition, you will only have to change two parameters (in the input.i3d file): the domain decomposition layout (the dimension of the pencils).

Size of the 2D pencils (input.i3d file)

When p_row and p_col are set to zero, the size of the pencils will be automatically decided during the execution. The simulations are performed with [p_row x p_col] MPI_processes (with one MPI_process per core). For instance, if you want to use 128 cores/MPI_processes, you can have p_row x p_col = 2 x 64, 4 x 32, 8 x 16, 16 x 8, 32 x 4, 62 x 2 (p_row and p_col should be equal or larger than 2). For the competition, it is advised to try difference values for best performance.

For debugging or quick testing, you can reduce ilast value in iput.i3d but make sure it is set to 2000 for the final test.

Tasks and Submissions

Input files will be provided during the competition.

For the competition, decompress WindTurbines.tgz in the examples folder.

Performance

Identify the best configuration to achieve the fastest wall clock time.

You can change the Fortran compiler and associated option, the MPI library, the number of MPI_processes per node and the size for the 2D pencils (p_row, p_col). Compiler options can be changed in the Makefile with the variable FFLAGS.

Submit the Makefile, job submission file, input.i3d and the log file of your best run.