/
Getting Started with OpenFOAM for ISC26 SCC (Virtual Part)

Getting Started with OpenFOAM for ISC26 SCC (Virtual Part)

OpenFOAM "Open-source Field Operation And Manipulation" is a C++ toolbox for the development of customized numerical solvers, and pre-/post-processing utilities for the solution of continuum mechanics problems, including computational fluid dynamics (CFD).

Website: https://www.openfoam.com/

Presentation:

Build and Compile (from source)

Download and build OpenFOAM v2412 using the instructions detailed in the following link:

https://gitlab.com/openfoam/core/openfoam/-/blob/OpenFOAM-v2412/doc/Build.md?ref_type=tags

 

Sample build script using HPCX and Intel compilers on Iris cluster. Do not use SP.

#!/bin/bash module load intel/2023.2 compiler mkl module load hpcx/2.25.1 export OMPI_CC=icc export OMPI_CXX=icpc export OMPI_FC=ifort tar xfp OpenFOAM-v2412.tgz tar xfp ThirdParty-v2412.tgz export FOAM_INST_DIR=$PWD source $PWD/OpenFOAM-${openfoam_ver}/etc/bashrc WM_COMPILER=Icc WM_MPLIB=SYSTEMOPENMPI WM_LABEL_SIZE=64 cd $WM_PROJECT_DIR && ./Allwmake -j 16

Tasks and Submissions

This task should be done only on Iris cluster, not on Romeo.

  1. Your task is to simulate the incompressible flow over the DrivAer automotive model, using OpenFOAM. First, get acquainted with the test case and how to run it:
    https://develop.openfoam.com/committees/hpc/-/tree/develop/incompressible/simpleFoam/occDrivAerStaticMesh Download the 65M mesh (polyMesh_65M.tar.gz) & and decompress it to occDrivAerStaticMesh/constant/ folder. Change the endTime to 300 in the ‘system/controlDict.noWrite’ file.

     

    Modify the Allrun file to accommodate the SLURM parameters like below for Iris cluster.

    #SBATCH -p iris #SBATCH --job-name "DrivAer_RANS" #SBATCH --nodes=4 #SBATCH --ntasks-per-node=56 #SBATCH --output=%x-%j.out #SBATCH --error=%x-%j.err #SBATCH --time=1:00:00 #SBATCH --exclusive #SBATCH -d singleton

Run the OpenFOAM (v2412) on 4 nodes from the Iris cluster and submit your best performance, build script and run script. Do not submit multiple results nor binary files.

  1. Analyze the scalability of the simulation over 1, 2 and 4 nodes. Show your analysis in the team’s presentation.

  2. Run any MPI profiler to profile the simulation on 4 nodes. What are the 3 most used MPI calls in terms of number of calls and in terms of total CPU time consumed? Show your analysis in the team’s presentation.

  3. Visualize the results using Paraview.
    To enable solution output for visualization, change writeInterval=500 and endTime=1500 in the ‘system/controlDict.Write’ file, and make sure it is being used by the ‘Allrun’ script.
    Try to reproduce the set of images shown in the reference results, and extract a convergence plot of the drag coefficient from the produced log files.

  4. BONUS task: Modify the mesh decomposition method in 'system/decomposeParDict', and demonstrate how it affects the performance.