Nalu is a Low-Mach Number turbulent flow simulator for energy applications.
This post describes the steps followed to build it on Thor, a Linux cluster with Intel Xeon (Broadwell) processor and Mellanox EDR InfiniBand interconnect.
...
| Table of Contents |
|---|
References
Installation
First, consult the base document at http://nalu.readthedocs.io/en/latest/source/user/build_manually.html, specifically following document , specifically the Linux section. The
Note that the two subsections with heading Nalu Releases aren't especially clear, but the build will work if one follows the directions for "Head Code Base". Many packages need to be built before one has a working Nalu executable.
1. Set up the build environment
...
...
a. Load the compiler and HPC-X modules you wish to use
| Code Block |
|---|
module load intel/compiler/2017.4.196 hpcx-1.9/icc-2017 |
...
export CC=icc |
...
export CXX=icpc |
...
export FC=ifort |
...
export F90=ifort |
...
export F77=ifort |
...
export OMPI_CC=icc |
...
export OMPI_CXX=icpc |
...
export OMPI_FC=ifort |
...
export OMPI_F77=ifort |
...
b. Replace the path that follows with whatever is appropriate for you
| Code Block |
|---|
cd /mnt/beegfs3/gerardo/nrel_esif/Nalu/ |
...
mkdir build |
...
cd build |
...
export nalu_build_dir=$PWD |
...
mkdir $nalu_build_dir/packages |
...
mkdir $nalu_build_dir/install |
...
mkdir $nalu_build_dir/install/lib |
...
cd $nalu_build_dir/packages |
...
export nalu_install_dir=$nalu_build_dir/install |
2. Install
...
CMake
| Code Block |
|---|
wget http://www.cmake.org/files/v3.1/cmake-3.1.0-rc2.tar.gz |
...
tar zxvf cmake-3.1.0-rc2.tar.gz |
...
cd cmake-3.1.0-rc2/ |
...
./configure --prefix=$nalu_build_dir/install |
...
make |
...
make install |
...
Make sure your newly-built CMake can be found
| Code Block |
|---|
export PATH=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build/install/bin:$PATH |
3. Install SuperLU
...
| Code Block |
|---|
cd $nalu_build_dir/packages |
...
wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz |
...
tar zxvf superlu_4.3.tar.gz |
...
cd $nalu_build_dir/packages/SuperLU_4.3 |
...
cp MAKE_INC/make.linux make.inc |
...
Edit make.inc so that the following macros are defined as indicated:
| Code Block |
|---|
# PLAT = _x86_64 |
...
# SuperLUroot = /mnt/beegfs3/gerardo/nrel_esif/Nalu/build/packages/SuperLU_4.3 |
...
# BLASLIB = -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_blas95_lp64 -lmkl_core -lmkl_rt -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_avx2 -liomp5 |
...
# CC = mpicc |
...
# FORTRAN = mpif77 |
Create directory and make
| Code Block |
|---|
mkdir $nalu_build_dir/install/SuperLU_4.3 |
...
mkdir $nalu_build_dir/install/SuperLU_4.3/lib |
...
mkdir $nalu_build_dir/install/SuperLU_4.3/include |
...
make |
...
cp SRC/*.h $nalu_build_dir/install/SuperLU_4.3/include |
...
Note: The following copy step is missing from the directions in the base document, it will trip you up if not done.
| Code Block |
|---|
cp -p lib/libsuperlu_4.3.a $nalu_build_dir/install/SuperLU_4.3/lib |
4. Install
...
libml2
| Code Block |
|---|
cd $nalu_build_dir/packages |
...
wget http://www.xmlsoft.org/sources/libxml2-2.9.2.tar.gz |
...
tar zxvf libxml2-2.9.2.tar.gz |
...
cd $nalu_build_dir/packages/libxml2-2.9.2 |
...
CC=mpicc CXX=mpicxx ./configure -without-python --prefix=$nalu_build_dir/install |
...
make -k install |
5. Install Boost
...
| Code Block |
|---|
cd $nalu_build_dir/packages |
...
wget http://iweb.dl.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.gz |
...
tar zxvf boost_1_55_0.tar.gz |
...
cd $nalu_build_dir/packages/boost_1_55_0 |
...
echo "using mpi : `which mpicxx` ;" >> ./tools/build/v2/user-config.jam |
...
./bootstrap.sh --prefix=$nalu_build_dir/install --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception |
...
./b2 -j 4 2>&1 | tee boost_build_one |
...
./b2 -j 4 install 2>&1 | tee boost_build_instal |
6. Install
...
Yaml
| Code Block |
|---|
cd $nalu_build_dir/packages |
...
git clone https://github.com/jbeder/yaml-cpp |
...
cd $nalu_build_dir/packages/yaml-cpp |
...
mkdir build |
...
cd build |
...
cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_CXX_FLAGS=-std=c++11 -DCMAKE_CC_COMPILER=mpicc -DCMAKE_INSTALL_PREFIX=$nalu_build_dir/install .. |
...
make |
...
make install |
7. Install
...
Zlib
Note that the base document above calls for downloading http://zlib.net/zlib-1.2.8.tar.gz, which is no longer available
| Code Block |
|---|
cd $nalu_build_dir/packages wget http://zlib.net/zlib-1.2.11.tar.gz |
...
tar zxvf zlib-1.2.11.tar.gz |
...
cd $nalu_build_dir/packages/zlib-1.2.11 |
...
## CC=gcc CXX=g++ CFLAGS=-O3 CXXFLAGS=-O3 ./configure --prefix=$nalu_build_dir/install/ |
...
CC=icc CXX=icpc CFLAGS=-O3 CXXFLAGS=-O3 ./configure --prefix=$nalu_build_dir/install/ |
...
make |
...
make install |
8. Install
...
HDFS5
Note that the URL provided in the base document above, http://www.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8.12/src/hdf5-1.8.12.tar.gz changed.
| Code Block |
|---|
cd $nalu_build_dir/packages/ wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8/hdf5-1.8.12/src/hdf5-1.8.12.tar.gz |
...
tar zxvf hdf5-1.8.12.tar.gz |
...
cd $nalu_build_dir/packages/hdf5-1.8.12 |
...
./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" --enable-parallel --with-zlib=$nalu_build_dir/install --prefix=$nalu_build_dir/install |
...
make |
...
make install |
...
make check |
9. Install Parallel
...
NetCDF
| Code Block |
|---|
cd $nalu_build_dir/packages/ |
...
wget http://cucis.ece.northwestern.edu/projects/PnetCDF/Release/parallel-netcdf-1.6.1.tar.gz |
...
tar zxvf parallel-netcdf-1.6.1.tar.gz |
...
cd parallel-netcdf-1.6.1 |
...
./configure --prefix=$nalu_install_dir CC=mpicc FC=mpif90 CXX=mpicxx CFLAGS="-I$nalu_install_dir/include -O3" LDFLAGS=-L$nalu_install_dir/lib --disable-fortran |
...
make |
...
make install |
10. Install NetCDF
...
| Code Block |
|---|
cd $nalu_build_dir/packages/ |
...
curl -o netcdf-c-4.3.3.1.tar.gz https://codeload.github.com/Unidata/netcdf-c/tar.gz/v4.3.3.1 |
...
tar zxvf netcdf-c-4.3.3.1.tar.gz |
...
cd netcdf-c-4.3.3.1 |
...
./configure --prefix=$nalu_install_dir CC=mpicc FC=mpif90 CXX=mpicxx CFLAGS="-I$nalu_install_dir/include -O3" LDFLAGS=-L$nalu_install_dir/lib --enable-pnetcdf --enable-parallel-tests --enable-netcdf-4 --disable-shared --disable-fsync --disable-cdmremote --disable-dap --disable-doxygen --disable-v2 |
...
make -j 4 |
...
make install |
...
make check |
...
Trilinos (this is the most time-consuming step)
...
11. Install Trilinos
Note that this step takes a lot of time.
| Code Block |
|---|
cd $nalu_build_dir/packages/ |
...
git clone https://github.com/trilinos/Trilinos.git |
...
cd Trilinos/ |
...
Use the "Head Code Base"
| Code Block |
|---|
mkdir build |
...
curl -o $nalu_build_dir/packages/Trilinos/build/do-configTrilinos_release https://raw.githubusercontent.com/NaluCFD/Nalu/master/build/do-configTrilinos_release |
...
cd build |
...
Edit do-
...
configTrilinos_release to provide the MPI base dir & the Nalu build dir; in this example:
| Code Block |
|---|
## mpi_base_dir=/opt/hpcx-1.9/ompi-v2.x.i2017 |
...
## nalu_build_dir=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build |
...
## In addition, F\fix -DBoost_INCLUDE_DIRS and -DBoost_LIBRARY_DIRS, i.e., add the following lines: |
...
## -DBoost_INCLUDE_DIRS=$boost_dir/include \ |
...
## -DBoost_LIBRARY_DIRS=$boost_dir/lib \ |
Change mode and install
| Code Block |
|---|
chmod +x do-configTrilinos_release |
...
./do-configTrilinos_release |
...
make |
...
Go do something else, the above 'make' takes several hours
| Code Block |
|---|
make install |
12. Install Nalu
...
| Code Block |
|---|
cd $nalu_build_dir/packages/ |
...
git clone https://github.com/NaluCFD/Nalu.git |
...
cd Nalu/build/ |
...
cp do-configNalu_release do-configNaluNonTracked |
...
Edit do-
...
configNaluNonTracked to adjust the nalu_build_dir path
| Code Block |
|---|
./do-configNaluNonTracked |
...
make |
...
This make takes another 70 minutes or so. When complete, you will have a NaluX executable under $nalu_build_dir/packages/Nalu/build
Usage
Nalu was supplied with two datasets: abl_3km_256 and abl_3km_512. Running Nalu on 768 cores (24 nodes having 32 cores each) takes about 1 hour for the small (abl_3km_256) dataset; 10 times as much for the large dataset. Here is a Slurm script used to test Nalu with the small dataset:
| Code Block |
|---|
#!/bin/bash #SBATCH -N 24 #SBATCH -o nalu_256-24n-%j.out #SBATCH --tasks-per-node=32 #SBATCH -J Nalu256 #SBATCH -p pthor #SBATCH -t 06:00:00 #SBATCH --exclusive #SBATCH -d singleton ## Load any modules required here module load intel/compiler/2017.4.196 hpcx-1.9/icc-2017 NODES=$SLURM_NNODES NPROC=$SLURM_NPROCS export OMP_NUM_THREADS=1 MPI=ompi LOG=${NPROC}-${MPI}-plain-mbn-$SLURM_JOB_ID MPIFLAGS="" MPIFLAGS+="--map-by node --rank-by core " MPIFLAGS+="-report-bindings --display-map " MPIFLAGS+="-mca btl_sm_use_knem 1 " ## ulimit -s 10485760 ulimit -s unlimited echo Running on host `hostname` echo Time is `date` echo Directory is `pwd` echo '----------------------------------------------------' echo ' NODES USED = '$SLURM_NODELIST echo ' SLURM_JOB_ID = '$SLURM_JOB_ID echo ' CORES = '$NPROC echo '----------------------------------------------------' EXE=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build/install/bin/naluX echo "ldd $EXE" ldd $EXE echo "Launch command: /usr/bin/time -p mpirun -np ${NPROC} $MPIFLAGS $EXE -i abl_3km_256.i -o abl_3km_256_${NPROC}.log" /usr/bin/time -p mpirun -np ${NPROC} $MPIFLAGS $EXE -i abl_3km_256.i -o abl_3km_256_${NPROC}.log mv abl_3km_256_${NPROC}.log abl_3km_256_${LOG} |