Building Nalu on Thor

Owned by Gerardo Cisneros-Stoianowski (Unlicensed)
Last updated: Oct 30, 2017 by Ophir Maor

Nalu is a Low-Mach Number turbulent flow simulator for energy applications.  

This post describes the steps followed to build it on Thor, a Linux cluster with Intel Xeon (Broadwell) processor and Mellanox EDR InfiniBand interconnect.


References 


Installation

First, consult the following document , specifically the Linux section. 

Note that the two subsections with heading Nalu Releases aren't especially clear, but the build will work if one follows the directions for "Head Code Base".  Many packages need to be built before one has a working Nalu executable.


1. Set up the build environment


a. Load the compiler and HPC-X modules you wish to use

module load intel/compiler/2017.4.196 hpcx-1.9/icc-2017
export CC=icc
export CXX=icpc
export FC=ifort
export F90=ifort
export F77=ifort
export OMPI_CC=icc
export OMPI_CXX=icpc
export OMPI_FC=ifort
export OMPI_F77=ifort


b.  Replace the path that follows with whatever is appropriate for you

cd /mnt/beegfs3/gerardo/nrel_esif/Nalu/
mkdir build
cd build
export nalu_build_dir=$PWD
mkdir $nalu_build_dir/packages
mkdir $nalu_build_dir/install
mkdir $nalu_build_dir/install/lib
cd $nalu_build_dir/packages
export nalu_install_dir=$nalu_build_dir/install


2. Install CMake 


wget http://www.cmake.org/files/v3.1/cmake-3.1.0-rc2.tar.gz
tar zxvf cmake-3.1.0-rc2.tar.gz
cd cmake-3.1.0-rc2/
./configure --prefix=$nalu_build_dir/install
make
make install


Make sure your newly-built CMake can be found


export PATH=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build/install/bin:$PATH


3. Install SuperLU

cd $nalu_build_dir/packages
wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
tar zxvf superlu_4.3.tar.gz
cd $nalu_build_dir/packages/SuperLU_4.3
cp MAKE_INC/make.linux make.inc


Edit make.inc so that the following macros are defined as indicated:

# PLAT = _x86_64
# SuperLUroot = /mnt/beegfs3/gerardo/nrel_esif/Nalu/build/packages/SuperLU_4.3
# BLASLIB = -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_blas95_lp64 -lmkl_core -lmkl_rt -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_avx2 -liomp5
# CC = mpicc
# FORTRAN = mpif77


Create directory and make 

mkdir $nalu_build_dir/install/SuperLU_4.3
mkdir $nalu_build_dir/install/SuperLU_4.3/lib
mkdir $nalu_build_dir/install/SuperLU_4.3/include
make
cp SRC/*.h $nalu_build_dir/install/SuperLU_4.3/include


Note: The following copy step is missing from the directions in the base document, it will trip you up if not done.

cp -p lib/libsuperlu_4.3.a $nalu_build_dir/install/SuperLU_4.3/lib


4. Install libml2

cd $nalu_build_dir/packages
wget http://www.xmlsoft.org/sources/libxml2-2.9.2.tar.gz
tar zxvf libxml2-2.9.2.tar.gz
cd $nalu_build_dir/packages/libxml2-2.9.2
CC=mpicc CXX=mpicxx ./configure -without-python --prefix=$nalu_build_dir/install
make -k install



5. Install Boost

cd $nalu_build_dir/packages
wget http://iweb.dl.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.gz
tar zxvf boost_1_55_0.tar.gz
cd $nalu_build_dir/packages/boost_1_55_0
echo "using mpi : `which mpicxx` ;" >> ./tools/build/v2/user-config.jam
./bootstrap.sh --prefix=$nalu_build_dir/install --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception
./b2 -j 4 2>&1 | tee boost_build_one
./b2 -j 4 install 2>&1 | tee boost_build_instal


6. Install Yaml 

cd $nalu_build_dir/packages
git clone https://github.com/jbeder/yaml-cpp
cd $nalu_build_dir/packages/yaml-cpp
mkdir build
cd build
cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_CXX_FLAGS=-std=c++11 -DCMAKE_CC_COMPILER=mpicc -DCMAKE_INSTALL_PREFIX=$nalu_build_dir/install ..
make
make install


7. Install Zlib 

Note that the base document above calls for downloading http://zlib.net/zlib-1.2.8.tar.gz, which is no longer available

cd $nalu_build_dir/packages
wget http://zlib.net/zlib-1.2.11.tar.gz
tar zxvf zlib-1.2.11.tar.gz
cd $nalu_build_dir/packages/zlib-1.2.11
## CC=gcc CXX=g++ CFLAGS=-O3 CXXFLAGS=-O3 ./configure --prefix=$nalu_build_dir/install/
CC=icc CXX=icpc CFLAGS=-O3 CXXFLAGS=-O3 ./configure --prefix=$nalu_build_dir/install/
make
make install


8. Install HDFS5


Note that the URL provided in the base document above, http://www.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8.12/src/hdf5-1.8.12.tar.gz changed.

cd $nalu_build_dir/packages/
wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8/hdf5-1.8.12/src/hdf5-1.8.12.tar.gz
tar zxvf hdf5-1.8.12.tar.gz
cd $nalu_build_dir/packages/hdf5-1.8.12
./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" --enable-parallel --with-zlib=$nalu_build_dir/install --prefix=$nalu_build_dir/install
make
make install
make check


9. Install Parallel NetCDF 

cd $nalu_build_dir/packages/
wget http://cucis.ece.northwestern.edu/projects/PnetCDF/Release/parallel-netcdf-1.6.1.tar.gz
tar zxvf parallel-netcdf-1.6.1.tar.gz
cd parallel-netcdf-1.6.1
./configure --prefix=$nalu_install_dir CC=mpicc FC=mpif90 CXX=mpicxx CFLAGS="-I$nalu_install_dir/include -O3" LDFLAGS=-L$nalu_install_dir/lib --disable-fortran
make
make install


10. Install NetCDF

cd $nalu_build_dir/packages/
curl -o netcdf-c-4.3.3.1.tar.gz https://codeload.github.com/Unidata/netcdf-c/tar.gz/v4.3.3.1
tar zxvf netcdf-c-4.3.3.1.tar.gz
cd netcdf-c-4.3.3.1
./configure --prefix=$nalu_install_dir CC=mpicc FC=mpif90 CXX=mpicxx CFLAGS="-I$nalu_install_dir/include -O3" LDFLAGS=-L$nalu_install_dir/lib --enable-pnetcdf --enable-parallel-tests --enable-netcdf-4 --disable-shared --disable-fsync --disable-cdmremote --disable-dap --disable-doxygen --disable-v2
make -j 4
make install
make check


11. Install Trilinos 

Note that this step takes a lot of time.

cd $nalu_build_dir/packages/
git clone https://github.com/trilinos/Trilinos.git
cd Trilinos/


Use the "Head Code Base"

mkdir build
curl -o $nalu_build_dir/packages/Trilinos/build/do-configTrilinos_release https://raw.githubusercontent.com/NaluCFD/Nalu/master/build/do-configTrilinos_release
cd build


Edit do-configTrilinos_release to provide the MPI base dir & the Nalu build dir; in this example:

## mpi_base_dir=/opt/hpcx-1.9/ompi-v2.x.i2017
## nalu_build_dir=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build
## In addition, F\fix -DBoost_INCLUDE_DIRS and -DBoost_LIBRARY_DIRS, i.e., add the following lines:
## -DBoost_INCLUDE_DIRS=$boost_dir/include \
## -DBoost_LIBRARY_DIRS=$boost_dir/lib \


Change mode and install 

chmod +x do-configTrilinos_release
./do-configTrilinos_release
make


Go do something else, the above 'make' takes several hours

make install


12. Install Nalu

cd $nalu_build_dir/packages/
git clone https://github.com/NaluCFD/Nalu.git
cd Nalu/build/
cp do-configNalu_release do-configNaluNonTracked


Edit do-configNaluNonTracked to adjust the nalu_build_dir path

./do-configNaluNonTracked
make


This make takes another 70 minutes or so.  When complete, you will have a NaluX executable under $nalu_build_dir/packages/Nalu/build


Usage

Nalu was supplied with two datasets: abl_3km_256 and abl_3km_512.  Running Nalu on 768 cores (24 nodes having 32 cores each) takes about 1 hour for the small (abl_3km_256) dataset; 10 times as much for the large dataset.  Here is a Slurm script used to test Nalu with the small dataset:

#!/bin/bash
#SBATCH -N 24
#SBATCH -o  nalu_256-24n-%j.out
#SBATCH --tasks-per-node=32
#SBATCH -J Nalu256
#SBATCH -p pthor
#SBATCH -t 06:00:00
#SBATCH --exclusive
#SBATCH -d singleton

## Load any modules required here
module load intel/compiler/2017.4.196 hpcx-1.9/icc-2017

NODES=$SLURM_NNODES
NPROC=$SLURM_NPROCS
export OMP_NUM_THREADS=1

MPI=ompi

LOG=${NPROC}-${MPI}-plain-mbn-$SLURM_JOB_ID

MPIFLAGS=""
MPIFLAGS+="--map-by node --rank-by core "
MPIFLAGS+="-report-bindings --display-map "
MPIFLAGS+="-mca btl_sm_use_knem 1 "

## ulimit -s 10485760
ulimit -s unlimited

echo Running on host `hostname`
echo Time is `date`
echo Directory is `pwd`

echo '----------------------------------------------------'
echo ' NODES USED = '$SLURM_NODELIST
echo ' SLURM_JOB_ID = '$SLURM_JOB_ID
echo ' CORES = '$NPROC
echo '----------------------------------------------------'

EXE=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build/install/bin/naluX
echo "ldd $EXE"
ldd $EXE

echo "Launch command: /usr/bin/time -p mpirun -np ${NPROC} $MPIFLAGS $EXE -i abl_3km_256.i -o abl_3km_256_${NPROC}.log"
/usr/bin/time -p mpirun -np ${NPROC} $MPIFLAGS $EXE -i abl_3km_256.i -o abl_3km_256_${NPROC}.log
mv abl_3km_256_${NPROC}.log abl_3km_256_${LOG}