Building Nalu on Thor
Nalu is a Low-Mach Number turbulent flow simulator for energy applications.
This post describes the steps followed to build it on Thor, a Linux cluster with Intel Xeon (Broadwell) processor and Mellanox EDR InfiniBand interconnect.
References
Installation
First, consult the following document , specifically the Linux section.
Note that the two subsections with heading Nalu Releases aren't especially clear, but the build will work if one follows the directions for "Head Code Base". Many packages need to be built before one has a working Nalu executable.
1. Set up the build environment
a. Load the compiler and HPC-X modules you wish to use
module load intel/compiler/2017.4.196 hpcx-1.9/icc-2017 export CC=icc export CXX=icpc export FC=ifort export F90=ifort export F77=ifort export OMPI_CC=icc export OMPI_CXX=icpc export OMPI_FC=ifort export OMPI_F77=ifort
b. Replace the path that follows with whatever is appropriate for you
cd /mnt/beegfs3/gerardo/nrel_esif/Nalu/ mkdir build cd build export nalu_build_dir=$PWD mkdir $nalu_build_dir/packages mkdir $nalu_build_dir/install mkdir $nalu_build_dir/install/lib cd $nalu_build_dir/packages export nalu_install_dir=$nalu_build_dir/install
2. Install CMake
wget http://www.cmake.org/files/v3.1/cmake-3.1.0-rc2.tar.gz tar zxvf cmake-3.1.0-rc2.tar.gz cd cmake-3.1.0-rc2/ ./configure --prefix=$nalu_build_dir/install make make install
Make sure your newly-built CMake can be found
export PATH=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build/install/bin:$PATH
3. Install SuperLU
cd $nalu_build_dir/packages wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz tar zxvf superlu_4.3.tar.gz cd $nalu_build_dir/packages/SuperLU_4.3 cp MAKE_INC/make.linux make.inc
Edit make.inc so that the following macros are defined as indicated:
# PLAT = _x86_64 # SuperLUroot = /mnt/beegfs3/gerardo/nrel_esif/Nalu/build/packages/SuperLU_4.3 # BLASLIB = -L/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 -lmkl_blas95_lp64 -lmkl_core -lmkl_rt -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_avx2 -liomp5 # CC = mpicc # FORTRAN = mpif77
Create directory and make
mkdir $nalu_build_dir/install/SuperLU_4.3 mkdir $nalu_build_dir/install/SuperLU_4.3/lib mkdir $nalu_build_dir/install/SuperLU_4.3/include make cp SRC/*.h $nalu_build_dir/install/SuperLU_4.3/include
Note: The following copy step is missing from the directions in the base document, it will trip you up if not done.
cp -p lib/libsuperlu_4.3.a $nalu_build_dir/install/SuperLU_4.3/lib
4. Install libml2
cd $nalu_build_dir/packages wget http://www.xmlsoft.org/sources/libxml2-2.9.2.tar.gz tar zxvf libxml2-2.9.2.tar.gz cd $nalu_build_dir/packages/libxml2-2.9.2 CC=mpicc CXX=mpicxx ./configure -without-python --prefix=$nalu_build_dir/install make -k install
5. Install Boost
cd $nalu_build_dir/packages wget http://iweb.dl.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.gz tar zxvf boost_1_55_0.tar.gz cd $nalu_build_dir/packages/boost_1_55_0 echo "using mpi : `which mpicxx` ;" >> ./tools/build/v2/user-config.jam ./bootstrap.sh --prefix=$nalu_build_dir/install --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception ./b2 -j 4 2>&1 | tee boost_build_one ./b2 -j 4 install 2>&1 | tee boost_build_instal
6. Install Yaml
cd $nalu_build_dir/packages git clone https://github.com/jbeder/yaml-cpp cd $nalu_build_dir/packages/yaml-cpp mkdir build cd build cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_CXX_FLAGS=-std=c++11 -DCMAKE_CC_COMPILER=mpicc -DCMAKE_INSTALL_PREFIX=$nalu_build_dir/install .. make make install
7. Install Zlib
Note that the base document above calls for downloading http://zlib.net/zlib-1.2.8.tar.gz, which is no longer available
cd $nalu_build_dir/packages wget http://zlib.net/zlib-1.2.11.tar.gz tar zxvf zlib-1.2.11.tar.gz cd $nalu_build_dir/packages/zlib-1.2.11 ## CC=gcc CXX=g++ CFLAGS=-O3 CXXFLAGS=-O3 ./configure --prefix=$nalu_build_dir/install/ CC=icc CXX=icpc CFLAGS=-O3 CXXFLAGS=-O3 ./configure --prefix=$nalu_build_dir/install/ make make install
8. Install HDFS5
Note that the URL provided in the base document above, http://www.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8.12/src/hdf5-1.8.12.tar.gz changed.
cd $nalu_build_dir/packages/ wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8/hdf5-1.8.12/src/hdf5-1.8.12.tar.gz tar zxvf hdf5-1.8.12.tar.gz cd $nalu_build_dir/packages/hdf5-1.8.12 ./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" --enable-parallel --with-zlib=$nalu_build_dir/install --prefix=$nalu_build_dir/install make make install make check
9. Install Parallel NetCDF
cd $nalu_build_dir/packages/ wget http://cucis.ece.northwestern.edu/projects/PnetCDF/Release/parallel-netcdf-1.6.1.tar.gz tar zxvf parallel-netcdf-1.6.1.tar.gz cd parallel-netcdf-1.6.1 ./configure --prefix=$nalu_install_dir CC=mpicc FC=mpif90 CXX=mpicxx CFLAGS="-I$nalu_install_dir/include -O3" LDFLAGS=-L$nalu_install_dir/lib --disable-fortran make make install
10. Install NetCDF
cd $nalu_build_dir/packages/ curl -o netcdf-c-4.3.3.1.tar.gz https://codeload.github.com/Unidata/netcdf-c/tar.gz/v4.3.3.1 tar zxvf netcdf-c-4.3.3.1.tar.gz cd netcdf-c-4.3.3.1 ./configure --prefix=$nalu_install_dir CC=mpicc FC=mpif90 CXX=mpicxx CFLAGS="-I$nalu_install_dir/include -O3" LDFLAGS=-L$nalu_install_dir/lib --enable-pnetcdf --enable-parallel-tests --enable-netcdf-4 --disable-shared --disable-fsync --disable-cdmremote --disable-dap --disable-doxygen --disable-v2 make -j 4 make install make check
11. Install Trilinos
Note that this step takes a lot of time.
cd $nalu_build_dir/packages/ git clone https://github.com/trilinos/Trilinos.git cd Trilinos/
Use the "Head Code Base"
mkdir build curl -o $nalu_build_dir/packages/Trilinos/build/do-configTrilinos_release https://raw.githubusercontent.com/NaluCFD/Nalu/master/build/do-configTrilinos_release cd build
Edit do-configTrilinos_release to provide the MPI base dir & the Nalu build dir; in this example:
## mpi_base_dir=/opt/hpcx-1.9/ompi-v2.x.i2017 ## nalu_build_dir=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build ## In addition, F\fix -DBoost_INCLUDE_DIRS and -DBoost_LIBRARY_DIRS, i.e., add the following lines: ## -DBoost_INCLUDE_DIRS=$boost_dir/include \ ## -DBoost_LIBRARY_DIRS=$boost_dir/lib \
Change mode and install
chmod +x do-configTrilinos_release ./do-configTrilinos_release make
Go do something else, the above 'make' takes several hours
make install
12. Install Nalu
cd $nalu_build_dir/packages/ git clone https://github.com/NaluCFD/Nalu.git cd Nalu/build/ cp do-configNalu_release do-configNaluNonTracked
Edit do-configNaluNonTracked to adjust the nalu_build_dir path
./do-configNaluNonTracked make
This make takes another 70 minutes or so. When complete, you will have a NaluX executable under $nalu_build_dir/packages/Nalu/build
Usage
Nalu was supplied with two datasets: abl_3km_256 and abl_3km_512. Running Nalu on 768 cores (24 nodes having 32 cores each) takes about 1 hour for the small (abl_3km_256) dataset; 10 times as much for the large dataset. Here is a Slurm script used to test Nalu with the small dataset:
#!/bin/bash
#SBATCH -N 24
#SBATCH -o nalu_256-24n-%j.out
#SBATCH --tasks-per-node=32
#SBATCH -J Nalu256
#SBATCH -p pthor
#SBATCH -t 06:00:00
#SBATCH --exclusive
#SBATCH -d singleton
## Load any modules required here
module load intel/compiler/2017.4.196 hpcx-1.9/icc-2017
NODES=$SLURM_NNODES
NPROC=$SLURM_NPROCS
export OMP_NUM_THREADS=1
MPI=ompi
LOG=${NPROC}-${MPI}-plain-mbn-$SLURM_JOB_ID
MPIFLAGS=""
MPIFLAGS+="--map-by node --rank-by core "
MPIFLAGS+="-report-bindings --display-map "
MPIFLAGS+="-mca btl_sm_use_knem 1 "
## ulimit -s 10485760
ulimit -s unlimited
echo Running on host `hostname`
echo Time is `date`
echo Directory is `pwd`
echo '----------------------------------------------------'
echo ' NODES USED = '$SLURM_NODELIST
echo ' SLURM_JOB_ID = '$SLURM_JOB_ID
echo ' CORES = '$NPROC
echo '----------------------------------------------------'
EXE=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build/install/bin/naluX
echo "ldd $EXE"
ldd $EXE
echo "Launch command: /usr/bin/time -p mpirun -np ${NPROC} $MPIFLAGS $EXE -i abl_3km_256.i -o abl_3km_256_${NPROC}.log"
/usr/bin/time -p mpirun -np ${NPROC} $MPIFLAGS $EXE -i abl_3km_256.i -o abl_3km_256_${NPROC}.log
mv abl_3km_256_${NPROC}.log abl_3km_256_${LOG}