Versions Compared

Old Version 3

changes.mady.by.user Former user

Saved on

compared with

New Version Current

changes.mady.by.user Ophir Maor

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This post describes the steps followed to build it on Thor, a Linux cluster with Intel Xeon (Broadwell) processor and Mellanox EDR InfiniBand interconnect.


Table of Contents

References 

...

1. Set up the build environment


a. Load the compiler and HPC-X modules you wish to use

Code Block
module load intel/compiler/2017.4.196 hpcx-1.9/icc-2017
export CC=icc
export CXX=icpc
export FC=ifort
export F90=ifort
export F77=ifort
export OMPI_CC=icc
export OMPI_CXX=icpc
export OMPI_FC=ifort
export OMPI_F77=ifort


b.  Replace the path that follows with whatever is appropriate for you

Code Block
cd /mnt/beegfs3/gerardo/nrel_esif/Nalu/
mkdir build
cd build
export nalu_build_dir=$PWD
mkdir $nalu_build_dir/packages
mkdir $nalu_build_dir/install
mkdir $nalu_build_dir/install/lib
cd $nalu_build_dir/packages
export nalu_install_dir=$nalu_build_dir/install

...

This make takes another 70 minutes or so.  When complete, you will have a NaluX executable under $nalu_build_dir/packages/Nalu/build


Usage

Nalu was supplied with two datasets: abl_3km_256 and abl_3km_512.  Running Nalu on 768 cores (24 nodes having 32 cores each) takes about 1 hour for the small (abl_3km_256) dataset; 10 times as much for the large dataset.  Here is a Slurm script used to test Nalu with the small dataset:

Code Block
#!/bin/bash
#SBATCH -N 24
#SBATCH -o  nalu_256-24n-%j.out
#SBATCH --tasks-per-node=32
#SBATCH -J Nalu256
#SBATCH -p pthor
#SBATCH -t 06:00:00
#SBATCH --exclusive
#SBATCH -d singleton

## Load any modules required here
module load intel/compiler/2017.4.196 hpcx-1.9/icc-2017

NODES=$SLURM_NNODES
NPROC=$SLURM_NPROCS
export OMP_NUM_THREADS=1

MPI=ompi

LOG=${NPROC}-${MPI}-plain-mbn-$SLURM_JOB_ID

MPIFLAGS=""
MPIFLAGS+="--map-by node --rank-by core "
MPIFLAGS+="-report-bindings --display-map "
MPIFLAGS+="-mca btl_sm_use_knem 1 "

## ulimit -s 10485760
ulimit -s unlimited

echo Running on host `hostname`
echo Time is `date`
echo Directory is `pwd`

echo '----------------------------------------------------'
echo ' NODES USED = '$SLURM_NODELIST
echo ' SLURM_JOB_ID = '$SLURM_JOB_ID
echo ' CORES = '$NPROC
echo '----------------------------------------------------'

EXE=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build/install/bin/naluX
echo "ldd $EXE"
ldd $EXE

echo "Launch command: /usr/bin/time -p mpirun -np ${NPROC} $MPIFLAGS $EXE -i abl_3km_256.i -o abl_3km_256_${NPROC}.log"
/usr/bin/time -p mpirun -np ${NPROC} $MPIFLAGS $EXE -i abl_3km_256.i -o abl_3km_256_${NPROC}.log
mv abl_3km_256_${NPROC}.log abl_3km_256_${LOG}