...
This post describes the steps followed to build it on Thor, a Linux cluster with Intel Xeon (Broadwell) processor and Mellanox EDR InfiniBand interconnect.
...
This make takes another 70 minutes or so. When complete, you will have a NaluX executable under $nalu_build_dir/packages/Nalu/build
Usage
Nalu was supplied with two datasets: abl_3km_256 and abl_3km_512. Running Nalu on 768 cores (24 nodes having 32 cores each) takes about 1 hour for the small (abl_3km_256) dataset; 10 times as much for the large dataset. Here is a Slurm script used to test Nalu with the small dataset:
| Code Block |
|---|
#!/bin/bash
#SBATCH -N 24
#SBATCH -o nalu_256-24n-%j.out
#SBATCH --tasks-per-node=32
#SBATCH -J Nalu256
#SBATCH -p pthor
#SBATCH -t 06:00:00
#SBATCH --exclusive
#SBATCH -d singleton
## Load any modules required here
module load intel/compiler/2017.4.196 hpcx-1.9/icc-2017
NODES=$SLURM_NNODES
NPROC=$SLURM_NPROCS
export OMP_NUM_THREADS=1
MPI=ompi
LOG=${NPROC}-${MPI}-plain-mbn-$SLURM_JOB_ID
MPIFLAGS=""
MPIFLAGS+="--map-by node --rank-by core "
MPIFLAGS+="-report-bindings --display-map "
MPIFLAGS+="-mca btl_sm_use_knem 1 "
## ulimit -s 10485760
ulimit -s unlimited
echo Running on host `hostname`
echo Time is `date`
echo Directory is `pwd`
echo '----------------------------------------------------'
echo ' NODES USED = '$SLURM_NODELIST
echo ' SLURM_JOB_ID = '$SLURM_JOB_ID
echo ' CORES = '$NPROC
echo '----------------------------------------------------'
EXE=/mnt/beegfs3/gerardo/nrel_esif/Nalu/build/install/bin/naluX
echo "ldd $EXE"
ldd $EXE
echo "Launch command: /usr/bin/time -p mpirun -np ${NPROC} $MPIFLAGS $EXE -i abl_3km_256.i -o abl_3km_256_${NPROC}.log"
/usr/bin/time -p mpirun -np ${NPROC} $MPIFLAGS $EXE -i abl_3km_256.i -o abl_3km_256_${NPROC}.log
mv abl_3km_256_${NPROC}.log abl_3km_256_${LOG}
|