Getting Started with POT3D for ISC23 SCC

Overview

POT3D is a Fortran code that computes potential field solutions to approximate the solar coronal magnetic field using observed photospheric magnetic fields as a boundary condition. It can be used to generate potential field source surface (PFSS), potential field current sheet (PFCS), and open field (OF) models. It has been (and continues to be) used for numerous studies of coronal structure and dynamics. The code is highly parallelized using MPI and is GPU-accelerated using Fortran standard parallelism (do concurrent) and OpenACC, along with an option to use the NVIDIA cuSparse library. The HDF5 file format is used for input/output.

Downloading and compiling POT3D

git clone https://github.com/predsci/POT3D

There are sample scripts under build_examples directory.

We chose build_examples/build_cpu_mpi-only_intel_ubuntu20.04.sh.

1. Copy build_examples/build_cpu_mpi-only_intel_ubuntu20.04.sh to root directory.

   • cp build_examples/build_cpu_mpi-only_intel_ubuntu20.04.sh rebuild.sh

2. Build HDF5 and edit rebuild.sh with HDF5 location.

3. FC = mpif90” FC = $(MPIFC)

4. Execute the build script.

Sample build script for Fritz:

We used modules for MPI and HDF5 on Firtz.

# MPI=openmpi-4.1.2-intel2021.4.0
MPI=intelmpi-2021.7.0
if [[ "$MPI" =~ ^intel ]]; then
        module load hdf5/1.10.7-impi-intel
        export I_MPI_F90=ifort
else
        module load hdf5/1.10.7-ompi-intel
        export OMPI_MPIF90=ifort
fi
#################################################################

# Location of local hdf5 installed with same compiler being used for POT3D:
HDF5_INCLUDE_DIR="$HDF5_ROOT/include"
HDF5_LIB_DIR="$HDF5_ROOT/lib"
# Fortran HDF5 library flags (these can be version dependent):
HDF5_LIB_FLAGS="-lhdf5_fortran -lhdf5_hl_fortran -lhdf5 -lhdf5_hl""
...

Running Example

cd testsuite
TEST=small
POT3D_HOME=$PWD/..

cp ${POT3D_HOME}/testsuite/${TEST}/input/* ${POT3D_HOME}/testsuite/${TEST}/run/
cd ${POT3D_HOME}/testsuite/${TEST}/run

echo "Running POT3D with $NP MPI rank..."
CMD="mpirun -np $NP $MPIFLAGS $POT3D_HOME/bin/pot3d"

/usr/bin/time $CMD > pot3d.out
echo "Done!"

runtime=($(tail -n 5 timing.out | head -n 1))
echo "Wall clock time: ${runtime[6]} seconds"
echo " "

#Validate run:
${POT3D_HOME}/scripts/pot3d_validation.sh pot3d.out ${POT3D_HOME}/testsuite/${TEST}/validation/pot3d.out

Task and submission

1. Use the input under testsuite/isc2023 folder.

• https://github.com/predsci/POT3D/tree/main/testsuite/isc2023

1. Profile the input
   Use any of the remote clusters to run an MPI profile (such as IPM profile or any other profiler) over 4 nodes, full PPN to profile the given input.

   • Submit the profile as PDF to the team's folder.

   • Add to your presentation the 3 main MPI calls that are being used plus the MPI time being used.

2. Run POT3D with isc2023 input on both PSC bridges-2 and FAU Fritz CPU clusters.

   Your job should converge and print outputs like below.
    ### The CG solver has converged.
    Iteration:    25112   Residual:   9.972489313728662E-13

3. Run POT3D on the FAU Alex cluster using A100 GPU partition. Use only 4 GPUs for the run. Submit the results to the team's folder.

4. Submission and Presentation:
   - Submit all the build scripts, run scripts and input & outputs (pot3d.dat, pot3d.out, timing.out)
   - Do not submit the output data or the source code.
   - Prepare slides for the team’s interview based on your work for this application.