DFTB+ is a simulation code for approximately solving the Schrödinger equation by using methods derived from Density Functional Theory. Tasks for this challenge will find the time-independent quantum mechanical ground state of models of crystals while making efficient use of limited computing resources.

Presentation:

Build and Run

Manual compilation

Prerequisites: In order to build DFTB+, you will need
- A recent working Fortran 2008 compiler (e.g. GFortran >= 13, Intel oneAPI >= 2024, NAG >= 7.2)
- An MPI framework (supporting MPI3) working with your compiler
- CMake (>= 3.16)
- BLAS and LAPACK library implementations (such as OpenBLAS or the Intel oneMKL)
- A ScaLAPACK implementation (such as netlib ScaLAPACK or Intel oneMKL)
- The ELSI library (>= 2.9) which provides additional optimized matrix solvers
Download DFTB+ release 25.1.
Follow the DFTB+ build instructions. Make sure that the directories of the external library dependencies are included in the CMAKE_PREFIX_PATH environment variable. This allows CMake to find them. Also make sure that the FC environment variable contains your (MPI aware) Fortran compiler command (mpifort is a common alias). You should specify the following CMake build options:
- -DWITH_MPI=1 to enable MPI
- -DWITH_ELSI=1 to build with ELSI integration enabled
- -DSCALAPACK_LIBRARY=scalapack (in case CMake does not find your ScaLAPACK library. If your library is not called libscalapack.a or libscalapack.so adjust the value accordingly)

Sample build script on Iris

module load intel/2023.2 compiler mkl
module load cmake
module load hpcx/2.25.1

export OMPI_CC=icc
export OMPI_CXX=icpc
export OMPI_FC=ifort

ELSI_DIR=<path>/elsi-2.12.0
INSDIR=<path to install>
mkdir _build
# To use correct Blacs library for hpcx/openmpi.
sed -e "s/mkl_blacs_intelmpi_lp64/mkl_blacs_openmpi_lp64/g" -i sys/intel.cmake
CC=mpicc FC=mpif90 cmake -DCMAKE_INSTALL_PREFIX=$INSDIR \
        -DWITH_MPI=Y -DWITH_ELSI=Y \
        -DCMAKE_PREFIX_PATH=$ELSI_DIR \
        -B _build .
cd _build
make install

Building via Spack

Spack has a recipe for DFTB+, you should be able to build it with all this dependencies via this recipe. Specifying a variant that includes ELSI is also possible. Note: the Spack recipe is neither maintained nor tested by the DFTB+ developers, so any questions regarding it should go to the corresponding Spack forums.

Testing DFTB+

Once you have built DFTB+ from source, you can verify the build using CTest. Running ctest inside the build directory will execute a set of small validation tests. Before doing so, ensure that you have obtained the required parameter files by running the helper script ./utils/get_opt_externals slakos, as described in the README and the build instructions.
To test DFTB+ on a system comparable to (but much smaller than) the one intended for the competition, you can use the 64-atom SiC supercell. This system is small enough to complete the calculation within a minute, even on a single core. Use the following input: . You will also need the corresponding parameters . Extract the parameter archive into any directory (for example into ~/opt/slakos). Then run DFTB+ from within the SiC.0064 directory while specifying the parent directory of the parameter set via the DFTBPLUS_PARAM_DIR environment variable (for example DFTBPLUS_PARAM_DIR=~/opt/slakos dftb+). You should obtain a Total Energy of -2656.9258 eV in the output.

Sample run script on Iris

#!/bin/bash -l
#SBATCH -p iris
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=56
#SBATCH --threads-per-core=1
#SBATCH -J df
#SBATCH --time=1:00:00
#SBATCH --exclusive
#SBATCH -d singleton

module load intel/2023.2
module load compiler mkl
module load hpcx/2.25.1

MPIFLAGS+="-x UCX_NET_DEVICES=mlx5_0:1 "

ELSI=1
if [ $ELSI -eq 1 ]; then
        cp dftb_in.hsd.elsi dftb_in.hsd
        INSDIR=install-elsi
else
        cp dftb_in.hsd.no_elsi dftb_in.hsd
        INSDIR=install
fi
mpirun -np $SLURM_NPROCS-x UCX_NET_DEVICES=mlx5_0:1 <path>/$INSDIR/bin/dftb+

Tasks and Submissions

Run the application on 4 CPU nodes only on Iris with the given input and submit the results (standard output) in the team’s folder. DFTB+ supports hybrid (process based MPI and thread based OpenMP) parallelization. Submit your best performance, build script and run script. Do not submit multiple results nor binary files.
Use an MPI profiler to examine the application with the given input on 4 CPU nodes. What are the 3 main MPI calls in use during these runs (with the most calls to the function)? Show your work in the teams presentation.
Check the execution time of the application using different numbers of CPUs / nodes. Test scaling on one socket, one node, two nodes and 4 nodes, and analyze the strong scaling of the input. Also try various different matrix diagonalizers (Scalapack QR vs. ELPA). Present a graph in the team’s presentation. Discuss the parallel efficiency of the scaling (using the one socket case as the reference).
Use any visualization tool to display the results, create a short video or set of images and show your work in the teams presentation.