References

• https://wiki.fysik.dtu.dk/gpaw/

• https://wiki.fysik.dtu.dk/gpaw/install.html

GPAW Overview

Slides:

Open

Description

GPAW is a density-functional theory (DFT) code based on the projector-augmented wave (PAW) method and various basis sets. The wave functions can be described with:

• Uniform real-space grids

• Plane waves

• Localized atomic orbitals

GPAW is implemented in Python and C programming languages and it relies on high performance libraries for linear algebra operations, FFTs, etc.

Parallelization is primarily with MPI, complementary OpenMP parallelization can improve the performance in some cases.

Requirements

• Python 3.6 or later

• NumPy 1.9 or later (base N-dimensional array package)

• SciPy 0.14 or later (library for scientific computing)

• ASE 3.18.0 or later (atomic simulation environment)

• a C-compiler

• LibXC 3.x or 4.x

• BLAS library

• an MPI library (required for parallel calculations)

Optional, but highly recommended:

• FFTW (for increased performance)

• BLACS and ScaLAPACK

Downloading and installing requirements

The required Python packages (NumPy, SciPy, ASE) will be downloaded and installed (if not present in the system) automatically by the GPAW’s installation process, and do not normally need to be separately installed. Typically, LibXC is the only dependency that needs to be separately installed (for BLAS, MPI, etc. one should rely on high performance system libraries). Installing LibXC from source (also package management tools yum, apt etc. can be used when appropriate):

Downloading and installing GPAW

There are several ways to install GPAW, refer to https://wiki.fysik.dtu.dk/gpaw/install.html for all options. Here, we recommend installing GPAW from source, version 21.1.0 will be used for the competition.

By default GPAW looks for BLAS library libblas in the default locations, and if it is available installation can be done at simplest with:

For some testing one should install also the optional pytest package

By default, the same compiler and compiler options as when building the Python interpreter are used. In HPC systems it is, however, often recommended to customize the installation in order to use other libraries, compilers or compiler options. This can accomplished via siteconfig.py file, sample file is provided in siteconfig_example.py. As an example, in order to use icc compiler with extra optimization flags and MKL BLAS library one should add into siteconfig.py:

One should also set the GPAW_CONFIG corresponding to this siteconfig.py file:

and proceed then with pip3 install …. (Note: siteconfig.py can be also named differently when playing with different installation options, i.e. siteconfig_intel.py, siteconfig_gcc.py … )

More information about customizing the installation can be found in GPAW wiki

Once the installation is complete, the PATH variable needs to be set for GPAW bins:

Finally, one needs to install the PAW datasets

Testing the installation

Once installation is complete and PYTHONUSERBASE and PATH are set, one can check the installation information as:

Next, one should make short serial test calculation

followed by short parallel calculation

GPAW contains also an extensive test set which should be run when developing the code, more details in GPAW wiki

Running GPAW

Sample output

In short benchmarks we will be looking the time for SCF-cycle (neglecting the initialization of the calculation).

GPAW Example on Niagara cluster

Download and Install

There are several ways to install GPAW, refer to https://wiki.fysik.dtu.dk/gpaw/install.html for all options.
Here is an basic example with CentOS 7/8:

Note: In case you get the error during installation, export the below variables and rebuild gpaw.

Building GPAW with Intel compilers:

1. Create siteconfig.py with blow lines.

1. Install GPAW

How to use HPC-X module

1. Check the bin files:

Install dataset

Running GPAW

Sample output

Tasks and Submissions

Note: You will need to supply runs on both clusters Niagara and Aspire-1 (4 CPU nodes only).

Input Files

Can be download from this location: https://github.com/jussienko/gpaw-isc-2021

Building and running GPAW

Build the code both on NCSS and Niagara. Run the provided input copper.py on one, two, and four nodes using pure MPI parallelization. The text output is in the file output_M_xxx.txt (xxx = number of MPI tasks). Input file has check for the correctness, so there is an error message if the results are wrong. Runtime should be less than 45 minutes with a single node.
Discuss the differences in performance and scalability in the two clusters. No modifications to the input file and source code are allowed.

Submit the output files to your shared folder in OneDrive.

Visualization

Run the provided input nanoribbon.py in any of the clusters (it is recommended to use one node, running time should be max. few minutes). As a result you will obtain a file named elf_ribbon.cube which contains electron localization function of the simulated system. Make a visualization (picture or animation) of the electron localization function. See e.g.
https://wiki.fysik.dtu.dk/gpaw/tutorials/plotting/plot_wave_functions.html
for hints for visualizing .cube files.

Submit the figure or animation to your team's shared folder.

Profiling

Use IPM profiler to profile the application copper.py over 4 node run.

Submit the profiler output as text or PDF file.

Performance tuning

Try to optimize the performance of the copper.py input in one of the clusters. Use the result in task 1 as baseline. You can try different compilers, compiler options, and high performance libraries. You can try also hybrid OpenMP / MPI parallelization with different number of OpenMP threads. You are allowed to modify parallelization options in copper.py (see https://wiki.fysik.dtu.dk/gpaw/documentation/parallel_runs/parallel_runs.html#manual-parallelization-types),
and modify also the source code as long as the correctness check in the input passes. Any modifications to the source code have to be made available e.g. in github. Submit report about the steps you did for performance tuning and about the performance improvements you achieved.

Bonus task: bug fix

The scalapack parallelization in GPAW fails with certain input:
Using Scalapack does not work (#269) · Issues · gpaw / gpaw · GitLab
Try to fix the bug and if successful make a merge request to GPAW's master branch.

Note that you need to build GPAW with Scalapack support for this task.