# Introduction The **ICON modelling framework** is a joint project between the [German Weather Service](https://www.dwd.de/EN/Home/home_node.html) and the [Max Planck Institute for Meteorology](https://www.mpimet.mpg.de/en/home/) for developing a unified next-generation global numerical weather prediction and climate modelling system. The ICON model has been introduced into DWD's operational forecast system in January 2015. This document provides technical details on how to build the model, as well as the description of basic steps required to run it. More information about ICON is available in the [corresponding section](https://code.mpimet.mpg.de/projects/iconpublic/wiki/Documentation) of the [project's public web page](https://code.mpimet.mpg.de/projects/iconpublic). # Table of contents 1. [Quick start](#quick-start) 2. [Configuration](#configuration) - [ICON dependencies](#icon-dependencies) - [Bundled libraries](#bundled-libraries) - [Compilers and tools](#compilers-and-tools) - [Compiler flags](#compiler-flags) - [Dynamic libraries](#dynamic-libraries) - [Configuration and building environments](#configuration-and-building-environments) - [Configuration wrappers](#configuration-wrappers) - [Out-of-source configuration (building)](#out-of-source-configuration-building) 3. [Building](#building) - [Source file collection](#source-file-collection) - [Preprocessing](#preprocessing) - [Source dependency tracking](#source-dependency-tracking) - [Code consistency checks](#code-consistency-checks) - [Compilation cascade prevention](#compilation-cascade-prevention) - [Building of the bundled libraries](#building-of-the-bundled-libraries) - [Source provenance collection](#source-provenance-collection) 4. [Running](#running) - [Out-of-build runs](#out-of-build-runs) 5. [FAQ](#faq) # Quick start The process of building ICON consists of two parts: *configuring* the options and compiler flags, and *building* the source code with those options and flags. The configuration step is normally done by running the [./configure](./configure) script with command-line arguments, which, among other things, tell the script where to locate libraries and tools required for building. The list of arguments enabling a successful configuration might be quite long and difficult to compose, therefore, instead of running the generic [./configure](./configure) script directly, users are recommended to execute a corresponding platform- or machine-specific *configuration wrapper* that sets the required compiler and linker flags as well as the recommended set of configure options. The wrappers can be found in the respective subdirectories of the [./config](./config) directory. For example, if you need to build ICON on [Mistral@DKRZ](https://www.dkrz.de/up/systems/mistral) with OpenMP feature enabled using Intel compiler, you can run: ```console $ cd /path/to/icon $ ./config/dkrz/mistral.intel --enable-openmp ``` Alternatively, you can create a directory and perform an *out-of-source* build: ```console $ mkdir build && cd build $ /path/to/icon/config/dkrz/mistral.intel --enable-openmp ``` This way, you can build ICON in several different configurations, i.e. with different compilers and features, using the same copy of the source code. > **_NOTE:_** If you want to build and run ICON on your personal computer/laptop, consider using [generic configure wrappers](./config/generic). > **_NOTE:_** If there is no configure wrapper script for your platform or machine, refer to section [Configuration](#configuration) for information on how to work with the [./configure](./configure) script directly. The building step is done by running `make` command with an optional argument specifying the number of jobs to run simultaneously. For example, ```console $ make -j8 ``` The result of the building — the executable file of ICON — is saved to the `./bin` subdirectory of the build directory. > **_NOTE:_** For more information on the building step, refer to section [Building](#building). In case of an *in-source* build the *runscripts* can be generated by running: ```console $ ./make_runscripts ``` To run the model, switch to the `run` subdirectory of the build directory and submit the runscript of your choice, for example: ```console $ cd ./run && sbatch ./exp.atm_amip.run ``` > **_NOTE:_** For more information on the runscript generation and running the model, refer to section [Running](#running). # Configuration The configuration step is done by calling the [configure](./configure) script with arguments specifying the location of libraries and tools required for building, as well as options enabling or disabling particular features of the model. For example: ```console $ ./configure CC=mpicc FC=mpif90 LIBS='-lnetcdff -lnetcdf -llapack -lblas' --disable-ocean --disable-coupling ``` > **_NOTE:_** Users are recommended to get familiar with the full list of configuration options and their default values by running: >```console >$ ./configure --help >``` The [configure](./configure) script of ICON is implemented using [Autoconf](https://www.gnu.org/software/autoconf/) and its interface should be familiar to those who have experience with [Autotools](https://en.wikipedia.org/wiki/GNU_Autotools)-based building systems. The following sections provide information on some features and implementation details of the configuration process. ## ICON dependencies [Fig. 1](#icon-depgraph) shows a partial dependency graph of the model. A dependency can be either *mandatory* (i.e. the library is required regardless of the specified configure options) or *optional* (i.e. the library is required only if particular features of the model are enabled), and some of the dependencies are provided together with the source code of ICON as git submodules and refered to as *bundled* later in the text. > **_NOTE:_** The term *bundled library* does not apply to all packages listed in [.gitmodules](/.gitmodules): some of them, e.g. JSBACH and ART, have circular dependencies with the ICON source code and therefore are treated as part of it.  *Fig. 1. ICON dependency graph* The list of libraries (packages) required for successful configuration and building depends on the selected options. To make the configuration process more transparent, the [configure](./configure) script does not accept paths to the installation directories of the packages, which would be used to extend the corresponding compiler flags. Instead, paths to the header and library files of the packages must be provided by the user as compiler and linker flags, i.e. in the `FCFLAGS`, `CPPFLAGS`, and `LDFLAGS` arguments. Moreover, the script does not try to guess the list of libraries to be used, therefore all the `-l` linker flags need to be specified in the `LIBS` argument in the correct order. The recommended (topologically sorted) order for the `LIBS` argument is presented in [Table 1](#icon-deptable), and the recommended order for the `FCFLAGS`, `CPPFLAGS`, `LDFLAGS` is the reversed one. *Table 1. Topologically sorted list of the ICON dependency graph* | Linking order | Package | Dependency condition¹ | Required flags¹ | | :---: | :---: | :---: | :---: | | 1 | [ICON-TIXI](https://gitlab.dkrz.de/icon-libraries/libtixi) (a modified version of [TIXI](https://github.com/DLR-SC/tixi)) | `--enable-art --with-external-tixi` | `FCFLAGS='-I/path/to/tixi/include' LDFLAGS='-L/path/to/tixi/lib' LIBS='-licon_tixi'` | | 2 | [YAC](https://doc.redmine.dkrz.de/YAC/html/) | `--enable-coupling --with-external-yac` | `FCFLAGS='-I/path/to/yac/include' LDFLAGS='-L/path/to/yac/lib' LIBS='-lyac'` | | 3 | [XML2](http://www.xmlsoft.org/) | `--enable-coupling`² or `--enable-art`²| `CPPFLAGS='-I/path/to/libxml2/include/libxml2' LDFLAGS='-L/path/to/libxml2/lib' LIBS='-lxml2'` | | 4 | [MTIME](https://gitlab.dkrz.de/icon-libraries/libmtime) (Fortran interface) | `--with-external-mtime`³ | `FCFLAGS='-I/path/to/mtime/include' LDFLAGS='-L/path/to/mtime/lib' LIBS='-lmtime'` | | 5 | [MTIME](https://gitlab.dkrz.de/icon-libraries/libmtime) (C interface) | `--enable-coupling --without-external-yac --with-external-mtime` | `CPPFLAGS='-I/path/to/mtime/include' LDFLAGS='-L/path/to/mtime/lib' LIBS='-lmtime'` | | 6 | [SERIALBOX](https://github.com/GridTools/serialbox) | `--enable-serialization` | `FCFLAGS='-I/path/to/serialbox2/include' LDFLAGS='-L/path/to/serialbox2/lib' LIBS='-lSerialboxFortran'` | | 7 | [CDI](https://code.mpimet.mpg.de/projects/cdi/) or CDI-PIO | `--with-external-cdi`⁴ | `FCFLAGS='-I/path/to/libcdi/include' LDFLAGS='-L/path/to/libcdi/lib' LIBS='-lcdi_f2003 -lcdi'` (or `LIBS='-lcdi_f2003 -lcdipio -lcdi'`) | | 8 | [ECCODES](https://confluence.ecmwf.int/display/ECC) (Fortran interface) | `--enable-emvorado` | `FCFLAGS='-I/path/to/eccodes/include' LDFLAGS='-L/path/to/eccodes/lib' LIBS='-leccodes_f90'` | | 9 | [ECCODES](https://confluence.ecmwf.int/display/ECC) or [GRIB-API](https://www.ecmwf.int/en/newsletter/152/news/end-road-grib-api)⁵ (C interface) | `--enable-grib2 --without-external-cdi` | `CPPFLAGS='-I/path/to/eccodes/include' LDFLAGS='-L/path/to/eccodes/lib' LIBS='-leccodes'` (or `LIBS='-lgrib_api'`) | | 10 | [YAXT](https://gitlab.dkrz.de/dkrz-sw/yaxt) (Fortran interface) | `--enable-yaxt --with-external-yaxt` or `--enable-cdi-pio --with-external-yaxt`⁶ | `FCFLAGS='-I/path/to/yaxt/include' LDFLAGS='-L/path/to/yaxt/lib' LIBS='-lyaxt'` | | 11 | [YAXT](https://gitlab.dkrz.de/dkrz-sw/yaxt) (C interface) | `--enable-coupling --with-external-yaxt`^{7, 8} | `CPPFLAGS='-I/path/to/yaxt/include' LDFLAGS='-L/path/to/yaxt/lib' LIBS='-lyaxt_c'` | | 12 | [SCT](https://gitlab.dkrz.de/dkrz-sw/sct) (Fortran interface) | `--enable-sct --with-external-sct` | `FCFLAGS='-I/path/to/sct/include' LDFLAGS='-L/path/to/sct/lib' LIBS='-lsct'` | | 13 | RTTOV (a modified version of [RTTOV](https://www.nwpsaf.eu/site/software/rttov/)) | `--enable-rttov` | `FCFLAGS='-I/path/to/rttov/include' LDFLAGS='-L/path/to/rttov/lib' LIBS='-lradiance -lrttov10.2'` | | 14 | [LAPACK](http://www.netlib.org/lapack/) (or analogue) | mandatory | `LDFLAGS='-L/path/to/lapack/lib' LIBS='-llapack'` (depends on the implementation) | | 15 | [BLAS](http://www.netlib.org/blas/) (or analogue) | mandatory | `LDFLAGS='-L/path/to/blas/lib' LIBS='-lblas'` (depends on the implementation) | | 16 | [ECRAD](https://confluence.ecmwf.int/display/ECRAD/ECMWF+Radiation+Scheme+Home) | `--enable-ecrad --with-external-ecrad` | `FCFLAGS='-I/path/to/ecrad/include' LDFLAGS='-L/path/to/ecrad/lib' LIBS='-lradiation -lifsrrtm -lutilities -lifsaux'` | | 17 | [RTE+RRTMGP](https://github.com/earth-system-radiation/rte-rrtmgp) | `--enable-rte-rrtmgp --with-external-rte-rrtmgp` | `FCFLAGS='-I/path/to/rte-rrtmgp/include' LDFLAGS='-L/path/to/rte-rrtmgp/lib' LIBS='-lrrtmgp -lrte'` | | 18 | [NetCDF-Fortran](https://www.unidata.ucar.edu/software/netcdf/docs-fortran/) | mandatory | `FCFLAGS='-I/path/to/netcdf-fortran/include' LDFLAGS='-L/path/to/netcdf-fortran/lib' LIBS='-lnetcdff'` | | 19 | [NetCDF-C](https://www.unidata.ucar.edu/software/netcdf/docs/) | `--without-external-cdi` or `--enable-coupling`⁹ | `CPPFLAGS='-I/path/to/netcdf/include' LDFLAGS='-L/path/to/netcdf/lib' LIBS='-lnetcdf'` | | 20 | [HDF5](https://support.hdfgroup.org/HDF5/) (low- and high-level Fortran interfaces) | `--enable-emvorado` or `--enable-rttov`¹⁰ | `FCFLAGS='-I/path/to/hdf5/include' LDFLAGS='-L/path/to/hdf5/lib' LIBS='-lhdf5_hl_fortran -lhdf5_fortran'` | | 21 | [HDF5](https://support.hdfgroup.org/HDF5/) (low-level C interface) | `--enable-sct --without-external-sct` | `CPPFLAGS='-I/path/to/hdf5/include' LDFLAGS='-L/path/to/hdf5/lib' LIBS='-lhdf5'` | | 22 | [ZLIB](https://zlib.net/) | `--enable-emvorado` | `LDFLAGS='-L/path/to/zlib/lib' LIBS='-lz'`¹¹ | | 23 | [AEC](https://gitlab.dkrz.de/k202009/libaec) or [SZIP](https://support.hdfgroup.org/doc_resource/SZIP/) | static linking | `LDFLAGS='-L/path/to/aec/lib' LIBS='-laec'` (or `LIBS='-lsz'`) | | 24 | [MPI](https://www.mpi-forum.org/) (Fortran interface) | `--enable-mpi`¹² or `--enable-yaxt` or `--enable-coupling --without-external-yac` | `FC='/path/to/mpi/bin/mpif90'` or `FCFLAGS='-I/path/to/mpi/include' LDFLAGS='-L/path/to/mpi/lib' LIBS='-lmpifort -lmpi'` (depends on the implementation) | | 25 | [MPI](https://www.mpi-forum.org/) (C interface) | `--enable-yaxt --without-external-yaxt` or `--enable-coupling`¹³ or `--enable-mpi --enable-sct --without-external-sct` | `CC=/path/to/mpi/bin/mpicc` or `CPPFLAGS=-I/path/to/mpi/include LDFLAGS='-L/path/to/mpi/lib' LIBS='-lmpi'` (depends on the implementation) | | 26 | [CUB](https://nvlabs.github.io/cub/) | `--enable-gpu --with-external-cub` | `NVCFLAGS=-I/path/to/cub`¹⁴ | | 27 | [CUDA](https://developer.nvidia.com/cuda-zone) | `--enable-gpu` | `LDFLAGS=-L/path/to/cuda/lib LIBS=-lcudart`¹⁵ | | 28 | [STDC++](https://gcc.gnu.org/onlinedocs/libstdc++/) | `--enable-gpu` | `LDFLAGS=-L/path/to/gcc/used/by/nvcc/lib LIBS=-lstdc++` | 1. The dependency conditions and required flags are specified assuming that the shared versions of the libraries containing `RPATH` entries pointing to their dependencies are used. Also note that the configure scripts of some of the bundled libraries (e.g. YAXT) run checks that link executables using libtool and then run them. Since some of the dependencies (e.g. ECCODES) are not libtool libraries (i.e. there are not .la files for them), the resulting executables do not have the corresponding `RPATH` entries, which triggers false negative results of the checks and fails the configuration with misleading messages. To circumvent this problem, the paths to the non-libtool dependencies must be passed to the linker before running the configure script of ICON, e.g. by means of the `LD_LIBRARY_PATH` environment variable. [↩](#f1-back) 2. There are no shared versions of YAC and ICON-TIXI libraries, which could link XML2 library implicitly, therefore, the latter needs to be linked explicitly, regardless of whether external or the bundled versions of YAC and ICON-TIXI are used. [↩](#f2-back) 3. When the coupling is enabled (`--enable-coupling`) and an external version of YAC (`--with-external-yac`) is used, usage of an external MTIME library (`--with-external-mtime`) is mandatory (must be the library that YAC has been built with). [↩](#f3-back) 4. Currently, the bundled version of CDI does not provide CDI-PIO features, therefore the usage of an external version of CDI (`--with-external-cdi`) is the only option when the parallel I/O features (`--enable-cdi-pio`) are required. [↩](#f4-back) 5. GRIB-API is supported only in cases when ECCODES is not required, e.g. when the radar forward operator EMVORADO, which requires the Fortran interface of ECCODES, is disabled (`--disable-emvorado`). [↩](#f5-back) 6. When usage of the parallel features of CDI is enabled (`--enable-cdi-pio`) and an external version of CDI (`--with-external-cdi`) is used, usage of an external YAXT library (`--with-external-yaxt`) is mandatory (must be the library that CDI has been built with). [↩](#f6-back) 7. When the coupling is enabled (`--enable-coupling`) and an external version of YAC (`--with-external-yac`) is used, usage of an external YAXT library (`--with-external-yaxt`) is mandatory (must be the library that YAC has been built with). [↩](#f7-back) 8. There is no shared version of YAC library, which could link YAXT (C interface) library implicitly, therefore, the latter needs to be linked explicitly, regardless of whether an external or the bundled version of YAC is used. [↩](#f8-back) 9. There is no shared version of YAC library, which could link NetCDF-C library implicitly, therefore, the latter needs to be linked explicitly, regardless of whether an external or the bundled version of YAC is used. [↩](#f9-back) 10. There is no shared version of RTTOV library, which could link Fortran HDF5 libraries implicitly, therefore, the latter need to be linked explicitly when usage of the radiative transfer model for TOVS is enabled (`--enable-rttov`). [↩](#f10-back) 11. ZLIB is used via the `ISO_C_BINDING` interface and does not require additional preprocessor flags. [↩](#f11-back) 12. When usage of the parallel features of CDI is enabled (`--enable-cdi-pio`), MPI (parallelization) support (`--enable-mpi`) is mandatory. [↩](#f12-back) 13. There is no shared version of YAC library, which could link MPI (C interface) library implicitly, therefore, the latter needs to be linked explicitly when MPI (parallelization) support is enabled (`--enable-mpi`), regardless of whether an external or the bundled version of YAC is used. [↩](#f13-back) 14. CUB is a header-only library and does not require additional linker flags. [↩](#f14-back) 15. Currently, the only Fortran compiler that supports all features required to build the GPU version of ICON is PGI, which specifies linker flags enabling CUDA automatically. [↩](#f15-back) ## Bundled libraries As it was mentioned in the previous section, some of the libraries are bundled together with the ICON source code. However, users can download and install those libraries before configuring ICON and use them instead. This is controlled by the `--with-external-` arguments of the [configure](./configure) script of ICON. The arguments accept either `yes` or `no`. If the usage of an external version of a library is requested (i.e. `--with-external-=yes`), the compiler and linker flags, i.e. `FCFLAGS`, `CPPFLAGS`, `LDFLAGS` and `LIBS`, are supposed to be extended accordingly (see [Table 1](#icon-deptable)). The [configure](./configure) script of ICON fails if the flags are not set correctly or enable a version of the library that is known to be incompatible with ICON. By default, the bundled versions of the libraries are used. In this case, the [configure](./configure) script of ICON runs the configure scripts of the libraries and extends the compiler and linker flags automatically. The arguments that are passed to the configure scripts of the bundled libraries are composed based on the arguments provided to the [configure](./configure) script of ICON as follows: - by default, the arguments are passed unchanged, which means that if you need to give an additional argument to the configure script of a bundled library, you can specify it when calling the [configure](./configure) script of ICON, even though, the argument is not listed in its help message; - arguments that potentially break the configuration and building of ICON are filtered out, for example, it is important that the bundled libraries are linked statically, therefore the argument `--enable-shared` is never passed to the configure scripts that support it; - the list of arguments is extended with ones that enforce consistent building, for example, the configure scripts of the libraries receive additional arguments `--disable-shared` and `--enable-static` (see calls to the `ACX_CONFIG_SUBDIR` macro in [configure.ac](./configure.ac) for the full lists of argument modifications done by the [configure](./configure) script of ICON for each particular library); - compiler flags are modified as described in section [Compiler flags](#compiler-flags). > **_NOTE:_** Oftentimes, error and warning messages that a printed at configure time are emitted not by the [configure](./configure) script of ICON but by the configure scripts of the bundled libraries. Each of the scripts generates its own `config.log` file, which can help in better understanding of the reported issue. The log files are put to the corresponding subdirectories of the `./externals` directory residing in the root build directory of ICON. ## Compilers and tools Compilers and tools to be used for building are read by the [configure](./configure) script of ICON from the following environment variables: - `FC` — Fortran compiler command; - `CC` — C compiler command; - `NVCC` — [NVIDIA CUDA Compiler](https://developer.nvidia.com/cuda-llvm-compiler) command (used only when the GPU support is enabled); - `PYTHON` — [Python](https://www.python.org/) interpreter command (used for [source dependency tracking](#source-dependency-tracking) and some of the [preprocessing](#preprocessing) steps); - `PERL` — [Perl](https://www.perl.org/) interpreter command (used for [source provenance collection](#source-provenance-collection)); - `CLAW` — [CLAW](https://claw-project.github.io/) compiler (source preprocessor) command (see section [Preprocessing](#preprocessing) for more details); - `FPP` — Fortran preprocessor command (used when explicit Fortran preprocessing is enabled, see section [Preprocessing](#preprocessing) for more details), must treat the first positional command-line argument as the path to the input source file and print the result to the standard output stream; - `SB2PP` — [Serialbox2](https://gridtools.github.io/serialbox/) preprocessor command (used when the Serialbox2 serialization is enabled, see section [Preprocessing](#preprocessing) for more details); - `MPI_LAUNCH` — interactive (synchronous) MPI launcher command (used by the bundled libraries for configure-time checks). If the variables are set, the [configure](./configure) script will check whether their values meet the requirements, otherwise the script will try to guess suitable values for them. Thus, if you want to make sure that a particular command for a particular operation is used, you need to specify the corresponding variable explicitly. For example, the usage of NAG compiler is enforced with the following additional command-line argument of the [configure](./configure) script: ```console $ ./configure FC=nagfor ``` ## Compiler flags The configure script supports several groups of compiler flags. Each group is associated with one of the following environment variables: - `FCFLAGS` — Fortran compiler flags to be used when configuring, compiling and linking ICON, as well as passed to the configure scripts of the bundled libraries (in contrast to standard [Autoconf](https://www.gnu.org/software/autoconf/)-based scripts, the [configure](./configure) script of ICON does not set `FCFLAGS` to `-g -O2` by default); - `ICON_FCFLAGS` — Fortran compiler flags to be appended to `FCFLAGS` when configuring, compiling and linking ICON; - `ICON_OCEAN_FCFLAGS` — Fortran compiler flags to be appended to `FCFLAGS` instead of `ICON_FCFLAGS` when compiling the ocean component of ICON, i.e. the Fortran source files residing in subdirectories of the [src/hamocc](./src/hamocc), [src/ocean](./src/ocean), and [src/sea_ice](./src/sea_ice) directories (defaults to `ICON_FCFLAGS`, which can be overridden by setting the variable to an empty value: `ICON_OCEAN_FCFLAGS=`); - `ICON_BUNDLED_FCFLAGS` — Fortran compiler flags to be appended to `FCFLAGS` when configuring the bundled libraries (defaults to `ICON_FCFLAGS`, which can be overridden by setting the variable to an empty value: `ICON_BUNDLED_FCFLAGS=`); - `ICON_RTE_RRTMGP_FCFLAGS`, `ICON_ECRAD_FCFLAGS`, etc. — Fortran compiler flags to be appended to `FCFLAGS` when configuring the respective bundled libraries (defaults to `ICON_BUNDLED_FCFLAGS`, which can be overridden by setting the variablies to empty values: `ICON_RTE_RRTMGP_FCFLAGS=`, `ICON_ECRAD_FCFLAGS=`, etc.); - `CFLAGS` — C compiler flags to be used when configuring and compiling ICON, as well as passed to the configure scripts of the bundled libraries (in contrast to standard [Autoconf](https://www.gnu.org/software/autoconf/)-based scripts, the [configure](./configure) script of ICON does not set `FCFLAGS` to `-g -O2` by default); - `CPPFLAGS` — C preprocessor flags to be used when configuring and compiling ICON, as well as passed to the configure scripts of the bundled libraries; - `ICON_CFLAGS` — C compiler flags to be appended to `CFLAGS` when configuring and compiling ICON; - `ICON_BUNDLED_CFLAGS` — C compiler flags to be appended to `CFLAGS` when configuring the bundled libraries (defaults to `ICON_CFLAGS`, which can be overridden by setting the variable to an empty value: `ICON_BUNDLED_CFLAGS=`); - `ICON_CDI_CFLAGS`, `ICON_MTIME_CFLAGS`, etc. — C compiler flags to be appended to `CFLAGS` when configuring the respective bundled libraries (defaults to `ICON_BUNDLED_CFLAGS`, which can be overridden by setting the variablies to empty values: `ICON_CDI_CFLAGS=`, `ICON_MTIME_CFLAGS=`, etc.); - `NVCFLAGS` — [NVIDIA CUDA Compiler](https://developer.nvidia.com/cuda-llvm-compiler) flags to be used when configuring and compiling ICON; - `CLAWFLAGS` — extra [CLAW](https://claw-project.github.io/) compiler flags to be used at the building stage together with the flags that are composed automatically based on the [configure](./configure) options (e.g. Fortran compiler flags specifiying search paths (`-I`) and macros (`-D`) found in `FCFLAGS` are passed to the CLAW preprocessor); - `LDFLAGS` — common Fortran and C compiler flags to be used when configuring and linking ICON, as well as passed to the configure scripts of the bundled libraries; - `ICON_LDFLAGS` — Fortran compiler flags to be appended to `LDFLAGS` when configuring and linking ICON; - `LIBS` — a list of libraries (see [Table 1](#icon-deptable) for the recommended order) to be passed to the linker by the Fortran compiler when linking ICON and to the configure scripts of the bundled libraries (which might use both Fortran and C compiler for linking). > **_NOTE:_** It is recommended to specify the environment variables that influence the configuration as command line arguments of the [configure](./configure) script in the form `VAR=VALUE`, for example: >```console >$ ./configure FCFLAGS='-O2 -g' >``` The general recommendation to follow when composing the flags is: 1. Flags specifying search paths for header and module files, i.e. the `-I` flags, should be specified as `FCFLAGS` and `CPPFLAGS`, depending on whether they need to be passed to Fortran or C compiler, respectively. 2. By default, other flags, e.g. the optimization ones, that are meant for Fortran and C compilers should be appended to `FCFLAGS` and `CFLAGS`, respectively. 3. Fortran and C compiler flags that need to be used when configuring, compiling and linking ICON but at the same time can break the configuration (a flag is too restrictive for the configure checks to pass, e.g. `-fimplicit-none` for Gfortran) or the functionality of the bundled libraries (e.g. the optimization level required for ICON is too high and leads to errors in the functionality of the bundled libraries) can be put to `ICON_FCFLAGS`, `ICON_CFLAGS` or `ICON_LDFLAGS`. 4. Special optimization flags for the ocean component of ICON can be put to `ICON_OCEAN_FCFLAGS`. 5. Fortran and C compiler flags that need to be used when compiling and linking the bundled libraries but at the same time conflict with the flags required for ICON (e.g. you want to compile ICON with `-O3` flag but the bundled libraies need to be compiled with `-O2`) can be specified as `ICON_BUNDLED_FCFLAGS` and `ICON_BUNDLED_CFLAGS`, respectively. 6. If a set of Fortran (or C) compiler flags needs to be passed only to some particular bundled library, it can be specified in the respective variable, e.g. in `ICON_CDI_FCFLAGS` (or `ICON_CDI_CFLAGS`). ## Dynamic libraries For each `-L` flag found in the `LDFLAGS` and `LIBS` variables, the [configure](./configure) script of ICON generates an additional linker flag that puts the `` on the list of runtime library search paths of the ICON executable. This allows for automatic location of the required libraries by the *dynamic linker* at the runtime. The flags are appended to `LDFLAGS` at the configure time and their actual form depends on the Fortran compiler in use. By default, the flags are composed using the template `-Wl,-rpath -Wl,` with currently the only exception for NAG compiler, which accepts the flags in the form `-Wl,-Wl,,-rpath -Wl,-Wl,,`. If the `-rpath` flags generated by the configure script break the building or you perform a completely static linking, you can disable the feature by calling the [configure](./configure) script with an additional argument `--disable-rpaths`. > **_NOTE:_** The GNU Linker (GNU ld) implements a feature called `new-dtags`. If this feature is enabled (usually by default), the linker treats the `-rpath ` flag differently: besides setting the `DT_RPATH` attribute of the output shared library file to the ``, it also sets the `DT_RUNPATH` attribute of the file to the same value. This alters the way the dynamic linker locates the required dynamic libraries at the runtime: if the dynamic linker finds a `DT_RUNPATH` attribute, it **ignores** the value of the `DT_RPATH` attribute, with the effect that the `LD_LIBRARY_PATH` environmane variable is checked first and the paths in the `DT_RUNPATH` attribute are only searched afterwards. Moreover, the dynamic linker does not search `DT_RUNPATH` locations for transitive dependencies, unlike `DT_RPATH`. Therefore, it is important to keep in mind that: > 1. The ICON executable is not agnostic to the environment if it has been linked with the `new-dtags` feature enabled: the `LD_LIBRARY_PATH` environment variable can override the rpath entries set by the linker. The feature can be disabled by appending the `-Wl,--disable-new-dtags` flags to the `LDFLAGS` variable. > 2. If an immediate ICON dependency, e.g. `libnetcdf.so`, has at least one `DT_RUNPATH` entry but none of them points to a directory containing one of the libraries required by that dependency, e.g. `libsz.so`, the dynamic linker will not be able to locate the latter at the runtime even if the ICON executable has been linked without the `new-dtags` feature and an `-Wl,-rpath` flag pointing to the right location, e.g. `-Wl,--disable-new-dtags -Wl,-rpath -Wl,/path/to/libsz`. A possible workaround for this is to make the secondary dependency of ICON a primary one by *overlinking* (i.e. linking to a library, which is not used by the executable directly) to it, e.g. by adding the `-lsz` to the `LIBS` variable. This way, the dependency will become a non-transitive one and the dynamic linker will be able to locate it using either `DT_RUNPATH` or `DT_RPATH` entries of the ICON executable. > > For more details, refer to the man pages of the linker (`man ld`) and the dynamic linker (`man ld.so`). > **_NOTE:_** Some of the bundled libraries employ [Libtool](https://www.gnu.org/software/libtool/), which is known to be **not** fully compatible with some compilers. For example, the flags `-Wl,-Wl,,-rpath -Wl,-Wl,,`, which are valid for NAG compiler, are incorrectly transformed by Libtool into `-Wl,-Wl -Wl,"" -Wl,-rpath -Wl,-Wl -Wl,"" -Wl,`. A possible solution for this problem is to add the flags in the form understood by NAG compiler not to `LDFLAGS` but to `ICON_LDFLAGS`. > **_NOTE:_** Due to a significant maintenance overhead (see https://gitlab.dkrz.de/icon/icon/-/commit/36ab00dd5641419e5afbb918b47cdf697c54b737#87db583be5c13c1f7b3c958b10e03d67b6a2ca06), the generated `-rpath` flags are **not** passed to the configure scripts of the bundled libraries. However, some of their checks imply running executables linked with the flags listed in the `LIBS` variable. To prevent those checks from false negative results, which oftentimes are reported with misleading messages, the dynamic linker needs to be able to locate all the libraries referenced in the `LIBS` variable. A way to achive that is to list paths to the libraries in the `LD_LIBRARY_PATH` variable and export it, i.e. run `export LD_LIBRARY_PATH="::..."` before running the [configure](./configure) script. ## Configuration and building environments It is important that both the configuration and the building stages are performed in the same environment, i.e. the environment variables that might influence the way the compilers and the linker work are set to the same values when running the [configure](./configure) script and when running the `make` command for building. For example, NAG compiler will not work if the environment variable `NAG_KUSARI_FILE` is not set properly. Keeping track of all the steps required to re-initialize the environment for the building stage, e.g. when the configuration stage has already been done but in another terminal session, might be challenging, especially in HPC environments offering multiple compilers and libraries. One way to make sure that the configuration and the building environments are consistent is to set the `BUILD_ENV` argument of the [configure](./configure) script to a set of shell commands that initialize the environment variables with the required values. If the `BUILD_ENV` variable is not empty, the [configure](./configure) script will run the commands it contains before running any checks. Additionally, the commands will be saved to the `Makefile`, so they will be executed each time the `make` command is launched for building. The shell script that is provided as value of the `BUILD_ENV` argument must be a one-liner ending with a semicolon (;) symbol, for example: ```console $ ./configure BUILD_ENV='. /etc/profile.d/modules.sh; module purge; module load intel;' ``` Also, a proper implementation of the `BUILD_ENV` script allows for switching between multiple build directories (see section [Out-of-source configuration (building)](#out-of-source-configuration-building)) without having to re-initialize the environment accordingly. ## Configuration wrappers Real case configuration commands might be rather long and complex. For example, below, you can find an example of the configuration command for [Mistral@DKRZ](https://www.dkrz.de/up/systems/mistral): ```console $ ./configure \ AR=xiar \ BUILD_ENV=". /sw/rhel6-x64/etc/profile.mistral; \ . ./config/dkrz/module_switcher; \ switch_for_module gcc/6.4.0 intel/17.0.6 openmpi/2.0.2p1_hpcx-intel14;" \ CC=mpicc \ CFLAGS='-gdwarf-4 -O3 -qno-opt-dynamic-align -ftz -march=native -g' \ CPPFLAGS="-I/sw/rhel6-x64/hdf5/hdf5-1.8.18-parallel-openmpi2-intel14/include \ -I/sw/rhel6-x64/netcdf/netcdf_c-4.4.0-parallel-openmpi2-intel14/include \ -I/sw/rhel6-x64/grib_api/grib_api-1.15.0-gcc48/include \ -I/usr/include/libxml2" \ FC=mpif90 \ FCFLAGS="-I/sw/rhel6-x64/netcdf/netcdf_fortran-4.4.3-parallel-openmpi2-intel14/include \ -gdwarf-4 -g -march=native -pc64 -fp-model source" \ ICON_FCFLAGS='-O2 -assume realloc_lhs -ftz' \ ICON_OCEAN_FCFLAGS='-O3 -assume norealloc_lhs -reentrancy threaded -qopt-report-file=stdout -qopt-report=0 -qopt-report-phase=vec' \ LDFLAGS="-L/sw/rhel6-x64/hdf5/hdf5-1.8.18-parallel-openmpi2-intel14/lib \ -L/sw/rhel6-x64/netcdf/netcdf_c-4.4.0-parallel-openmpi2-intel14/lib \ -L/sw/rhel6-x64/netcdf/netcdf_fortran-4.4.3-parallel-openmpi2-intel14/lib \ -L/sw/rhel6-x64/grib_api/grib_api-1.15.0-gcc48/lib \ -mkl=sequential" \ LIBS='-Wl,--as-needed -lxml2 -lgrib_api -lnetcdff -lnetcdf -lhdf5' \ MPI_LAUNCH=mpiexec \ --enable-intel-consistency \ --enable-vectorized-lrtm \ --enable-parallel-netcdf ``` Obviously, repeatedly composing such a command is exhausting and error-prone. Therefore, each team involved in the development of ICON is encouraged to implement and maintain *configuration wrappers*, which would simplify the configuration stage for their users. The wrappers should be put in a subdirectory with a relevant name of the [./config](config) directory. Although there are no hard requirements on how the scripts should be implemented, we recommend to consider the following features: 1. Pass the command-line arguments of the wrapper script to the [configure](./configure) script, so that the users are able to override the default values of the configure options, for example: ```console $ ./config/dkrz/mistral.intel --enable-openmp ``` Also, account for the case of calling the wrapper with the `--help` argument. 2. Account for out-of-source building: - the wrapper script should be able to locate the [configure](./configure) script when called from a directory other than the source root directory, for example: ```bash SCRIPT_DIR=$(cd "$(dirname "$0")"; pwd) ICON_DIR=$(cd "${SCRIPT_DIR}/../.."; pwd) "${ICON_DIR}/configure" "$@" ``` - the wrapper script should prepare the current working directory for the following runscript generation (see section [Running](#running) for more details). 3. Prepend the `LIBS` variable with `-Wl,--as-needed` flag so that the actual list of the libraries the ICON executable depends on would include only those required for the particular configuration of the model (see the man pages of the linker for more details: `man ld`). 4. Allow for running the `make check` command (see section [Building of the bundled libraries](#building-of-the-bundled-libraries) for more details) by extending the `LD_LIBRARY_PATH` environmane variable in the `BUILD_ENV` script, instead of doing so in the wrapper script itself. ## Out-of-source configuration (building) The building system of ICON supports so-called *out-of-source* builds. This means that you can build ICON in a directory other than the *source* root directory. The main advantage of this is that you can easily switch between several different configurations (each in its own *build* directory) of the model, while working on the same source code. Hence, it is possible to introduce changes into the source code and test them with different compilers, flags and options without having to re-configure the model or copy the updated source files to other directories. The [configure](./configure) script (also when called via a configuration wrapper) prepares the *current working directory* for the following building. The particular case of the current working directory being the source root directory is called *in-source* configuration. > **_NOTE:_** It is not allowed to mix in-source and out-of-source builds: the source directory must be cleaned from the files generated as a result of a prior in-source configuration and building before an out-of-source configuration can take place. The following example shows how ICON can be configured on [Mistral@DKRZ](https://www.dkrz.de/up/systems/mistral) in two different directories using Intel and GCC compiler toolchains (assuming that the source root directory of ICON is `/path/to/icon-srcdir`): ```console $ mkdir intel && cd intel $ /path/to/icon-srcdir/config/dkrz/mistral.intel $ cd .. $ mkdir gcc && cd gcc $ /path/to/icon-srcdir/config/dkrz/mistral.gcc $ cd.. ``` As a result of the previous commands, you will have two directories `intel` and `gcc` prepared for the following building of ICON using Intel and GCC compilers, respectively. # Building The building stage is done with [GNU make](https://www.gnu.org/software/make/) upon successful completion of the configuration stage. The oldest supported version of `make` is **3.81**, however, it has significant limitations and it is recommended to use version **4.1** or later. The building step is done by running `make` command with an optional argument specifying the number of jobs to run simultaneously. For example: ```console $ make -j8 ``` > **_NOTE:_** Users are recommended to get familiar with the list of basic *targets*, i.e. supported subcommands, by running: >```console >$ make help >``` The *makefiles* of ICON, i.e. the instructions for `make` on how to build the model, are generated at the final step of the configuration stage based on the template files. Unlike most of the [Autotools](https://en.wikipedia.org/wiki/GNU_Autotools)-based packages, the makefile templates of ICON are **not** generated automatically with [Automake](https://www.gnu.org/software/automake/) but implemented manually. The main reasons for that are, first, avoid the need to maintain an explicit list of source files, and, second, keep the makefiles human-readable and, therefore, human-hackable, which is often required for productive model development. By default, `make` reads the instructions from a file called `Makefile`, which in the case of ICON is generated based on the [Makefile.in](./Makefile.in) template and is mainly responsible for the initialization of the building environment. The initialization is performed by the shell script provided to the [configure](./configure) script as the `BUILD_ENV` argument (see section [Configuration and building environments](#configuration-and-building-environments)). The script is saved in the `Makefile`. Each time `make` is executed in the root of the ICON build directory, it reads the `Makefile`, runs the initialization script and [recursively](https://www.gnu.org/software/make/manual/html_node/Recursion.html) executes itself in the same directory but with another input makefile called `icon.mk`. The latter is generated based on the [icon.mk.in](./icon.mk.in) template and contains the instructions on how to perform the following building steps required to generate the ICON executable. > **_NOTE:_** All modifications of the makefiles and other files that are automatically generated at the configuration stage can be reverted by calling the `./config.status` script residing in the build directory. The following sections provide information on some features and implementation details of the building process. > **_NOTE:_** By default, `make` produces very minimalistic output to the standard output stream. This allows for better recogniction of the warnings messages emitted by the compilers and other tools. This can be altered either at the configuration stage by calling the [configure](./configure) script with an additional option `--disable-silent-rules` or at the building stage by calling `make` with an additional command-line argument `V`, which can be set either to `1` (enable verbose output) or to `0` (disable verbose output). For example, if you want to see the exact commands executed by `make`, you can run: >```console >$ make V=1 >``` ## Source file collection The list of source files that need to be compiled to produce the ICON executable is generated dynamically each time the `make` command is executed. This is done using `find` command (both [GNU](https://www.gnu.org/software/findutils/) and [BSD](https://www.freebsd.org/cgi/man.cgi?find(1)) versions are supported) with the assumption that the source files have the following filename extensions: - `.f90` — Fortran source files, regardless of whether they contain Fortran preprocessor directives; - `.inc` — Fortran header files included with the quoted form of the Fortran preprocessor `#include` directives, e.g. `#include "filename.inc"`; - `.incf` — Fortran header files included with the Fortran `INCLUDE` statements, e.g. `include 'filename.incf'` (these files are not allowed to have Fortran preprocessor directives); - `.c` — C source files; - `.cu` — CUDA source files. The list of source files is a result of recursive search for files that have the aforementioned extenions and reside in the [src](./src) and [support](./support) subdirectories of the source root directory of ICON. Additionaly, depending on the whether the corresponding components of the model were enabled at the configuration stage, the list is extended with Fortran source files from the `./externals/jsbach/src`, `./externals/dace_icon/src_for_icon`, `./externals/emvorado` and `./externals/art` subdirectories. > **_NOTE:_** In general, you can extend the source base of ICON just by adding the source files to the [src](./src) subdirectory of the source root directory of ICON. ## Preprocessing Depending on the configuration, Fortran source files undergo one or more of the following preprocessing precedures. 1. Fortran source files residing in the `./externals/jsbach/src` are preprocessed with the [dsl4jsb.py](https://gitlab.dkrz.de/jsbach/jsbach/blob/master/scripts/dsl4jsb/dsl4jsb.py) script. This is done only if the JSBACH component has been enabled at the configuration stage (`--enable-jsbach`). Otherwise, the source files of the component are completely ignored. The result of this procedure is stored to the `./pp/dsl4jsb` subdirectory of the root build directory. 2. Depending on whether the *CLAW preprocessing* (`--enable-claw`) is enabled, the result of the previous step (currently, only JSBACH files are preprocessed at this step) are preprocessed with the [CLAW](https://claw-project.github.io/) compiler. The result of this procedure is stored to the `./pp/claw` subdirectory. > **_NOTE:_** Unlike the rest of the preprocessing steps, the instructions for CLAW preprocessing are moved to a separate makefile called `claw.mk`, which is generated based on the [claw.mk.in](./claw.mk.in) template. This is done to enable an extra dependency generation step required for the preprocessing. 3. Depending on whether the *explicit Fortran preprocessing* is enabled (`--enable-explicit-fpp`), the results of the **actual** previous preprocessing step, together with Fortran source files that have not been preprocessed yet, are preprocessed with the standard Fortran preprocessor. The result of this procedure is stored to the `./pp/fpp` subdirectory. 4. If the *Serialbox2 serialization* is enabled (`--enable-serialization`), the results of the **actual** previous preprocessing step, together with Fortran source files that have not been preprocessed yet, are preprocessed with the [corresponding script](https://github.com/GridTools/serialbox/blob/master/src/serialbox-python/pp_ser/pp_ser.py) of the [Serialbox2](https://gridtools.github.io/serialbox/) toolkit. The result of this procedure is stored to the `./pp/sb2` subdirectory. > **_NOTE:_** The explicit Fortran preprocessing is enabled automatically when the Serialbox2 serialization is enabled. You can override this by disabling the preprocessing with the `--disable-explicit-fpp` option. The output directories of the preprocessing steps have the same layout as the directories containing their input files and the output files have the same basenames as the corresponding input files. For example if the original source file of JSBACH `./externals/jsbach/src/base/mo_jsb_base.f90` is preprocessed by each of the preprocessing steps, the corresponding output files are saved as follows: - `./pp/dsl4jsb/externals/jsbach/src/base/mo_jsb_base.f90` — JSBACH preprocessing output; - `./pp/claw/pp/dsl4jsb/externals/jsbach/src/base/mo_jsb_base.f90` — CLAW preprocessing output; - `./pp/fpp/pp/claw/pp/dsl4jsb/externals/jsbach/src/base/mo_jsb_base.f90` — explicit Fortran preprocessing output; - `./pp/sb2/pp/fpp/pp/claw/pp/dsl4jsb/externals/jsbach/src/base/mo_jsb_base.f90` — Serialbox2 preprocessing output. > **_NOTE:_** Source files are additionaly preprocessed with the corresponding standard language-specific (i.e. Fortran, C, CUDA) preprocessors as part of the compilation process. In contrast to the procedures described in this section, the compilation-time preprocessing is non-optional and run by the compilers *implicitly*. ## Source dependency tracking Before the compilation can take place, it is required to identify the exact list of source files that need to be compiled in order to produce the ICON executable. The actual content of the list depends not only on how the code is [configured](#configuration), possible modifications of the source code made since the last call of `make` need to be taken into account too. Moreover, in the case of Fortran source files, the compilation order becomes important since a source file declaring a Fortran module must be compiled before any other source file using that module. Both tasks are accomplished as follows. Once the [preprocessing](#preprocessing) is finished, all source files of ICON (including the enabled components) or their **final** preprocessed versions are processed with the [dependency generator](./utils/mkhelper/depgen.py). The tool parses each source file, detects which header and module files are required for its succesful compilation, and stores this information in the form of a makefile. The makefiles a then read by `make` and the [dependency listing script](./utils/mkhelper/deplist.py). The former makes sure that the source files are compiled in the right order, the latter identifies the list of source files that need to be compiled. > **_NOTE:_** The [dependency listing script](./utils/mkhelper/deplist.py) recognizes only the basic makefile syntax. For example, it does not support [variables](https://www.gnu.org/software/make/manual/make.html#Using-Variables) and [functions](https://www.gnu.org/software/make/manual/make.html#Functions). The [dependency generator](./utils/mkhelper/depgen.py) recognizes preprocessor `#include`, `#if` and the associated directives as well as Fortran `INCLUDE`, `USE` and `MODULE` statements. If the usage of a module or a header file is surrounded with the `#ifdef SOME_MACRO` and `#endif` directives, it will be put on the list of files required for the compilation only if the macro `SOME_MACRO` is defined. The list of macro definitions enabling various features of the model is generated at the [configuration](#configuration) stage in the form of compiler flags, e.g. `-DSOME_MACRO -DSOME_OTHER_MACRO`, which are appended to `FCFLAGS`. This way, some Fortran modules do not need to be generated in particular configurations of the model and, therefore, the source files declaring them are not compiled. > **_NOTE:_** There are two types of source dependencies that cannot be detected by the [dependency generator](./utils/mkhelper/depgen.py). Undetectable dependencies of the first type are related to [C/Fortran interoperability](http://fortranwiki.org/fortran/show/C+interoperability): if a Fortran source file contains a declaration of a binding to a function defined in a C source file, the dependency of the respective object files must be reflected in the recipe for target `c_binding.d` of the [icon.mk.in](./icon.mk.in) template. The second type of undetectable dependencies is associated with Fortran external procedures: if a Fortran source file contains a call to an external procedure, i.e. a function or a subroutine that is not part of any Fortran module, the dependency of the respective object files must be specified in the recipe for target `extra_f90.d` of the [icon.mk.in](./icon.mk.in) template. The [dependency listing script](./utils/mkhelper/deplist.py) reads the dependency makefiles, builds a source dependency graph and traverses it starting with the vertex associated with the `src/drivers/icon.o` object file. Each vertex of the graph accessible from the starting one is printed to the output. The output is then filtered by `make` in order to generate the list of object files required for the ICON executable. This is the main purpose of the listing script. Additionaly, the tool can run [code consistency checks](#code-consistency-checks) described in the following subsection. ### Code consistency checks Each failed code consistency check run by the [dependency listing script](./utils/mkhelper/deplist.py) is reported to the standard error stream. The identified problems are expressed in terms of files and makefile dependencies and, therefore, require additional explanation provided in this subsection. Normally, the codebase is kept consistent and users do not see the messages described below until they introduce a modification to the source code that breaks the consistency. Currently, the [dependency listing script](./utils/mkhelper/deplist.py) checks the source dependency graph for the following problems: 1. **Two or more Fortran source files declare modules with the same name.** This type of inconsistency is reported as follows: ``` deplist.py: WARNING: target 'mod/some_module.mod.proxy' has more than one immediate prerequisite matching pattern '*.o': some/dir/some_file.o some/other/dir/some_other_file.o ``` This means that the Fortran module `some_module` is declared twice. The first declaration is found in the file `some/dir/some_file.f90` and the second declaration is found in `some/other/dir/some_other_file.f90`. 2. **Two or more Fortran modules circularly depend on each other.** This type of inconsistency is reported as follows: ``` deplist.py: WARNING: the dependency graph has a cycle: src/drivers/icon.o ... mod/some_module.mod.proxy some/dir/some_file.o mod/some_module_1.mod.proxy <- start of cycle some/other/dir/some_file_1.o mod/some_module_2.mod.proxy some/other/dir2/some_file_2.o mod/some_module_1.mod.proxy <- end of cycle ``` This reads as that the module `some_module_1` (declared in `some/dir/some_file_1.f90`) uses module `some_module_2` (declared in `some/other/dir/some_file_2.f90`), which in turn uses `some_module_1`. Usually, this means that the compilation of `some/dir/some_file_1.f90` will fail. 3. **A Fortran module is used but not declared.** This problem is reported by the dependency listing script with the following message: ``` deplist.py: WARNING: target 'mod/missing_module.mod.proxy' does not have an immediate prerequisite matching any of the patterns: '*.o' ``` This means that the module `missing_module` is used in one of the source files but there is no Fortran source file in the ICON codebase that declares it. It might be the case, however, that the module is not actually missing but just not part of the ICON codebase, e.g. `mpi`, `sct`, `yaxt`, etc. Such modules are external to ICON and need to be explicitly specified as such in the file `depgen.f90.config` residing in the current build directory (the file is generated at configuration time based on a template file residing in the source directory. Therefore, in order to make the modifications persistent, you need to introduce them in the file [depgen.f90.config.in](./depgen.f90.config.in). 4. **Two or more source files have the same basename.** The problem is reported as follows: ``` deplist.py: WARNING: the dependency graph contains more than one target with basename 'some_file.o': some/dir/some_file.o some/other/dir/some_file.o ``` This message reports about two (or more) source (not necessarily Fortran) files `some/dir/some_file.f90` and `some/other/dir/some_file.f90` that compile into objects with the same [basename](https://docs.python.org/2.7/library/os.path.html#os.path.basename). Although handled by the building system in most cases, having several source files with the same basename in a project is considered bad practice, potentially has negative side effects, and, therefore, is deprecated. ### Compilation cascade prevention It is important, especially for the development process, that the modifications of the source code done after the initial compilation trigger as few recompilations as possible. One of the basic features of `make` is to keep track of the file modification timestamps. Based on the information from the makefiles generated by the [dependency generator](./utils/mkhelper/depgen.py), the tool triggers recompilation of a source file only if the file itself or a header file it includes, or a Fortran module file it uses has been modified since the last execution. Unfortunately, most of the Fortran compilers (with an exception of Gfortran) update the module files even if their relevant contents do not change, i.e. the modification timestamp of a module file gets updated even if the declaration of the associated Fortran module in the source file remains the same. This leads to so-called *compilation cascades*. Partially, this issue is circumvented in the building system of ICON as follows. 1. If a Fortran source file `filename.f90` uses a module `modulename`, the corresponding dependency makefile `filename.f90.d` (created by the [dependency generator](./utils/mkhelper/depgen.py)) gets an entry declaring the dependency of the respective object file on a module *proxy* file: ```make filename.o: mod/modulename.mod.proxy ``` 2. When the compilation of the file declaring the module `modulename` takes place for the first time, the original module file `mod/modulename.mod` generated by the compiler is backed up under the name `mod/modulename.mod.proxy`. 3. When `make` checks whether the object file `filename.o` needs to be updated (i.e. the source file `filename.f90` needs to be recompiled), it compares the potentially updated original module file `mod/modulename.mod` with the proxy file `mod/modulename.mod.proxy` and triggers the recompilation only if they are *significantly different*. The latter fact is determined in two steps: first, the files a compared for binary identity with the `cmp` command, second, if the first check shows the difference, the contents of the files are compared with the [fortmodcmp.py](./utils/mkhelper/fortmodcmp.py) script, which employs compiler-specific heuristics. 4. Each time the proxy file `mod/modulename.mod.proxy` is detected to be significantly different from the original module file `mod/modulename.mod`, it is replaced with a new copy of the latter. The described mechanism helps to avoid compilation cascades in many cases. However, the structure of the module files generated by most of the compilers is usually not documented, which makes the comparison of the module files difficult. Thus, the redundant recompilations are not guaranteed to be eliminated entirely. ## Building of the bundled libraries The building of the [bundled libraries](#bundled-libraries) is based on the makefiles generated by their configure scripts. The makefiles are put to the corresponding subdirectories of the `./externals` directory residing in the root build directory of ICON. The libraries are built before the compilation of any Fortran source file of ICON takes place. This is done to make sure that the interface Fortran modules of the libraries are available in advance. > **_NOTE:_** Some of the bundled libraries, e.g. [CUB](https://nvlabs.github.io/cub/), do not require configuration and building and, therefore, are not handled at this building step. Once called in the build directory of ICON, `make` [recursively](https://www.gnu.org/software/make/manual/html_node/Recursion.html) runs itself in the build directories of the bundled libraries. The list of targets passed to the instances of `make` running in the directories of the bundled libraries depends on the list of targets specifed by the user when calling `make` in the build directory of ICON. The targets `all`, `mostlyclean`, `clean`, `distclean`, or `check` are preserved and passed over. All other targets are filtered out. The described mechanism imposes the following requirements on the makefiles of the bundled libraries: - the default targets of the makefiles (i.e. targets that are triggered when no targets are specified on the command-line) must be called `all` and are supposed to generate all files (libraries, module files, etc.) required for ICON; - the makefiles must implement at least dummy rules for targets `mostlyclean`, `clean`, and `distclean`, ideally, following the [heuristic adopted in Automake](https://www.gnu.org/software/automake/manual/html_node/Clean.html); - the makefiles must implement at least a dummy rule for target `check`, which ideally triggers self-tests of the library. > **_NOTE:_** Many of the bundled libraries have a collection of tests, which can be triggered by running `make check` command from the root build directory of ICON. The tests cant help to identify potential runtime problems at an early stage and make sure that the core functionality of a library works as expected in the given software environment. ## Source provenance collection Source provenance information is collected at the building stage and injected in the ICON executable. This information is saved at runtime in the output files of the model, so that the latter could be matched with the exact version of ICON that was used to produce them. The information is collected automatically with the help of the [pvcs.pl](./utils/pvcs.pl) script. The script generated a source file `version.c` containing the url of the git repository, the name of the git branch, and the hash of the git commit. The source file is then treated by `make` as part of the ICON codebase. # Running The executable of ICON is usually not launched directly. The common practice is to implement or generate a *runscript*, which sets up a working directory, populates it with all required input files (grid files, namelists, etc.), runs the model, and postprocesses its output. This section does not cover the details of how to set up an experiment but ruther provides superficial information on how to generate runscripts using standard templates. The building system of ICON is currently fully detached from the runscript generation mechanism. Therefore, the latter requires additional configuration steps before it can be used. > **_NOTE:_** Normally, the additional configuration steps enabling the runscript generation mechanism are performed by the [configuration wrappers](#configuration-wrappers). The generation of the runscripts is done with the [make_runscripts](./make_runscripts) shell script, which does not yet support the case of out-of-source builds. In particular, this means that the script itself, together with some other input files, needs to be copied from the source directory to the build directory of ICON. The following example shows how this can be done using the [rsync](https://rsync.samba.org/) tool: ```console $ cd /path/to/icon-builddir $ rsync -uavz /path/to/icon-srcdir/run . --exclude='*in' --exclude='.*' $ rsync -uavz /path/to/icon-srcdir/externals . --exclude='.git' \ --exclude='*.f90' --exclude='*.F90' --exclude='*.c' --exclude='*.h' \ --exclude='*.Po' --exclude='tests' --exclude='rrtmgp*.nc' --exclude='*.mod' \ --exclude='*.o' $ rsync -uavz /path/to/icon-srcdir/make_runscripts . $ ln -sf /path/to/icon-srcdir/data $ ln -sf /path/to/icon-srcdir/vertical_coord_tables ``` Once the runscript generation mechanism is initialized following the instructions above, you can switch to the root build directory of ICON and run the [make_runscripts](./make_runscripts) script: ```console $ cd /path/to/icon-builddir $ ./make_runscripts ``` > **_NOTE:_** By default, the generator creates runscripts for multiple experiments. The users can also chose to generate a single runscript for the experiment of their choice, for example: >```console >$ ./make_runscripts -s atm_amip >``` The generated runscripts are saved to the `./run` subdirectory of the build directory. The headers of the runscripts containing arguments for the HPC workload manager, e.g. [SLURM](https://slurm.schedmd.com/), might require additional manual adjustments regarding CPU time accounting, node allocation, etc. > **_NOTE:_** Alternatively, the users can employ the [low-level tool](./run/make_target_runscript) for runscript generation offering a more fine-grained control over certain parameters. For example, the wall clock limit and the number of allocated node can be injected into the runscript as follows: >```console >$ cd ./run && ln -sf ./checksuite.ocean_internal/omip/exp.ocean_omip_long exp.ocean_omip_long >$ ./make_target_runscript in_script=exp.ocean_omip_long in_script=exec.iconrun \ > EXPNAME=ocean_omip_long cpu_time=08:00:00 no_of_nodes=20 >``` Once created and adjusted, the runscript can be submitted for execution. For example: ```console $ cd ./run && sbatch ./exp.atm_amip.run ``` ## Out-of-build runs The configuration and building of ICON require significant amount of time, which heavily depends on the performance of the file system. Therefore, the users are recommended to consider a scenario in which they build the model on a fast local disk partition of a *login* node of an HPC cluster and then transfer the executable and all data required for their experiment to a slow shared partition available on *compute* nodes. This way, they can effectively reduce the time-to-solution. The transfer of the data can be done with the help of the [move_to_prefix.sh](./utils/move_to_prefix.sh) script as follows: 1. Create a copy of the ICON source directory on the fast partition, e.g. by cloning the git repository: ```console $ git clone --recursive git@gitlab.dkrz.de:icon/icon.git /path/to/fast/partition/icon-srcdir ``` 2. Create a build directory on the fast partition, switch to it, and run the [configure](./configure) script or a [configuration wrappers](#configuration-wrappers) with an *additional* argument `--prefix` pointing to the shared (but slow) partition: ```console $ mkdir /path/to/fast/partition/icon-builddir $ cd /path/to/fast/partition/icon-builddir $ /path/to/fast/partition/icon-srcdir/configure --prefix=/path/to/shared/partition/icon-rundir ``` 3. Build the model: ```console $ make -j8 ``` 4. Transfer the executable together with required data to the shared partition: ```console $ /path/to/fast/partition/icon-srcdir/utils/move_to_prefix.sh ``` 5. Switch to the shared partition and generate the runscripts: ```console $ cd /path/to/shared/partition/icon-rundir $ ./make_runscripts ``` 6. Switch to the `./run` subdirectory and submit the runscript of your choice, for example: ```console $ cd ./run && sbatch ./exp.atm_amip.run ``` # FAQ 1. **I run the configure script without any arguments and it fails. What should I do?** First, you are recommended to check whether there is a suitable [configuration wrapper](configuration-wrappers) in the [./config](./config) directory that you could use instead of running the configure script directly. If that is not the case, you need at least to specify the `LIBS` variable telling the configure script which libraries to link the executables to. The content of the list depends on the configure options you specify (see [Table 1](#icon-deptable)), for example: ```console $ ./configure --disable-mpi --disable-coupling LIBS='-lnetcdff -lnetcdf -llapack -lblas' ``` If the libraries reside in nonstandard directories, you might also need to specify the `FCFLAGS`, `CPPFLAGS`, and `LDFLAGS` variables to tell the script which directories need to be searched for header and library files (see section [Configuration](#configuration) for more details). 2. **How can I reproduce the configuration of the model used in a Buildbot test?** Scripts run by Buildbot for configuration and building of the model reside in the [./config/buildbot](./config/buildbot) directory. You can run them manually on the corresponding machine. 3. **I get an error message from the configure script starting with _"configure: error: unable to find sources of..."_. What does this mean?** Most probably, you forgot to initialize and/or update git submodules. You can do that by switching to the *source* root directory of ICON and running the following command: ```console $ git submodule update --init ``` 4. **I get an error from the runscript about a missing file _"ERROR : ...pool/data/ICON... does not exist."_. How do I resolve the issue?** Most probably, you are running ICON on an unknown (unsupported) system. In that case, the runscript expects input data in your home directory `$HOME/pool/data/ICON`. You need to request the required input files from ICON developers and put them to the aformentioned folder. You can also save the data to a different directory and override the default path by setting the environment variable `icon_data_rootFolder` before generating the runscripts, for example: ```console $ export icon_data_rootFolder='/path/to/ICON/data' $ ./make_runscripts -s atm_amip_test ```