...

$ cd WRF_practice_kit
$ ls -R
.:
anova.f90        namelist.input-TIMING        runqr_wrfacr_large.csh-004qgV2.dat      VALIDATION            wrfinput_d01validate.csh
f2p.py        namelist.input-VALIDATE  timing.csh           qr_acr_qsV2.dat                  VALIDATION
freezeH2O.dat            RUN.slurm                        wrfbdy_d01
Makefile   README.txt               validaterun_wrf_large.csh-004            wrf_bench_checker.py
namelist.input-TIMING    RUN_wrf_practice_avx2_oob.slurm  wrfinput_d01
namelist.input-VALIDATE  timing.csh

./VALIDATION:
wrfout_d01_2019-05-05_22:00:00  wrfout_d01_2019-05-05_22:00:36
wrfout_d01_2019-05-05_22:00:18  wrfout_d01_2019-05-05_22:00:54

...

• run_wrf_large.csh-004 is a sample PBS script showing how WRF is expected to be run; it runs WRF twice: once for timing and once for validation. A simple version to possibly start with on niagara is RUN.slurm. A script with great examples for how to use the number of OpenMP threads and the number of MPI processes in computations is given with RUN_wrf_practice_avx2_oob.slurm.

• The data files required by WRF are namelist.input, wrfinput_d01 and wrfbdy_d01; the run script copies namelist.input-VALIDATE and namelist.input-TIMING in turn to namelist.input before running WRF. The three files with the .dat suffix are also read by WRF; they are not strictly necessary because they would be computed if not present, but we provide them to avoid the extra computation.

• WRF also requires a set of fixed, auxiliary data files that are found in the run directory of the main source directory, WRF_ISC21. You may copy those files to WRF_practice_kit or make symbolic links to them. Let's assume WRF_ISC21 at the same level as the directory WRF_practice_kit, and that you are currently “in” the WRF_practice_kit directory. Then, the following will create the symbolic links sufficient for you to run the WRF simulation, including a symbolic link to the WRF executable:

...

Part of the optimization is determining the number and aspect ratio of the MPI processes, and determining the optimal mix of MPI and OpenMP processes. In the namelist.input file, in the &domains namelist record, the default values for these settings are given here:

...

We want to ask a simple question that is hard to answer “is the exemplar data and your data the same”? The ANOVA method is a statistical technique that is widely utilized in industrial manufacturing to determine if populations are the same, but of course it is couched in that statistical jargon of “do we accept the null hypothesis”. There are a number of issues to consider, which are referred to as factors in ANOVA testing. We want to account for all possible factors, and then see which factors in a process are important when considering reproducibility and overall quality. In our usage, we a priori know that a number of factors would produce different results (when comparing the exemplar vs. your data).

For example, we know the following are important factors that cause a difference in the simulation:

• the time of the forecast: We use several time periods in the validation, the initial time of the forecast is, of course, different than time period four.

• the geophysical location of validation: We use 13 small boxes in the domain, as we expect the simulation to have different answers in different locations.

• which vertical level: We use one at the surface, one a km above ground, and one level towards the top of the model’s lid. Again the weather will be different at the ground level vs. where jets fly).

• the variable of interest: We separate wind, temperature, and moisture, as we would expect the temperature to behave differently than the wind.

• exemplar vs. your data: The important remaining factor is your data vs. our data - – and we do not want that factor to be a large source of difference. That (hopefully unconfounded) difference is what we specifically test.

There are many other factor that would impact the model results in meaningful ways, but we are not allowing you to exercise those options (such as different physical parameterization schemes, or different numerical schemes). However, it is likely that an optimized library for intrinsic functions would be entirely OK, as would using real*8 vs real*4 floats (that would be slower, though), enhanced optimization is OK, using a different compiler or chip are OK, etc. The ANOVA test, as long as we stay within the first few time steps is quite adept at identifying acceptable data and rejecting incorrect data.

The “VALIDATION” VALIDATION directory is the exemplar directory that is filled with data that we provide. The “VALIDATE” VALIDATE directory is populated by you with each WRF simulation (by both the run_wrf_large.csh-004 and RUN.slurm example scripts). Again, sorry for the too-similar naming convention!

...

./timing.csh TIMING/rsl.out.0000
MPI ranks used: 8 x 10 = 80
Timing for radiation step: 41.81620 s
Average Time for non-radiation:  5.64087 s

HAPPY COMPUTING!

Once you are comfortable running through these steps to build and run the code, and can generate the output for timing and validation, you may move onto the 3-domain case. Good luck!

...