Regression tests

For questions contact Purnendu at purnendu.chakraborty@nasa.gov

There are essentially 3 steps in system testing of GEOS-5:

• checkout BASELINE version, run '1 day' and '1 step' experiments (one time operation)
• checkout CURRENT version, run '1 day' and '1 step' experiments (daily operation)
• compare outputs of BASELINE and CURRENT verions. Compare the restart and NetCDF-4 files, (TOTAL) times taken by each GridComp and memory usages as reported by MAPL timers.

The idea here is to checkout a "correct" version of the code, run an experiment and consider the resulting output to be the BASELINE version. Next, checkout the UNSTABLE tag, build it, run the experiment and compare the resulting output (CURRENT version) with the BASELINE version.

Once the BASELINE version has been run, running the CURRENT version and comparing outputs can be combined in a script and run nightly.

Test scripts

$ cvs co systest
$ cd systest

Running GCM_Run.py without any (command line) options provided prints out the usage information:

[discover15:systest]$ ./GCM_Run.py 
Usage: GCM_Run.py [options]

Options:
  -h, --help            show this help message and exit
  --exp_typ=EXP_TYP     [RQD] experiment type (AGCM/AOGCM)
  --tag=CVSTAG          [RQD] cvs tag
  --mod=CVSMOD          [RQD] cvs module
  --home_dir=HOMDIR     [RQD] Where we create bld/, run/, log/ (full path)
  --gcm_opts=GCMOPTS    [RQD] input file for gcm_setup (full path)
  --rst_dir=RSTDIR      [RQD] dir containing restart files (full path)
  --how_long=HOWLONG    [RQD] options: 1day/1step
  --exstng_bld=EXSTNGBLD
                        [OPT] existing build dir (full path)
  --spcl_tag=SPCLTAG    [OPT] special tag to update to
  --pbld                [OPT] parallel build (default = False)
  --timer               [OPT] turn MAPL timers ON (default = OFF)
  --memutils            [OPT] turn MAPL memutils ON (dafault = OFF)
  --lt_test             [OPT] layout/transparency test (default = False)

Input file

The script GCM_Run.py requires an input file passed to it through the flag '--gcm_opts'. This is a template file specifying the input options to gcm_setup. Two example input files are provided

input/gcm_setup_AGCM_b_FVqq_NO_O1_NO.input
input/gcm_setup_AGCM_b_FV3_NO_O1_NO.input

The contents of gcm_setup_AGCM_b_FV_NO_O1_NO.input are:

Experiment ID                     = TO_BE_IGNORED
Experiment Description            = TO_BE_IGNORED
Atmospheric Horizontal Resolution = b
Dynamical Core                    = FV
COUPLED                           = NO
Data_ocean Horizontal Resolution  = o1
GOCART                            = NO
HISTORY template file             = HISTORY.AGCM.rc.tmpl
Home Dir                          = TO_BE_IGNORED
Experiment Dir                    = TO_BE_IGNORED
Build Dir                         = TO_BE_IGNORED
GROUP ID                          = g0620

The value TO_BE_IGNORED is (as the name suggests) ignored. The values for these fields are generated by the scripts depending on the experiment duration. For a '1 step' run, Experiment ID is set to '1step' and Experiment Description is set to '1step_run'. The location of Home/Experiment/Build directories are discussed below.

The remaining fields require a valid value to be specified.

• 'Atmospheric Horizontal Resolution' has to be one of b, c, d and e.
• 'Dynamical Core' is one of FV, FV3
• 'COUPLED' is NO for AGCM runs.
• 'Data_ocean Horizontal Resolution' is one of o1, o8.
• 'GOCART' is NO (GOCART is not currently supported).
• 'GROUP ID' is the user's group id.

BASELINE run (one time operation)

One day

At the time of writing, Ganymed-1_0_p1 was considered the BASELINE model (AGCM) against which we test the UNSTABLE model. We checkout a copy of Ganymed-1_0_p1 from the CVS repository, build it, setup a '1 day' experiment with options b, FV, COUPLED=NO, O1, GOCART=NO, and run the experiment through the following command:

$./GCM_Run.py --exp_typ AGCM --tag Ganymed-1_0_p1 --mod Ganymed 
              --home_dir /path/to/home/dir
              --gcm_opts input/gcm_setup_AGCM_b_FV_NO_O1_NO.input
              --rst_dir /path/to/dir/containing/relevant/restarts
              --how_long 1day --pbld --timer --memutils

In the following, HOME_DIR = /path/to/home/dir (typically under $NOBACKUP)

The script does the following:

• Creates the 3 dirs bld, run and log under HOME_DIR
• Writes the file HOME_DIR/info containing pertinent information
• Checks out the model tag in HOME_DIR/bld directory
• Sets environment (by source'ing HOME_DIR/bld/GEOSagcm/src/g5_modules)
• Submits a parallel build job, waits for it to complete. If parallel build fails, a serial build is attempted
• Creates input file for gcm_setup with Home and Experiment directories set to HOME_DIR/run/1day (note that the flag '--how_long' has been set to '1day') and Build dir set to HOME_DIR/bld
• Sets up experiment:
  • runs gcm_setup with options from the input file just created
  • copies restarts (cap_restart and *_rst from RSTDIR provided through flag '--rst_dir')
  • edits CAP.rc to ensure a '1 day' run and turns on MAPL timers and MAPL memutils
  • edits gcm_run.j so that multiple jobs are not submitted.
• Submits the job from HOME_DIR/run/1day, waits for job to complete, checks if EGRESS exists and copies the PBS output to LOG_DIR = HOME_DIR/log/1day.

Output is written to stdout and the file HOME_DIR/log/1day/log.systest. Also the outputs from CVS checkout, parallel build submission etc. are written to files in this log directory.

One step

Next, an '1 step' experiment is run using the same build. Since this run is only for comparing outputs in case of non-zero diff, we do not time the run.

$./GCM_Run.py --exp_typ AGCM --tag Ganymed-1_0_p1 --mod Ganymed

             --home_dir HOME_DIR
             --exstng_bld HOME_DIR/bld
             --gcm_opts input/gcm_setup_AGCM_b_FV_NO_O1_NO.input
             --rst_dir /path/to/dir/containing/relevant/restarts
             --how_long 1step

CURRENT run (daily operation)

One day & One step

Repeat the BASELINE step. Checkout a copy of the UNSTABLE tag (Ganymed-2_0_UNSTABLE at the time of writing), build it, setup a 1day/1step experiment with the same options as for the BASELINE step and run the experiments.

Compare BASELINE and CURRENT runs

The script to compare 2 runs is GCM_Cmp.py

[discover15:systest]$ ./GCM_Cmp.py 
Usage: GCM_Cmp.py [options]

Options:
  -h, --help       show this help message and exit
  --base=BASE      [RQD] base home dir
  --cur=CUR        [RQD] current home dir
  --h5diff=H5DIFF  [RQD] location of h5diff utility
  --diff=DIFF      [OPT] location of diff utility (default=/usr/bin/diff)

So, to compare the above two runs, one would issue the command

$ ./GCM_Cmp.py --bas /path/to/baseline/home/dir
               --cur /path/to/current/home/dir 
               --h5diff /path/to/h5diff

This script compares and prints out the following to stdout and /path/to/current/home/dir/log.cmp_out

• Diffs *_rst files (BASELINE and CURRENT) and prints out the result. In case of non-zero diff finds min and max of differences between BASELINE and CURRENT fvcore_internal_rst
• Diffs *.nc4 files (BASELINE and CURRENT) and prints out the result.
• Counts the number of layout/transparency successes (not for AOGCM)

Non-zero diff

In case of non-zero diff between the BASELINE and CURRENT versions and if the difference is explainable (e.g. there has been a science change), step 1 needs to be repeated - in BASELINE HOME_DIR delete directories bld, run and log (do NOT delete the file 'info', that way a log of BASELINE checkout and runs are maintained) and repeat step 1.

Other utilities in systest

1. Cmp_PBSoutputs.py: Compares PBS output files (what GEOS-5 writes to stdout) from two different runs. Syntax:

$./Cmp_PBSoutputs.py --bas /path/to/file1 --cur /path/to/file2

1. Cmp_Restarts.py: Compares two Experiment directories (compares cap_restart and *_rst). Syntax:

$ ./Cmp_Restarts.py --bas /path/to/base/run/dir --cur /path/to/cur/run/dir

1. Cmp_NC4s.py: Compares NetCDF-4 files from two experiment directories. Syntax:

$ ./Cmp_NC4s.py --bas /path/to/base/run/dir --cur /path/to/cur/run/dir