Regression tests

For questions contact Purnendu at purnendu.chakraborty@nasa.gov

There are essentially 3 steps in system testing of GEOS-5:

• checkout BASELINE version, run '1 day' and '1 step' experiments (one time operation)
• checkout CURRENT version, run '1 day' and '1 step' experiments (daily operation)
• compare outputs of BASELINE and CURRENT verions of '1 day' runs - compare restart and NetCDF-4 files, (TOTAL) times taken by each GridComp and memory usages as reported by MAPL timers. If the restarts produce non-zero diff report the minimum and maximum of differences in fvcore_internal_rst for '1 step' run.

The idea here is to checkout a "correct" version of the code, run an experiment and consider the resulting output to be the BASELINE version. Next, checkout the UNSTABLE tag, build it, run the experiment and compare the resulting output (CURRENT version) with the BASELINE version.

Test scripts

$ cvs co systest
$ cd systest

Running GCM_Run.py without any (command line) options provided prints out the usage information:

[discover15:systest]$ ./GCM_Run.py 
Usage: GCM_Run.py [options]

Options:
  -h, --help            show this help message and exit
  --exp_typ=EXP_TYP     [RQD] experiment type (AGCM/AOGCM)
  --tag=CVSTAG          [RQD] cvs tag
  --mod=CVSMOD          [RQD] cvs module
  --home_dir=HOMDIR     [RQD] Where we create bld/, run/, log/ (full path)
  --gcm_opts=GCMOPTS    [RQD] input file for gcm_setup (full path)
  --rst_dir=RSTDIR      [RQD] dir containing restart files (full path)
  --how_long=HOWLONG    [RQD] options: 1day/1step
  --exstng_bld=EXSTNGBLD
                        [OPT] existing build dir (full path)
  --spcl_tag=SPCLTAG    [OPT] special tag to update to
  --pbld                [OPT] parallel build (default = False)
  --timer               [OPT] turn MAPL timers ON (default = OFF)
  --memutils            [OPT] turn MAPL memutils ON (dafault = OFF)
  --lt_test             [OPT] layout/transparency test (default = False)

Input file

The script GCM_Run.py requires an input file passed to it through the flag '--gcm_opts'. This is a template file specifying the input options to gcm_setup. Two example input files are provided

input/gcm_setup_AGCM_b_FVqq_NO_O1_NO.input
input/gcm_setup_AGCM_b_FV3_NO_O1_NO.input

The contents of gcm_setup_AGCM_b_FV_NO_O1_NO.input are:

Experiment ID                     = TO_BE_IGNORED
Experiment Description            = TO_BE_IGNORED
Atmospheric Horizontal Resolution = b
Dynamical Core                    = FV
COUPLED                           = NO
Data_ocean Horizontal Resolution  = o1
GOCART                            = NO
HISTORY template file             = HISTORY.AGCM.rc.tmpl
Home Dir                          = TO_BE_IGNORED
Experiment Dir                    = TO_BE_IGNORED
Build Dir                         = TO_BE_IGNORED
GROUP ID                          = g0620

The value TO_BE_IGNORED is (as the name suggests) ignored. The values for these fields are generated by the scripts depending on the experiment duration. For a '1 step' run, Experiment ID is set to '1step' and Experiment Description is set to '1step_run'. The location of Home/Experiment/Build directories are discussed below.

The remaining fields require a valid value to be specified.

• 'Atmospheric Horizontal Resolution' has to be one of b, c, d and e.
• 'Dynamical Core' is one of FV, FV3
• 'COUPLED' is NO for AGCM runs.
• 'Data_ocean Horizontal Resolution' is one of o1, o8.
• 'GOCART' is NO (GOCART is not currently supported).
• 'GROUP ID' is the user's group id.

BASELINE run (one time operation)

One day

At the time of writing, Ganymed-1_0_p1 was considered the BASELINE model (AGCM) against which we test the UNSTABLE model. We checkout a copy of Ganymed-1_0_p1 from the CVS repository, build it, setup a '1 day' experiment with options b, FV, COUPLED=NO, O1, GOCART=NO, and run the experiment through the following command:

$./GCM_Run.py --exp_typ AGCM --tag Ganymed-1_0_p1 --mod Ganymed 
              --home_dir /path/to/home/dir
              --gcm_opts input/gcm_setup_AGCM_b_FV_NO_O1_NO.input
              --rst_dir /path/to/dir/containing/relevant/restarts
              --how_long 1day --pbld --timer --memutils

In the following, HOME_DIR = /path/to/home/dir (typically under $NOBACKUP)

The script does the following:

• Creates the 3 dirs bld, run and log under HOME_DIR
• Writes the file HOME_DIR/info containing pertinent information
• Checks out the model tag in HOME_DIR/bld directory
• Sets environment (by source'ing HOME_DIR/bld/GEOSagcm/src/g5_modules)
• Submits a parallel build job, waits for it to complete. If parallel build fails, a serial build is attempted
• Creates input file for gcm_setup with Home and Experiment directories set to HOME_DIR/run/1day (note that the flag '--how_long' has been set to '1day') and Build dir set to HOME_DIR/bld
• Sets up experiment:
  • runs gcm_setup with options from the input file just created
  • copies restarts (cap_restart and *_rst from RSTDIR provided through flag '--rst_dir')
  • edits CAP.rc to ensure a '1 day' run and turns on MAPL timers and MAPL memutils
  • edits gcm_run.j so that multiple jobs are not submitted.
• Submits the job from HOME_DIR/run/1day, waits for job to complete, checks if EGRESS exists and copies the PBS output to LOG_DIR = HOME_DIR/log/1day.

Output is written to stdout and the file HOME_DIR/log/1day/log.systest. Also the outputs from CVS checkout, parallel build submission etc. are written to files in this log directory.

One step

Next, an '1 step' experiment is run using the same build. Since this run is only for comparing outputs in case of non-zero diff, we do not time the run.

$./GCM_Run.py --exp_typ AGCM --tag Ganymed-1_0_p1 --mod Ganymed 
             --home_dir HOME_DIR
             --exstng_bld HOME_DIR/bld
             --gcm_opts input/gcm_setup_AGCM_b_FV_NO_O1_NO.input
             --rst_dir /path/to/dir/containing/relevant/restarts
             --how_long 1step

CURRENT run (daily operation)

One day & One step

Repeat the #BASELINE run (one time operation). Checkout a copy of the UNSTABLE tag (Ganymed-2_0_UNSTABLE at the time of writing), build it, setup a 1day/1step experiment with the same options as for the BASELINE step and run the experiments.

NOTE: Add flag --lt_test (1day run) for layout/transparency tests.

Compare BASELINE and CURRENT runs

The script to compare 2 runs is GCM_Cmp.py

[discover15:systest]$ ./GCM_Cmp.py 
Usage: GCM_Cmp.py [options]

Options:
  -h, --help       show this help message and exit
  --base=BASE      [RQD] base home dir
  --cur=CUR        [RQD] current home dir
  --h5diff=H5DIFF  [RQD] location of h5diff utility
  --diff=DIFF      [OPT] location of diff utility (default=/usr/bin/diff)

So, to compare the above two runs, one would issue the command

$ ./GCM_Cmp.py --bas /path/to/baseline/home/dir
               --cur /path/to/current/home/dir 
               --h5diff /path/to/h5diff

This script compares and prints out the following to stdout and /path/to/current/home/dir/log.cmp_out

• Diffs *_rst files (BASELINE and CURRENT) and prints out the result. In case of non-zero diff finds min and max of differences between BASELINE and CURRENT fvcore_internal_rst
• Diffs *.nc4 files (BASELINE and CURRENT) and prints out the result.
• Counts the number of layout/transparency successes (not for AOGCM)

Automated testing

Once the BASELINE version has been created, running the CURRENT version and comparing outputs can be combined in a script and run nightly. The following csh script assumes the BASELINE runs in /discover/swdev/pchakrab/SystemTests/AGCM/b_FV3_coupledNO_O1_gocartNO/BASELINE

#!/usr/bin/env csh

# date and time
# -------------
set DATE = `date +%F`
set TIME = `date +%T`


# all regression tests are under this directory
# ---------------------------------------------
set TST_HOM = /discover/swdev/pchakrab/SystemTests


# run and compare scripts
# -----------------------
set SCRPT_DIR = /discover/swdev/pchakrab/sandbox/systest
set GCM_RUN = $SCRPT_DIR/GCM_Run.py
set GCM_CMP = $SCRPT_DIR/GCM_Cmp.py


# location of h5diff
# ------------------
set BASEDIR = /discover/swdev/mathomp4/Baselibs/GMAO-Baselibs-3_2_0-Intel_11.0.083-MVAPICH2_1.8.1
set H5DIFF  = $BASEDIR/Linux/bin/h5diff


# CVS tag and module
# ------------------
set CVS_TAG = Ganymed-2_0_UNSTABLE
set CVS_MOD = Ganymed


# directory containing restarts
# -----------------------------
set RST_HOM = /discover/nobackup/mathomp4/Restarts-G20


# logging all output to
# ---------------------
set LOG_FILE = $TST_HOM/log/log_CS_${DATE}_${TIME}


echo "" >>& $LOG_FILE
echo " |---------------------------------------|" >>& $LOG_FILE
echo " | CURRENT: Cubed-sphere test - 2 degree |" >>& $LOG_FILE
echo " |---------------------------------------|" >>& $LOG_FILE
echo "" >>& $LOG_FILE
echo " Cubed-sphere test - 2 degree"
echo " ----------------------------"
echo ""

# input file for gcm_setup
# ------------------------
set INPUT_GCM_SETUP = $SCRPT_DIR/input/gcm_setup_AGCM_b_FV3_NO_O1_NO.input


# restart directory
# -----------------
set RST_DIR = $RST_HOM/c48


# baseline and current 'home' directories,
# if CUR_HOM exists, rename it as PREVIOUS
# ----------------------------------------
set BAS_HOM = $TST_HOM/AGCM/b_FV3_coupledNO_O1_gocartNO/BASELINE
set CUR_HOM = $TST_HOM/AGCM/b_FV3_coupledNO_O1_gocartNO/CURRENT
set PRV_HOM = $TST_HOM/AGCM/b_FV3_coupledNO_O1_gocartNO/PREVIOUS
if (-d $CUR_HOM) then
    echo -n " Renaming CURRENT as PREVIOUS..." |& tee --append $LOG_FILE
    if (-d $PRV_HOM) then
        \rm -rf $PRV_HOM
    endif
    mv $CUR_HOM $PRV_HOM
    echo "done." |& tee --append $LOG_FILE
    echo "" >>& $LOG_FILE
endif


# build, setup and run 1 day experiment (CURRENT)
# also run layout/transparency test
# -----------------------------------------------
echo " 1 day run" >>& $LOG_FILE
echo " ---------" >>& $LOG_FILE
echo -n " 1 day run..."
$GCM_RUN --exp_typ AGCM --tag $CVS_TAG --mod $CVS_MOD \
         --home_dir $CUR_HOM --gcm_opts $INPUT_GCM_SETUP  \
         --rst_dir $RST_DIR --how_long 1day --pbld --timer \
         --memutils --lt_test >>& $LOG_FILE
echo "done."


# once the model is built in $CUR_HOM/bld, we will re-use this build
# ------------------------------------------------------------------
set EXSTNG_BLD = $CUR_HOM/bld


# build, setup and run 1 step experiment (CURRENT)
# ------------------------------------------------
echo "" >>& $LOG_FILE
echo " 1 step run" >>& $LOG_FILE
echo " ----------" >>& $LOG_FILE
echo -n " 1 step run..."
$GCM_RUN --exp_typ AGCM --tag $CVS_TAG --mod $CVS_MOD \
         --exstng_bld $EXSTNG_BLD --home_dir $CUR_HOM \
         --gcm_opts $INPUT_GCM_SETUP --rst_dir $RST_DIR \
         --how_long 1step >>& $LOG_FILE
echo "done."


# compare BASELINE and CURRENT runs
# ---------------------------------
echo "" >>& $LOG_FILE
echo " Comparing BASELINE and CURRENT runs" >>& $LOG_FILE
echo " -----------------------------------" >>& $LOG_FILE
echo -n " Comparing BASELINE and CURRENT runs..."
$GCM_CMP --bas $BAS_HOM --cur $CUR_HOM --h5diff $H5DIFF >>& $LOG_FILE
echo "" >>& $LOG_FILE
echo "All done." >>& $LOG_FILE
echo "done."

# print the result of  'Tests passed?'
# ------------------------------------
cat $LOG_FILE | grep 'Tests passed?'

Non-zero diff

In case of non-zero diff between the BASELINE and CURRENT versions and if the difference is explainable (e.g. there has been a science change), step 1 needs to be repeated - in BASELINE HOME_DIR delete directories bld, run and log (do NOT delete the file 'info', that way a log of BASELINE checkouts/runs is maintained) and repeat #BASELINE run (one time operation).

Other utilities in systest

Cmp_PBSoutputs.py: Compares PBS output files (what GEOS-5 writes to stdout) from two different runs. Syntax:

$./Cmp_PBSoutputs.py --bas /path/to/file1 --cur /path/to/file2

Cmp_Restarts.py: Compares two Experiment directories (compares cap_restart and *_rst). Syntax:

$ ./Cmp_Restarts.py --bas /path/to/base/run/dir --cur /path/to/cur/run/dir

Cmp_NC4s.py: Compares NetCDF-4 files from two experiment directories. Syntax:

$ ./Cmp_NC4s.py --bas /path/to/base/run/dir --cur /path/to/cur/run/dir