GEOS-5

Regression tests: Difference between revisions

← Older edit
Pchakrab (talk | contribs)
1,048 edits
Pchakrab (talk | contribs)
1,048 edits
 
(21 intermediate revisions by the same user not shown)
Line 1: Line 1:
__FORCETOC__
{{rightTOC}}
'''For questions contact Purnendu at purnendu.chakraborty@nasa.gov'''
'''For questions contact Purnendu at purnendu.chakraborty@nasa.gov'''


Line 5: Line 5:
* checkout BASELINE version, run '1 day' and '1 step' experiments (one time operation)
* checkout BASELINE version, run '1 day' and '1 step' experiments (one time operation)
* checkout CURRENT  version, run '1 day' and '1 step' experiments (daily operation)
* checkout CURRENT  version, run '1 day' and '1 step' experiments (daily operation)
* compare outputs of BASELINE and CURRENT verions. Compare the restart and NetCDF-4 files, (TOTAL) times taken by each GridComp and memory usages as reported by MAPL timers.
* compare outputs of BASELINE and CURRENT verions of '1 day' runs - compare restart and NetCDF-4 files, (TOTAL) times taken by each GridComp and memory usages as reported by MAPL timers. If the restarts produce non-zero diff report the minimum and maximum of differences in fvcore_internal_rst for '1 step' run.


The idea here is to checkout a "correct" version of the code, run an experiment and consider the resulting output to be the BASELINE version. Next, checkout the UNSTABLE tag, build it, run the experiment and compare the resulting
The idea here is to checkout a "correct" version of the code, run an experiment and consider the resulting output to be the BASELINE version. Next, checkout the UNSTABLE tag, build it, run the experiment and compare the resulting
output (CURRENT version) with the BASELINE version.
output (CURRENT version) with the BASELINE version.
Once the BASELINE version has been run, running the CURRENT version and
comparing outputs can be combined in a script and run nightly.


==Test scripts==
==Test scripts==
Line 107: Line 104:
Next, an '1 step' experiment is run using the '''same build'''. Since this run is only for comparing outputs in case of non-zero diff, we do not time the run.
Next, an '1 step' experiment is run using the '''same build'''. Since this run is only for comparing outputs in case of non-zero diff, we do not time the run.


$./GCM_Run.py --exp_typ AGCM --tag Ganymed-1_0_p1 --mod Ganymed  
$./GCM_Run.py --exp_typ AGCM --tag Ganymed-1_0_p1 --mod Ganymed  
               --home_dir HOME_DIR
               --home_dir HOME_DIR
               --exstng_bld HOME_DIR/bld
               --exstng_bld HOME_DIR/bld
Line 118: Line 115:
====One day & One step====
====One day & One step====


Repeat the BASELINE step. Checkout a copy of the UNSTABLE tag (Ganymed-2_0_UNSTABLE at the time of writing), build it, setup a 1day/1step experiment with the same options as for the BASELINE step and run the experiments.
Repeat the [[#BASELINE run (one time operation)]]. Checkout a copy of the UNSTABLE tag (Ganymed-2_0_UNSTABLE at the time of writing), build it, setup a 1day/1step experiment with the same options as for the BASELINE step and run the experiments.
 
NOTE: Add flag --lt_test (1day run) for layout/transparency tests.


==Compare BASELINE and CURRENT runs==
==Compare BASELINE and CURRENT runs==
Line 133: Line 132:
   --h5diff=H5DIFF  [RQD] location of h5diff utility
   --h5diff=H5DIFF  [RQD] location of h5diff utility
   --diff=DIFF      [OPT] location of diff utility (default=/usr/bin/diff)
   --diff=DIFF      [OPT] location of diff utility (default=/usr/bin/diff)
So, to compare the above two runs, one would issue the command
$ ./GCM_Cmp.py --bas /path/to/baseline/home/dir
                --cur /path/to/current/home/dir
                --h5diff /path/to/h5diff
This script compares and prints out the following to stdout and /path/to/current/home/dir/log.cmp_out
* Diffs *_rst files (BASELINE and CURRENT) and prints out the result. In case of non-zero diff finds min and max of differences between BASELINE and CURRENT fvcore_internal_rst
* Diffs *.nc4 files (BASELINE and CURRENT) and prints out the result.
* Counts the number of layout/transparency successes (not for AOGCM)
==Automated testing==
Once the BASELINE version has been created, running the CURRENT version and comparing outputs can be combined in a script and run nightly. The following csh script assumes the BASELINE runs in /discover/swdev/pchakrab/SystemTests/AGCM/b_FV3_coupledNO_O1_gocartNO/BASELINE
#!/usr/bin/env csh
# date and time
# -------------
set DATE = `date +%F`
set TIME = `date +%T`
# all regression tests are under this directory
# ---------------------------------------------
set TST_HOM = /discover/swdev/pchakrab/SystemTests
# run and compare scripts
# -----------------------
set SCRPT_DIR = /discover/swdev/pchakrab/sandbox/systest
set GCM_RUN = $SCRPT_DIR/GCM_Run.py
set GCM_CMP = $SCRPT_DIR/GCM_Cmp.py
# location of h5diff
# ------------------
set BASEDIR = /discover/swdev/mathomp4/Baselibs/GMAO-Baselibs-3_2_0-Intel_11.0.083-MVAPICH2_1.8.1
set H5DIFF  = $BASEDIR/Linux/bin/h5diff
# CVS tag and module
# ------------------
set CVS_TAG = Ganymed-2_0_UNSTABLE
set CVS_MOD = Ganymed
# directory containing restarts
# -----------------------------
set RST_HOM = /discover/nobackup/mathomp4/Restarts-G20
# logging all output to
# ---------------------
set LOG_FILE = $TST_HOM/log/log_CS_${DATE}_${TIME}
echo "" >>& $LOG_FILE
echo " |---------------------------------------|" >>& $LOG_FILE
echo " | CURRENT: Cubed-sphere test - 2 degree |" >>& $LOG_FILE
echo " |---------------------------------------|" >>& $LOG_FILE
echo "" >>& $LOG_FILE
echo " Cubed-sphere test - 2 degree"
echo " ----------------------------"
echo ""
# input file for gcm_setup
# ------------------------
set INPUT_GCM_SETUP = $SCRPT_DIR/input/gcm_setup_AGCM_b_FV3_NO_O1_NO.input
# restart directory
# -----------------
set RST_DIR = $RST_HOM/c48
# baseline and current 'home' directories,
# if CUR_HOM exists, rename it as PREVIOUS
# ----------------------------------------
set BAS_HOM = $TST_HOM/AGCM/b_FV3_coupledNO_O1_gocartNO/BASELINE
set CUR_HOM = $TST_HOM/AGCM/b_FV3_coupledNO_O1_gocartNO/CURRENT
set PRV_HOM = $TST_HOM/AGCM/b_FV3_coupledNO_O1_gocartNO/PREVIOUS
if (-d $CUR_HOM) then
    echo -n " Renaming CURRENT as PREVIOUS..." |& tee --append $LOG_FILE
    if (-d $PRV_HOM) then
        \rm -rf $PRV_HOM
    endif
    mv $CUR_HOM $PRV_HOM
    echo "done." |& tee --append $LOG_FILE
    echo "" >>& $LOG_FILE
endif
# build, setup and run 1 day experiment (CURRENT)
# also run layout/transparency test
# -----------------------------------------------
echo " 1 day run" >>& $LOG_FILE
echo " ---------" >>& $LOG_FILE
echo -n " 1 day run..."
$GCM_RUN --exp_typ AGCM --tag $CVS_TAG --mod $CVS_MOD \
          --home_dir $CUR_HOM --gcm_opts $INPUT_GCM_SETUP  \
          --rst_dir $RST_DIR --how_long 1day --pbld --timer \
          --memutils --lt_test >>& $LOG_FILE
echo "done."
# once the model is built in $CUR_HOM/bld, we will re-use this build
# ------------------------------------------------------------------
set EXSTNG_BLD = $CUR_HOM/bld
# build, setup and run 1 step experiment (CURRENT)
# ------------------------------------------------
echo "" >>& $LOG_FILE
echo " 1 step run" >>& $LOG_FILE
echo " ----------" >>& $LOG_FILE
echo -n " 1 step run..."
$GCM_RUN --exp_typ AGCM --tag $CVS_TAG --mod $CVS_MOD \
          --exstng_bld $EXSTNG_BLD --home_dir $CUR_HOM \
          --gcm_opts $INPUT_GCM_SETUP --rst_dir $RST_DIR \
          --how_long 1step >>& $LOG_FILE
echo "done."
# compare BASELINE and CURRENT runs
# ---------------------------------
echo "" >>& $LOG_FILE
echo " Comparing BASELINE and CURRENT runs" >>& $LOG_FILE
echo " -----------------------------------" >>& $LOG_FILE
echo -n " Comparing BASELINE and CURRENT runs..."
$GCM_CMP --bas $BAS_HOM --cur $CUR_HOM --h5diff $H5DIFF >>& $LOG_FILE
echo "" >>& $LOG_FILE
echo "All done." >>& $LOG_FILE
echo "done."
# print the result of  'Tests passed?'
# ------------------------------------
cat $LOG_FILE | grep 'Tests passed?'
==Non-zero diff==
In case of non-zero diff between the BASELINE and CURRENT versions and if the difference is explainable (e.g. there has been a science change), step 1 needs to be repeated - in BASELINE HOME_DIR delete directories bld, run and log (do '''NOT''' delete the file 'info', that way a log of BASELINE checkouts/runs is maintained) and repeat [[#BASELINE run (one time operation)]].
==Other utilities in systest==
Cmp_PBSoutputs.py: Compares PBS output files (what GEOS-5 writes to stdout) from two different runs. Syntax:
$./Cmp_PBSoutputs.py --bas /path/to/file1 --cur /path/to/file2
Cmp_Restarts.py: Compares two Experiment directories (compares cap_restart and *_rst). Syntax:
$ ./Cmp_Restarts.py --bas /path/to/base/run/dir --cur /path/to/cur/run/dir
Cmp_NC4s.py: Compares NetCDF-4 files from two experiment directories. Syntax:
$ ./Cmp_NC4s.py --bas /path/to/base/run/dir --cur /path/to/cur/run/dir
[[Category:SI Team]]
Retrieved from "https://geos5.org/wiki/index.php?title=Regression_tests"