Running the GEOS-5 SBU Benchmark

Build and install the model

First, untar the model tarball (in $NOBACKUP!!!):

$ tar xf Heracles-UNSTABLE-MPT-Benchmark.2017Feb13.tar.gz

Next, set up ESMADIR:

$ setenv ESMADIR <directory-to>/Heracles-UNSTABLE-MPT-Benchmark/GEOSagcm

it is just below the src/ directory.

Go into the src/ directory of your model. Following above:

$ cd $ESMADIR/src

Setup the environment by sourcing the g5_modules file:

$ source g5_modules

To build the model, you have one of two choices. First, you can use the parallel_build.csh script to submit a PBS job that compiles the model:

$ ./parallel_build.csh

or you can interactively build the model using:

$ make install

To capture the install log, we recommend tee'ing the output to a file:

$ make install |& tee make.install.log (on tcsh)
$ make install 2>&1 | tee make.install.log (on bash)

Note you can also build in parallel interactively with:

$ make -j8 pinstall |& tee make.pinstall.log (on tcsh)

where N is the number of parallel processes. From testing, 8 jobs is about as much as is useful. You can use more, but no benefit will accrue.

By default, the Intel Fortran compiler (ifort) is used for the build process. For other compilers, contact matthew.thompson@nasa.gov for instructions to use GCC or PGI compilers.

Monitor build process

The build can be monitored using the utility gmh.pl in the directory Config. From the src directory:

$ Config/gmh.pl -Av make.install.log

outputs the build status as

                          --------
                          Packages
                          --------

         >>>> Fatal Error           .... Ignored Error

 [ok]      Config
 [ok]      GMAO_Shared
 [ok]      |    GMAO_mpeu
 [ok]      |    |    mpi0
 [ok]      |    GMAO_pilgrim
 [ok]      |    GMAO_gfio
 [ok]      |    |    r4
 [ok]      |    |    r8
 [ok]      |    GMAO_perllib
 [ok]      |    MAPL_cfio
 [ok]      |    |    r4
 [ok]      |    |    r8
 [ok]      |    MAPL_Base
 [ok]      |    |    TeX
 [ok]      |    GEOS_Shared
 [ 1] .... .... Chem_Base
 [ok]      |    Chem_Shared
 [ok]      |    GMAO_etc
 [ok]      |    GMAO_hermes
 [ 2] .... .... GFDL_fms
 [ok]      |    GEOS_Util
 [ok]      |    |    post 

                          -------
                          Summary
                          -------

IGNORED mpp_comm_sma.d mpp_transmit_sma.d Chem_AodMod.d (3 files in 2 packages)
All 22 packages compiled successfully.

In case of errors, gmh summarizes exactly where it happens by indicating the package where it occured. Caveat: it does not work in parallel (output is scrambled). So, if the parallel build fails, rerun it sequentially (it will go quickly and die in the same place) and run gmh on the output for a summary.

Run the model

Setting up an experiment

Go into the model application directory and do a couple of preliminary commands:

$ cd $ESMADIR/src/Applications/GEOSgcm_App
$ echo $NOBACKUP > ~/.HOMDIRroot
$ echo $NOBACKUP > ~/.EXPDIRroot

These echos set up some defaults. While they both don't have to be in the same location, it's highly recommended they are. Also, make sure they are in a nobackup directory.

Now you can run create_expt.py:

$ ~mathomp4/bin/create_exp.py -h
usage: create_expt.py [-h] [-v] [-q] [--expdsc EXPDSC]
                      [--horz {a,b,c,d,e,c12,c24,c48,c90,c180,c360,c720,c1440,c2880}]
                      [--vert {72,132}] [--ocean {o1,o2,o3}] [--land {1,2}]
                      [--runoff {yes,no}] [--gocart {A,C}]
                      [--emission {MERRA2,PIESA,CMIP,NR,MERRA2-DD,OPS}]
                      [--history HISTORY] [--account ACCOUNT] [--gpu]
                      expid

Utility to quickly create experiment. At present, it creates an experiment in
the same manner as gcm_setup would with home and experiment directories as
usual.

positional arguments:
  expid                 Experiment ID

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Verbose output
  -q, --quiet           Quietly Setup Experiment (no printing)
  --expdsc EXPDSC       Experiment Description (Default: same as expid)
  --horz {a,b,c,d,e,c12,c24,c48,c90,c180,c360,c720,c1440,c2880}
                        Horizontal Resolution (Default: c48 on clusters, c12
                        on desktop)
  --vert {72,132}       Vertical Resolution (Default: 72)
  --ocean {o1,o2,o3}    Data Ocean Resolution (Default: o1)
  --land {1,2}          Land Surface Model (Default: 1)
  --runoff {yes,no}     Runoff Routing Model (Default: no)
  --gocart {A,C}        GOCART aerosols: Actual (A) or Climatological (C)
                        (Default: A on clusters, C on desktops)
  --emission {MERRA2,PIESA,CMIP,NR,MERRA2-DD,OPS}
                        GOCART Emissions to use (Default: MERRA2)
  --history HISTORY     History Template (Default: Current)
  --account ACCOUNT     Account Number to Use (Default: g0620 at NCCS, g26141
                        at NAS)
  --gpu                 Setup Experiment to use GPUs

This is a script that attempts to ease setting up a GEOS-5 AGCM run. It has some "smarts" in that it will detect from an experiment ID some information, but you can be explicit (to the point you can specify impossible experiments too).

Run a one-day test

To run the job, first change to the experiment directory you specified in the above gcm_setup. Then, copy a set of restarts to model directory. Sample restarts are provided on Discover (at /discover/nobackup/mathomp4/Restarts-G10) and Pleiades (at /nobackup/gmao_SIteam/ModelData/Restarts-G10/).

Then, edit CAP.rc:

JOB_SGMT should be 1 day (00000001 000000)
NUM_SGMT should be 1

and gcm_run.j by inserting "exit" after

$RUN_CMD $NPES ./GEOSgcm.x
set rc =  $status
echo       Status = $rc 

A script that can automate these processes is:

#!/bin/bash

sed -i'.orig' -e '/^JOB_SGMT:/ s/000000[0-9][0-9]/00000001/' \
              -e '/^NUM_SGMT:/ s/[0-9][0-9]*/1/' \
              -e '/^MAPL_ENABLE_TIMERS:/ s/NO/YES/' CAP.rc

sed -i'.orig' -e '/^echo       Sta/ a exit' \
              -e '/^#PBS -j oe/ a #PBS -m abe\n#PBS -M email@myserver.com' \
              -e '/^  if(-e $EXPDIR\/$rst ) \/bin/ s/cp/ln -s/' gcm_run.j

Note: This script also sets up email notification as well as linking restarts to the scratch/ directory rather than copying (which is nice if running at high-res).

Compare to a "correct" run by using cmpscratch

a. If all success, runs are bit-identical b. If not => "What is correct?"