GEOS-5

Running the GEOS-5 SBU Benchmark: Difference between revisions

← Older edit
Clean up for Benchmark
 
(23 intermediate revisions by the same user not shown)
Line 1: Line 1:
==Build and install the model==
==Build and install the model==


First, untar the model tarball (in $NOBACKUP!!!):
First, untar the model tarball in <tt>nobackup</tt> or <tt>swdev</tt>; the model alone will break the <tt>home</tt> quota:


  $ tar xf Heracles-UNSTABLE-MPT-Benchmark.2017Feb13.tar.gz
  $ tar xf GEOSadas-5_16_5-Benchmark.tar.gz


Next, set up ESMADIR:
Next, set up ESMADIR:


  $ setenv ESMADIR <directory-to>/Heracles-UNSTABLE-MPT-Benchmark/GEOSagcm
  $ setenv ESMADIR <directory-to>/GEOSadas-5_16_5-Benchmark/GEOSadas


it is just below the src/ directory.
it is just below the src/ directory.
Line 84: Line 84:
  All 22 packages compiled successfully.
  All 22 packages compiled successfully.


In case of errors, gmh summarizes exactly where it happens by indicating the package where it occured. Caveat: it does not work in parallel (output is scrambled). So, if the parallel build fails, rerun it sequentially (it will go quickly and die in the same place) and run gmh on the output for a summary.
In case of errors, gmh summarizes exactly where it happens by indicating the package where it occurred. Caveat: it does not work in parallel (output is scrambled). So, if the parallel build fails, rerun it sequentially (it will go quickly and die in the same place) and run gmh on the output for a summary.
<!--
<!--
===Advanced features===
===Advanced features===
Line 106: Line 106:
These effectively let you change whatever you want - useful for debugging, etc. For example, you can set your timers in ~/.esma_base.mk.
These effectively let you change whatever you want - useful for debugging, etc. For example, you can set your timers in ~/.esma_base.mk.


-->
==Run the model==


===Setting up an experiment===
==Test Build with One-Day Run==
 
To make sure all works, we will first try setting up a simple one-day experiment.
 
 
===Setting up a one-day experiment===


Go into the model application directory and run the setup script:
Go into the model application directory and do a couple of preliminary commands:


  $ cd $ESMADIR/src/Applications/GEOSgcm_App
  $ cd $ESMADIR/src/Applications/GEOSgcm_App
$ ./gcm_setup


and provide required answers.
Now you can run <tt>create_expt.py</tt>:
 
<nowiki>$ ~mathomp4/bin/create_exp.py -h
usage: create_expt.py [-h] [-v] [-q] [--expdsc EXPDSC] [--expdir EXPDIR]
                      [--horz {a,b,c,d,e,c12,c24,c48,c90,c180,c360,c720,c1440,c2880}]
                      [--vert {72,132}] [--ocean {o1,o2,o3}] [--land {1,2}]
                      [--runoff {yes,no}] [--gocart {A,C}]
                      [--emission {MERRA2,PIESA,CMIP,NR,MERRA2-DD,OPS}]
                      [--history HISTORY] [--account ACCOUNT] [--gpu]
                      expid
 
Utility to quickly create experiment. At present, it creates an experiment in
the same manner as gcm_setup would with home and experiment directories as
usual.
 
positional arguments:
  expid                Experiment ID
 
optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose        Verbose output
  -q, --quiet          Quietly Setup Experiment (no printing)
  --expdsc EXPDSC      Experiment Description (Default: same as expid)
  --expdir EXPDIR      Experiment Directory Root *NOT CONTAINING EXPID*
                        (Default is what is in ~/.EXPDIRroot:
                        /discover/nobackup/mathomp4 )
  --horz {a,b,c,d,e,c12,c24,c48,c90,c180,c360,c720,c1440,c2880}
                        Horizontal Resolution (Default: c48 on clusters, c12
                        on desktop)
  --vert {72,132}      Vertical Resolution (Default: 72)
  --ocean {o1,o2,o3}    Data Ocean Resolution (Default: o1)
  --land {1,2}          Land Surface Model (Default: 1)
  --runoff {yes,no}    Runoff Routing Model (Default: no)
  --gocart {A,C}        GOCART aerosols: Actual (A) or Climatological (C)
                        (Default: A on clusters, C on desktops)
  --emission {MERRA2,PIESA,CMIP,NR,MERRA2-DD,OPS}
                        GOCART Emissions to use (Default: MERRA2)
  --history HISTORY    History Template (Default: Current)
  --account ACCOUNT    Account Number to Use (Default: g0620 at NCCS, g26141
                        at NAS)
  --gpu                Setup Experiment to use GPUs</nowiki>
 
This is a script that attempts to ease setting up a GEOS-5 AGCM run. It has some "smarts" in that it will detect from an experiment ID some information, but you can be explicit (to the point you can specify impossible experiments too).
 
Your first choice is where you'd like to run the experiment. $NOBACKUP or other disks with space are vital. For example, if you'd like to run the experiment in /discover/nobackup/username/experiment-name, then run the script with --expdir /discover/nobackup/username. To actually create the experiment, run the script and choose a C48 horizontal resolution:
 
$ ~mathomp4/bin/create_expt.py test-1day-expt --horz c48 --account <ACCOUNTID> --expdir /discover/nobackup/username
Horizontal resolution c48 passed in
Using c48 horizontal resolution
Assuming default vertical resolution of 72
Using 72 vertical resolution
Assuming default ocean resolution of o1
Using o1 ocean resolution
Using actual aerosols
  Running gcm_setup...done.
Experiment is located in directory: /discover/nobackup/username/test-1day-expt


'''''MAT: Add a sample run here'''''
If you don't pass in an account-id, you'll get the default of g0620 (the developer's account).


===Run a one-day test===
===Run a one-day test===


To run the job, first change to the experiment directory you specified in the above <code>gcm_setup</code>.  Then, copy a set of restarts to model directory. Sample restarts are provided on Discover (at <code>/discover/nobackup/mathomp4/Restarts-G10</code>) and Pleiades (at <code>/nobackup/gmao_SIteam/ModelData/Restarts-G10/</code>).
To run a test one-day run, go to the experiment directory:


Then, edit CAP.rc:
<nowiki>$ cd /discover/nobackup/mathomp4/test-1day-expt
total 96M
JOB_SGMT should be 1 day (00000001 000000)
-rwxr-xr-x 1 mathomp4 g0620 96M Feb 13 11:11 GEOSgcm.x*
  NUM_SGMT should be 1
-rwxr-xr-x 1 mathomp4 g0620 573 Feb 13 14:47 CAP.rc*
-rwxr-xr-x 1 mathomp4 g0620 364 Feb 13 14:47 fvcore_layout.rc*
-rw-r--r-- 1 mathomp4 g0620 25K Feb 13 14:47 AGCM.rc
-rwxr-xr-x 1 mathomp4 g0620 21K Feb 13 14:47 gcm_run.j*
-rw-r--r-- 1 mathomp4 g0620 61K Feb 13 14:47 HISTORY.rc
drwxr-xr-x 2 mathomp4 g0620 512 Feb 13 14:47 post/
drwxr-xr-x 2 mathomp4 g0620 512 Feb 13 14:47 plot/
drwxr-xr-x 2 mathomp4 g0620 512 Feb 13 14:47 archive/
drwxr-xr-x 2 mathomp4 g0620 512 Feb 13 14:47 regress/
drwxr-xr-x 2 mathomp4 g0620 512 Feb 13 14:47 convert/
drwxr-xr-x 2 mathomp4 g0620 512 Feb 13 14:47 forecasts/
drwxr-xr-x 2 mathomp4 g0620 16K Feb 13 14:47 RC/
Directory: /discover/nobackup/mathomp4/test-1day-expt</nowiki>
 
Now you'll run another script called <tt>makeoneday.bash</tt> that edits some .rc files, edits run scripts, as well as linking in some example restarts and sets up the model to stop after one day (with no options). Note: these restarts are write-protected, so if you want to run a multi-year run, don't use this script!
 
<nowiki>$ ~mathomp4/bin/makeoneday.bash
Using Heracles-5_0 directories
Making one-day experiment
 
Restoring AGCM.rc.save to AGCM.rc...
Copying AGCM.rc to AGCM.rc.save...
Restoring CAP.rc.save to CAP.rc...
Copying CAP.rc to CAP.rc.save...
Restoring HISTORY.rc.save to HISTORY.rc...
Copying HISTORY.rc to HISTORY.rc.save...
Restoring gcm_run.j.save to gcm_run.j...
Copying gcm_run.j to gcm_run.j.save...
Restoring regress/gcm_regress.j.save to regress/gcm_regress.j...
Copying regress/gcm_regress.j to regress/gcm_regress.j.save...
Restoring RC/GEOS_ChemGridComp.rc.save to RC/GEOS_ChemGridComp.rc...
Copying RC/GEOS_ChemGridComp.rc to RC/GEOS_ChemGridComp.rc.save...
DYN_INTERNAL_RESTART_TYPE not found. Assuming NC4
Found fvcore_internal_rst. Assuming you have needed restarts!
Changes made to CAP.rc:
9,10c9,10
< JOB_SGMT:    00000015 000000
< NUM_SGMT:    20
---
> JOB_SGMT:    00000001 000000
> NUM_SGMT:    1
25c25
< MAPL_ENABLE_TIMERS: NO
---
> MAPL_ENABLE_TIMERS: YES
 
 
Changes made to gcm_run.j:
7c7
< #PBS -l walltime=12:00:00
---
> #PBS -l walltime=0:15:00
11a12
> #SBATCH --mail-type=ALL
260c261
< if($numrs == 0) then
---
> if($numrs == 1) then
433a435
> exit
</nowiki>
 
At this point, you should be able to <tt>sbatch gcm_run.j</tt> and the model should run a day.
 
-->
 
==Benchmark Run==
 
The full benchmark run is a run of 5-days using a portable version of the GEOS-5 boundary conditions. This will use space and cores. For effective benchmarking of I/O, it's recommended to run on less congested than nobackup.
 
===Learn to love tcsh===
 
One preliminary note is that GEOS-5 is, in many ways, a collection of csh/tcsh scripts. If things start going wrong, the answer can often be "change your shell to tcsh and try". Yes, it's not bash/fish/zsh, but it is what it is. I don't think it's entirely necessary for just this automated work, but it could happen.
 
===Setting up benchmark experiment===
 
Go into the model application directory and do a couple of preliminary commands:
 
  $ cd $ESMADIR/src/Applications/GEOSgcm_App
 
These echos set up some defaults. While they both don't have to be in the same location, it's highly recommended they are and the use of the script below assumes they are. Also, make sure they are in a nobackup directory.


and gcm_run.j by inserting "exit" after
For the next few commands, you will need to know the location of the portable BCs directory used for this experiment, which is referred to below as <tt>$PORTBCS</tt>. On discover, a version will always be at:


  $RUN_CMD $NPES ./GEOSgcm.x
  /discover/nobackup/mathomp4/HugeBCs-H50
set rc =  $status
echo      Status = $rc


A script that can automate these processes is:
To create the experiment, run create_expt.py and choose a C720 horizontal resolution with climatological GOCART:


  #!/bin/bash
  $ $PORTBCS/scripts/create_expt.py benchmark-GEOSadas-5_16-5-5day-c720 --horz c720 --ocean o3 --gocart C --account <ACCOUNTID> --expdir <root-for-experiment>
Found c720 horizontal resolution in experiment name
Using c720 horizontal resolution
   
   
  sed -i'.orig' -e '/^JOB_SGMT:/ s/000000[0-9][0-9]/00000001/' \
  Assuming default vertical resolution of 72
              -e '/^NUM_SGMT:/ s/[0-9][0-9]*/1/' \
Using 72 vertical resolution
              -e '/^MAPL_ENABLE_TIMERS:/ s/NO/YES/' CAP.rc
   
   
  sed -i'.orig' -e '/^echo      Sta/ a exit' \
  Ocean resolution of o3 passed in
              -e '/^#PBS -j oe/ a #PBS -m abe\n#PBS -M ''email@myserver.com''' \
Using o3 ocean resolution
              -e '/^ if(-e $EXPDIR\/$rst ) \/bin/ s/cp/ln -s/' gcm_run.j
Using climatological aerosols
  Running gcm_setup...done.
Experiment is located in directory: <root-for-experiment>/benchmark-GEOSadas-5_16-5-5day-c720
 
Again, if you don't pass in an account-id, you'll get the default of g0620 (the developer's account).
 
===Setup and Run Benchmark===
 
Now ''change to the experiment directory'' and run MakeSBUBench.bash which will set up the experiment:
 
  $ $PORTBCS/scripts/MakeSBUBench.bash
 
The script sets the run up for 5 days using 5400 cores, and other flags are tripped to best emulate Ops.
 
'''NOTE''': This will also set the experiment to run in this SLURM environment:
 
#SBATCH --partition=preops
#SBATCH --qos=benchmark
 
This is how the script's developer can run 5400-core jobs. Others might have different partition/qos to submit to. Please edit these before sbatch submission to a <tt>partition/qos</tt> that you have access to that can accept a 5400-core job.


Note: This script also sets up email notification as well as linking restarts to the scratch/ directory rather than copying (which is nice if running at high-res).
Finally, submit the job:


Compare to a "correct" run by using cmpscratch
$ sbatch gcm_run.j


a. If all success, runs are bit-identical
b. If not => "What is correct?"


[[Category:SI Team]]
[[Category:SI Team]]
[[Category:CVS]]
[[Category:CVS]]
[[Category:Running the Model]]
[[Category:Running the Model]]
Retrieved from "https://geos5.org/wiki/index.php?title=Running_the_GEOS-5_SBU_Benchmark"