View source for Heracles 5.3 Quick Start

This page describes the minimum steps required to build and run GEOS-5 Heracles 5.3 on NCCS discover and NAS pleiades.  '''You should successfully complete the steps in these instructions before doing anything more complicated.  Also, it is helpful to read this page in its entirety before starting.'''  

'''Back to [[GEOS-5 Documentation for Heracles 5.3]]'''

== How to Obtain GEOS-5 and Compile Source Code ==

There are two options for obtaining the model source code: from the CVS repository on the NCCS progress server, and from the SVN "public" repository on the trac server.  Since the code on progress is more current, elgible users are strongly encouraged to obtain accounts from NCCS and use the progress repository.

=== Using the NCCS progress CVS code repository ===

The following assumes that you know your way around Unix, have successfully logged into your cluster account and have an account on the source code repository with the proper <code>ssh</code> configuration -- see the progress repository quick start pages at: https://progress.nccs.nasa.gov/trac/admin/wiki/QuickStart. The link requires your NCCS username and password. The recommend SSH .config setup for CVS on discover is:

 Host cvsacldirect
     HostName cvsacl.nccs.nasa.gov
     Port 22223

That's it. Progress is not needed unless you specifically know you need it. It won't hurt to add it, but at present it isn't needed. Also, you'll need to generate RSA and/or DSA keys and upload them (this is mentioned in the quick start page above) to https://progress.nccs.nasa.gov/keyupload/.

The commands below assume that your shell is <code>csh</code>.  Since the scripts to build and run GEOS-5  tend to be written in the same, you shouldn't bother trying to import too much into an alternative shell.  If you prefer a different shell, it is easiest just to open a <code>csh</code> process to build the model and your experiment.

Furthermore, model builds should be created in your space under <code>/discover/nobackup</code>, as creating them under your home directory will quickly wipe out your disk quota.

Set the following three environment variables:

 setenv CVS_RSH ssh
 setenv CVSROOT :ext:''USERID''@cvsacldirect:/cvsroot/esma

where ''USERID'' is, of course, your repository username, which should be the same as your NASA and NCCS username.  Then, issue the command:

 cvs co -r Heracles-5_2 GEOSagcm

This should check out the latest stable version of the model from the repository and create a directory called <code>GEOSagcm</code>.

=== Compiling the Model ===

First, you need to set <code>ESMADIR</code>. For example, if your <code>src/</code> directory is:

 /discover/nobackup/mathomp4/Models/Heracles-5_2/GEOSagcm/src

then you should set:

 setenv ESMADIR /discover/nobackup/mathomp4/Models/Heracles-5_2/GEOSagcm

Next, <code>cd</code> into <code>GEOSagcm/src</code> and <code>source</code> the file called <code>g5_modules</code>:

 source g5_modules

This will set up the build environment.  If you then type 

 module list 

you should see:

 Currently Loaded Modulefiles:
   1) comp/intel-15.0.2.164
   2) mpi/impi-5.0.3.048
   3) lib/mkl-15.0.2.164
   4) other/comp/gcc-4.6.3-sp1
   5) other/SIVO-PyD/spd_1.20.0_gcc-4.6.3-sp1_mkl-15.0.0.090


If this all worked, then type:

 gmake install

This will build the model.  It will take about 30 minutes.  If this works, it should create a directory under <code>GEOSagcm</code> called <code>Linux/bin</code>.  In here you should find the executable: <code>GEOSgcm.x</code> .

== Setting up a Run ==

First of all, to run jobs on the cluster you will need to set up passwordless <code>ssh</code> (which operates within the cluster, between the nodes running the job).  To do so, run the following from your '''discover''' home directory:

 cd .ssh
 ssh-keygen -t dsa
 cat id_dsa.pub >>  authorized_keys

Similarly, transferring the daily output files (in monthly tarballs) requires passwordless authentication from '''discover''' to '''dirac'''.  While in <code>~/.ssh</code> on '''discover''', run 

  ssh-keygen -t dsa

Then, log into  '''dirac''' and cut and paste the contents of the <code>id_rsa.pub</code> and <code>id_dsa.pub</code> files on '''discover''' into the  <code>~/.ssh/authorized_keys</code> file on   '''dirac'''.  Problems with <code>ssh</code> should be referred to NCCS support.

To set the model up to run, cd to <code>GEOSagcm/src/Applications/GEOSgcm_App</code> and run:

 ./gcm_setup

The <code>gcm_setup</code> script asks you to provide an experiment name :

 Enter the Experiment ID:

Your experiment name (later called ''EXPID'') should be one word with no spaces, not starting with a numeral.   Then the script will ask for a description:

 Enter a 1-line Experiment Description:

Spaces are ok here.  It will then ask for the source code version tag to associate with the model -- you should hit enter for the default:

 Enter an Experiment Source Tag for History (Default: Heracles-5_2):

Hit enter for the default to the next question:

 Do you wish to CLONE an old experiment? (Default: NO or FALSE)

It will also ask you for the atmospheric model resolution, expecting the code for one of the displayed resolutions. 

 Enter the Atmospheric Horizontal Resolution code:
 -----------------------------------------------------------
      Lat/Lon                     Cubed-Sphere
 -----------------------------------------------------------
    b --  2  deg                c48  --  2   deg 
    c --  1  deg                c90  --  1   deg 
    d -- 1/2 deg                c180 -- 1/2  deg (56-km) 
    e -- 1/4 deg (35-km)        c360 -- 1/4  deg (28-km)  
                                c720 -- 1/8  deg (14-km) 
                                c1440 - 1/16 deg ( 7-km) 
 
For your first time out you will probably want to enter <code>c48</code> (corresponding to ~2 degree resolution with the cubed sphere).  On the next eight questions, hitting enter to accept the default will let you run a PChem run:

 Enter the Atmospheric Model Vertical Resolution: LM (Default: 72)
 
 Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE)
 
 Enter the Data_Ocean Horizontal Resolution code: o1 (1  -deg,  360x180  Reynolds) Default
                                                  o2 (1/4-deg, 1440x720  MERRA-2)
                                                  o3 (1/8-deg, 2880x1440 OSTIA)
 
 Do you wish to run GOCART with Actual or Climatological Aerosols? (Enter: A (Default) or C)


 Enter the GOCART Emission Files to use: MERRA2 (Default), PIESA, CMIP, NR, MERRA2-DD or OPS:
 
 Enter the tag or directory (/filename) of the HISTORY.AGCM.rc.tmpl to use
 (To use HISTORY.AGCM.rc.tmpl from current build, Type:  Current         )
 -------------------------------------------------------------------------
 Hit ENTER to use Default Tag/Location: (Current)

'''NOTE''': Some/Many things are easier if your HOME and EXPERIMENT directories are the same. For the next two, look carefully at the default and make sure they are both pointing to the same <code>nobackup</code> location.

 Enter Desired Location for the HOME Directory (to contain scripts and RC files)
 Hit ENTER to use Default Location:
 ----------------------------------
 Default:  /discover/nobackup/''USER''/''EXPID''
 
 Enter Desired Location for the EXPERIMENT Directory (to contain model output and restart files)
 Hit ENTER to use Default Location:
 ----------------------------------
 Default:  ~''USER''/geos5/''EXPID''
 /discover/nobackup/''USER''/''EXPID'' 
 
 Enter Location for Build directory containing:  src/ Linux/ etc...
 Hit ENTER to use Default Location:
 ----------------------------------
 Default:  /discover/nobackup/''USER''/GEOSagcm 
 
After these it will ask you for a group ID -- the default for this writer is g0620 (GMAO modeling group).  Enter whatever is appropriate, as necessary.  

  Current GROUPS: g0620 gmaoint
 Enter your GROUP ID for Current EXP: (Default: g0620)
 -----------------------------------


The script will produce some messages and create an experiment directory (''EXPDIR'') in your space as <code>/discover/nobackup/''USERID''/''EXPID''</code>, which contains the files and sub-directories:

*<code>AGCM.rc</code> -- resource file with specifications of boundary conditions, initial conditions, parameters, etc.
*<code>archive/</code> -- contains job script for archiving output
*<code>CAP.rc</code> -- resource file with run job parameters
*<code>convert</code> -- contains job script that converts restarts (initial condition files) from older model versions
*<code>ExtData.rc</code> -- sample resource file for external data, not used
*<code>forecasts/</code> -- contains scripts used for data assimilation mode
*<code>fvcore_layout.rc</code> --
*<code>gcm_run.j</code> -- run script
*<code>GEOSgcm.x</code> -- model executable
*<code>HISTORY.rc</code> -- resource file specifying the fields in the model that are output as data
*<code>plot/</code> -- contains plotting job script template and .rc file
*<code>post/</code> -- contains the script template and .rc file for post-processing model output
*<code>RC/</code> -- contains resource files for various components of the model
*<code>regress/</code> -- contains scripts for doing regression testing of model
*<code>src</code> -- directory with a tarball of the model version's source code


The post-processing script will generate the archiving and plotting scripts as it runs.  The setup script that you ran also creates an experiment home directory (''HOMDIR'') as either in <code>~''USERID''/geos5/''EXPID''</code> (if you accepted the default) or in <code>/discover/nobackup/''USERID''/''EXPID''</code> (if you followed the above advice) containing the run scripts and GEOS resource (<code>.rc</code>) files.

== Running GEOS-5 ==

Before running the model, there is some more setup to be completed.  The run scripts need some environment variables set in <code>~/.cshrc</code> (regardless of which login shell you use -- the GEOS-5 scripts use <code>csh</code>).  Here are the minimum contents of a <code>.cshrc</code>:

 umask 022
 unlimit
 limit stacksize unlimited
 set arch = `uname`

The <code>umask 022</code> is not strictly necessary, but it will make the various files readable to others, which will facilitate data sharing and user support.  Your home directory <code>~''USERID''</code> is also inaccessible to others by default; running <code>chmod 755 ~</code> is helpful.

Copy the restart (initial condition) files and associated <code>cap_restart</code> into ''EXPDIR''.  You can get an arbitrary set of restarts by copying the contents of the directory <code>/discover/nobackup/mathomp4/Restarts-H10/nc4/Reynolds/c48</code>, containing 2-degree cubed sphere restarts from April 14, 2000, and their corresponding <code>cap_restart</code>.  

The script you submit, <code>gcm_run.j</code>, is in ''HOMEDIR''.  It should be ready to go as is.  The parameter END_DATE in <code>CAP.rc</code> can be set to the date you want the run to stop.  Submit the job with <code>sbatch gcm_run.j</code>.  You can keep track of it with <code>squeue</code> or <code>squeue -u ''USERID''</code>, or follow stdout with <code>tail -f ''EXPDIR''/slurm-''JOBID''.out</code>, ''JOBID'' being returned by <code>sbatch</code> and displayed with <code>squeue</code>.  Jobs can be killed with <code>scancel ''JOBID''</code>.

== Output and Plots ==

During a normal run, the <code>gcm_run.j</code> script will run the model for the segment length (current default is 15 days in model time).  The model creates output files (with an <code>nc4</code> extension), also called collections (of output variables), in  <code>''EXPDIR''/scratch</code> directory.  After each segment, the script moves the output to the <code>''EXPDIR''/holding</code> and spawns a post-processing batch job which partitions and moves the output files  within the <code>holding</code> directory to their own distinct collection directory, which is again partitioned into the appropriate year and month.  The  post processing script then checks to
see if  a full month of data is present.  If not, the post-processing job ends.  If there is a full month, the script will then run the time-averaging executable to produce a monthly mean file in <code>''EXPDIR''/geosgcm_*</code>.  The post-processing script then spawns a new batch job which will archive the data onto the mass-storage drives (<code>/archive/u/''USERID''/GEOS5.0/''EXPID''</code>).

If a monthly average file was made, the post-processing script will also
check to see if it should spawn a plot job.  Currently, our criteria for
plotting are:   1) if the month created was February or August,   AND
2) there are at least 3 monthly average files, then a plotting job for
the seasons DJF or JJA will be issued.  The plots are created as gifs in <code>''EXPDIR''/plots_CLIM</code>.

The post-processing script can be found in:
<code>GEOSagcm/src/GMAO_Shared/GEOS_Util/post/gcmpost.script</code>.  The <code>nc4</code> output files can be opened and plotted with <code>grads</code> -- see http://www.iges.org/grads/gadoc/tutorial.html for a tutorial, but use <code>sdfopen</code> instead of <code>open</code>.

The contents of the output files (including which variables get saved) may be configured in the  <code>''HOMEDIR''/HISTORY.rc</code> -- a good description of this file may be found at http://modelingguru.nasa.gov/clearspace/docs/DOC-1190 .

----

'''Back to [[GEOS-5 Documentation for Heracles 5.3]]'''

Contact Matthew Thompson at GMAO with questions and comments