Ganymed 3.0 Quick Start
This page describes the minimum steps required to build and run GEOS-5 Ganymed 3.0 on NCCS discover and NAS pleiades. You should successfully complete the steps in these instructions before doing anything more complicated. Also, it is helpful to read this page in its entirety before starting.
Back to GEOS-5 Documentation for Ganymed 3.0
How to Obtain GEOS-5 and Compile Source Code
There are two options for obtaining the model source code: from the CVS repository on the NCCS progress server, and from the SVN "public" repository on the trac server. Since the code on progress is more current, elgible users are strongly encouraged to obtain accounts from NCCS and use the progress repository.
Using the NCCS progress CVS code repository
The following assumes that you know your way around Unix, have successfully logged into your cluster account and have an account on the source code repository with the proper ssh configuration -- see the progress repository quick start: https://progress.nccs.nasa.gov/trac/admin/wiki/CVSACL. The link requires your NCCS username and password.
The commands below assume that your shell is csh. Since the scripts to build and run GEOS-5 tend to be written in the same, you shouldn't bother trying to import too much into an alternative shell. If you prefer a different shell, it is easiest just to open a csh process to build the model and your experiment.
Furthermore, model builds should be created in your space under /discover/nobackup, as creating them under your home directory will quickly wipe out your disk quota.
Set the following three environment variables:
setenv CVS_RSH ssh setenv CVSROOT :ext:USERID@cvsacl:/cvsroot/esma
where USERID is, of course, your repository username, which should be the same as your NASA and NCCS username. Then, issue the command:
cvs co -r Ganymed-3_0_p1 -d Ganymed-3_0_p1 Ganymed
This should check out the latest stable version of the model from the repository and create a directory called Ganymed-3_0_p1.
Compiling the Model
cd into Ganymed-3_0_p1/src and source the file called g5_modules:
source g5_modules
This will set up the build environment. If you then type
module list
you should see:
Currently Loaded Modulefiles: 1) comp/intel-11.0.083 3) lib/mkl-10.0.3.020 2) mpi/impi-3.2.2.006 4) other/SIVO-PyD/spd_1.6.0_gcc-4.3.4-sp1
If this all worked, then type:
gmake install
This will build the model. It will take about 40 minutes. If this works, it should create a directory under Ganymed-1_0 called Linux/bin. In here you should find the executable: GEOSgcm.x .
Setting up a Run
First of all, to run jobs on the cluster you will need to set up passwordless ssh (which operates within the cluster, between the nodes running the job). To do so, run the following from your discover home directory:
cd .ssh ssh-keygen -t dsa cat id_dsa.pub >> authorized_keys
Similarly, transferring the daily output files (in monthly tarballs) requires passwordless authentication from discover to dirac. While in ~/.ssh on discover, run
ssh-keygen -t dsa
Then, log into dirac and cut and paste the contents of the id_rsa.pub and id_dsa.pub files on discover into the ~/.ssh/authorized_keys file on dirac. Problems with ssh should be referred to NCCS support.
To set the model up to run, cd to Ganymed-1_0/src/Applications/GEOSgcm_App and run:
./gcm_setup
The gcm_setup script asks you to provide an experiment name :
Enter the Experiment ID:
Your experiment name (later called EXPID) should be one word with no spaces, not starting with a numeral. Then the script will ask for a description:
Enter a 1-line Experiment Description:
Spaces are ok here. It will also ask you for the atmospheric model resolution, expecting the code for one of the displayed resolutions.
Enter the Atmospheric Horizontal Resolution code:
b -- 2 deg
c -- 1 deg
d -- 1/2 deg
e -- 1/4 deg
o -- other
For your first time out you will probably want to enter b (corresponding to ~2 degree resolution). Then it asks you which dynamical core to use:
Enter the Dynamical Core to use: FV (Lat-Lon), FV3 (Cubed-Sphere)
Enter FV for lat-lon. On the next seven questions, hit enter to accept the default:
Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE)
Enter the Data_Ocean Horizontal Resolution code: o1 (1 -deg, 360x180, (e.g. Reynolds) Default)
o8 (1/8-deg, 2880x1440, (e.g. OSTIA))
Do you wish to run GOCART? (Default: NO or FALSE)
Enter the tag or directory (/filename) of the HISTORY.AGCM.rc.tmpl to use (To use HISTORY.AGCM.rc.tmpl from current build, Type: Current ) ------------------------------------------------------------------------- Hit ENTER to use Default Tag/Location: (Current)
Enter Desired Location for the HOME Directory (to contain scripts and RC files) Hit ENTER to use Default Location: ---------------------------------- Default: /discover/nobackup/USER/EXPID
Enter Desired Location for the EXPERIMENT Directory (to contain model output and restart files) Hit ENTER to use Default Location: ---------------------------------- Default: ~USER/geos5/EXPID
Enter Location for Build directory containing: src/ Linux/ etc... Hit ENTER to use Default Location: ---------------------------------- Default: /discover/nobackup/USER/GEOSgcm
After these it will ask you for a group ID -- the default is g0602 (GMAO modeling group). Enter whatever is appropriate, as necessary.
Current GROUPS: g0620 gmaoint Enter your GROUP ID for Current EXP: (Default: g0620) -----------------------------------
The script produces an experiment directory (EXPDIR) in your space as /discover/nobackup/USERID/EXPID, which contains the files and sub-directories:
AGCM.rc-- resource file with specifications of boundary conditions, initial conditions, parameters, etc.archive/-- contains job script for archiving outputCAP.rc-- resource file with run job parametersExtData.rc-- sample resource file for external data, not usedforecasts/-- contains scripts used for data assimilation modegcm_run.j-- run scriptGEOSgcm.x-- model executableHISTORY.rc-- resource file specifying the fields in the model that are output as dataplot/-- contains plotting job script template and .rc filepost/-- contains the script template and .rc file for post-processing model outputRC/-- contains resource files for various components of the modelregress/-- contains scripts for doing regression testing of model
The post-processing script will generate the archiving and plotting scripts as it runs. The setup script that you ran also creates an experiment home directory (HOMEDIR) as ~USERID/geos5/EXPID containing the run scripts and GEOS resource (.rc) files. (You can also specify the experiment home directory to be the same as the experiment directory.)
Running GEOS-5
Before running the model, there is some more setup to be completed. The run scripts need some environment variables set in ~/.cshrc (regardless of which login shell you use -- the GEOS-5 scripts use csh). Here are the minimum contents of a .cshrc:
umask 022
unlimit
limit stacksize unlimited
set arch = `uname`
setenv LD_LIBRARY_PATH ${LIBRARY_PATH}:${BASEDIR}/${arch}/lib
The umask 022 is not strictly necessary, but it will make the various files readable to others, which will facilitate data sharing and user support. Your home directory ~USERID is also inaccessible to others by default; running chmod 755 ~ is helpful.
Copy the restart (initial condition) files and associated cap_restart into EXPDIR. Keep the "originals" handy since if the job stumbles early in the run it might stop after having renamed them. The model expects restart filenames to end in "rst" but produces them with the date and time appended, so you may have to rename them back to ending in "rst". The cap_restart file is sometimes provided with a set of restarts, but if not you can create it: it is simply one line of text with the date of the restart files in the format YYYYMMDD HHMMSS (with a space). The boundary conditions/forcings are provided by symbolic links created by the run script. If you need an arbitrary set of restarts, you can copy them from /archive/u/aeichman/restarts/Ganymed-1_0/, where they are indexed by resolution and date. If you are unfamiliar with the way that the /archive filesystem works, keep in mind that a cp from there might appear to stall while the tape is loaded -- see the NCCS documentation for details.
The script you submit, gcm_run.j, is in HOMEDIR. It should be ready to go as is. The parameter END_DATE in CAP.rc can be set to the date you want the run to stop. An alternative way to stop the run is by commenting out the line if ( $capdate < $enddate ) qsub $HOMDIR/gcm_run.j at the end of the script, which will prevent the script from being resubmitted, or rename the script file, or kill the job (described below).
Submit the job with qsub gcm_run.j. You can keep track of it with qstat or qstat | grep USERID, or follow stdout with tail -f /discover/pbs_spool/JOBID.OU, JOBID being returned by qsub and displayed with qstat. Jobs can be killed with qdel JOBID. The standard out and standard error will be delivered as files to the working directory at the time you submitted the job.
Output and Plots
During a normal run, the gcm_run.j script will run the model for the segment length (current default is 15 days in model time). The model creates output files (with an nc4 extension), also called collections (of output variables), in EXPDIR/scratch directory. After each segment, the script moves the output to the EXPDIR/holding and spawns a post-processing batch job which partitions and moves the output files within the holding directory to their own distinct collection directory, which is again partitioned into the appropriate year and month. The post processing script then checks to
see if a full month of data is present. If not, the post-processing job ends. If there is a full month, the script will then run the time-averaging executable to produce a monthly mean file in EXPDIR/geosgcm_*. The post-processing script then spawns a new batch job which will archive the data onto the mass-storage drives (/archive/u/USERID/GEOS5.0/EXPID).
If a monthly average file was made, the post-processing script will also
check to see if it should spawn a plot job. Currently, our criteria for
plotting are: 1) if the month created was February or August, AND
2) there are at least 3 monthly average files, then a plotting job for
the seasons DJF or JJA will be issued. The plots are created as gifs in EXPDIR/plots_CLIM.
The post-processing script can be found in:
GEOSagcm/src/GMAO_Shared/GEOS_Util/post/gcmpost.script. The nc4 output files can be opened and plotted with gradsnc4 -- see http://www.iges.org/grads/gadoc/tutorial.html for a tutorial, but use sdfopen instead of open.
The contents of the output files (including which variables get saved) may be configured in the HOMEDIR/HISTORY.rc -- a good description of this file may be found at http://modelingguru.nasa.gov/clearspace/docs/DOC-1190 .