GEOS GCM Quick Start

This page describes the minimum steps required to build and run GEOS GCM on NCCS discover and NAS pleiades. You should successfully complete the steps in these instructions before doing anything more complicated. Also, it is helpful to read this page in its entirety before starting.

Back to Documentation for GEOS GCM v10

How to build GEOS GCM

Preliminary Steps

Load Build Modules

In your .bashrc or .tcshrc or other rc file add a line:

NCCS (SLES11)

module use -a /discover/swdev/gmao_SIteam/modulefiles-SLES11

NCCS (SLES12)

module use -a /discover/swdev/gmao_SIteam/modulefiles-SLES12

NAS

module use -a /nobackup/gmao_SIteam/modulefiles

GMAO Desktops

On the GMAO desktops, the SI Team modulefiles should automatically be part of running module avail but if not, they are in:

module use -a /ford1/share/gmao_SIteam/modulefiles

Also do this in any interactive window you have. This allows you to get module files needed to correctly checkout and build the model.

Now load the GEOSenv module:

module load GEOSenv

which obtains the latest git, CMake, and manage_externals modules.

Clone the Model

GEOS is now hosted on GitHub.

git clone -b v10.12.4 git@github.com:GEOS-ESM/GEOSgcm.git

Building GEOS

Single Step Building of the Model

If all you wish is to build the model, you can run parallel_build.csh from a head node. Doing so will checkout all the external repositories of the model and build it. When done, the resulting model build will be found in build/ and the installation will be found in install/ with setup scripts like gcm_setup and fvsetup in install/bin.

Develop Version of GEOS GCM

The user will notice two files in the main directory: Externals.cfg and Develop.cfg. The difference between these two is that Externals.cfg always refers to stable tested released subrepositories. The Develop.cfg points to the develop branch of @GEOSgcm_GridComp and @GEOSgcm_App. This is equivalent in the CVS days of the difference between a stable Jason-X_Y tag and the development Jason-UNSTABLE tag. In order to build the Develop.cfg version of the model with parallel_build.csh do:

parallel_build.csh -develop

Debug Version of GEOS GCM

To obtain a debug version, you can run parallel_build.csh -debug which will build with debugging flags. This will build in build-Debug/ and install into install-Debug/.

Mepo Version of GEOS GCM

GEOS GCM will soon be transitioning from using checkout_externals to using mepo, a GMAO-developed multi-repository management tool. If you wish to use it via parallel_build.csh you can run:

parallel_build.csh -mepo

along with any other flags you usually use (-develop and -debug).

Multiple Steps for Building the Model

The steps detailed below are essentially those that parallel_build.csh performs for you. Either method should yield identical builds.

Checkout externals

Using the checkout_externals command to compose the model is done by:

cd GEOSgcm
checkout_externals

Checking out develop

To use the Develop.cfg file, run:

checkout_externals -e Develop.cfg

Mepo

To checkout the full model with the mepo tool, you run:

mepo init
mepo clone

The first command initializes the multi-repository and the second one clones and assembles all the sub-repositories according to components.yaml

Checking out develop

To get development branches of GEOS GCM with mepo is different. mepo itself knows (via components.yaml) what the development branch of each subrepository is. The equivalent of Develop.cfg for mepo is to checkout the development branches of GEOSgcm_GridComp and GEOSgcm_App:

mepo develop GEOSgcm_GridComp GEOSgcm_App

This must be done after mepo clone as it is running a git command in each sub-repository.

Build the Model

Load Compiler, MPI Stack, and Baselibs

On tcsh:

source @env/g5_modules

or on bash:

source @env/g5_modules.sh

Create Build Directory

We currently do not allow in-source builds of GEOSgcm. So we must make a directory:

mkdir build

The advantages of this is that you can build both a Debug and Release version with the same clone if desired.

Run CMake

CMake generates the Makefiles needed to build the model.

cd build
cmake .. -DBASEDIR=$BASEDIR/Linux -DCMAKE_Fortran_COMPILER=ifort -DCMAKE_INSTALL_PREFIX=../install

This will install to a directory parallel to your build directory. If you prefer to install elsewhere change the path in:

-DCMAKE_INSTALL_PREFIX=<path>

and CMake will install there.

Build and Install with Make

make -j6 install

Running GEOS GCM

Passwordless Logins

First of all, to run jobs on the cluster you will need to set up passwordless ssh (which operates within the cluster, between the nodes running the job). To do so, run the following from your discover home directory:

 cat id_rsa.pub >>  authorized_keys

Similarly, transferring the daily output files (in monthly tarballs) requires passwordless authentication from discover to dirac. While in ~/.ssh on discover, run

 ssh-copy-id -i id_rsa.pub dirac

Then, log into dirac and cut and paste the contents of the id_rsa.pub file on discover into the ~/.ssh/authorized_keys file on dirac. Problems with ssh should be referred to NCCS support.

DSA Keys

Note: Due to evolution of security, it is recommended to not use DSA keys. NAS currently doesn't not allow them, and RSA and ED25519 keys are considered "better" anyway.

Setting up a model run

Once the model has built successfully, you will have an install/ directory in your checkout. To run gcm_setup go to the install/bin/ directory and run it there:

cd install/bin
./gcm_setup

The gcm_setup script asks you to provide an experiment name :

Enter the Experiment ID:

Your experiment name (later called EXPID) should be one word with no spaces, not starting with a numeral. Then the script will ask for a description:

Enter a 1-line Experiment Description:

Spaces are ok here. Next it will ask if you wish to CLONE an experiment. If yes, you can point this to another experiment and the setup script will try and duplicate all the RC, etc. files. For now, though, choose NO to create a new experiment.

Do you wish to CLONE an old experiment? (Default: NO or FALSE)

It will now ask you for the atmospheric model resolution, expecting the code for one of the displayed resolutions.

Enter the Atmospheric Horizontal Resolution code:
--------------------------------------
            Cubed-Sphere
--------------------------------------
   c48  --  2   deg
   c90  --  1   deg
   c180 -- 1/2  deg (56-km)
   c360 -- 1/4  deg (28-km)
   c720 -- 1/8  deg (14-km)
   c1440 - 1/16 deg ( 7-km)
             DYAMOND Grids
   c768 -- 1/8  deg (12-km)
   c1536 - 1/16 deg ( 6-km)
   c3072 - 1/32 deg ( 3-km)

For your first time out you will probably want to enter c48 (corresponding to ~2 degree resolution with the cubed sphere).

Next it will ask you about the vertical resolution:

Enter the Atmospheric Model Vertical Resolution: LM (Default: 72)

The next question is about using IOSERVER:

Do you wish to IOSERVER? (Default: NO or FALSE)

The "default" answer to this will change depending on the resolution you choose. For now, just accept the default.

Next is a question that asks what processor you wish to run on. For example, on discover at NCCS:

Enter the Processor Type you wish to run on:
   hasw (Haswell) (default)
   sky  (Skylake)

NOTE: At present you need access to special queues to use the Skylake, so choosing Haswell is usually a better option.

After this are questions involving the ocean model:

Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE)

Enter the Data_Ocean Horizontal Resolution code: o1 (1  -deg,  360x180  Reynolds) Default
                                                 o2 (1/4-deg, 1440x720  MERRA-2)
                                                 o3 (1/8-deg, 2880x1440 OSTIA)
                                                 CS (Cubed-Sphere OSTIA)

Then Land model:

Enter the choice of  Land Surface Boundary Conditions using: 1 (Default: Icarus), 2 (Latest Icarus-NL)

Then the aerosols:

Do you wish to run GOCART with Actual or Climatological Aerosols? (Enter: A (Default) or C)

Enter the GOCART Emission Files to use: MERRA2 (Default), PIESA, CMIP, NR, MERRA2-DD or OPS:

After this are some questions about various setups in the model. The default is often your best bet.

Enter the tag or directory (/filename) of the HISTORY.AGCM.rc.tmpl to use
(To use HISTORY.AGCM.rc.tmpl from current build, Type:  Current         )
-------------------------------------------------------------------------
Hit ENTER to use Default Tag/Location: (Current)

NOTE: Some/Many things are easier if your HOME and EXPERIMENT directories are the same. For the next two, look carefully at the default and make sure they are both pointing to the same nobackup location.

Enter Desired Location for the HOME Directory (to contain scripts and RC files)
Hit ENTER to use Default Location:
----------------------------------
Default:  /discover/nobackup/USER/EXPID

Enter Desired Location for the EXPERIMENT Directory (to contain model output and restart files)
Hit ENTER to use Default Location:
----------------------------------
Default:  ~USER/geos5/EXPID
/discover/nobackup/USER/EXPID 

Enter Location for Build directory containing:  src/ Linux/ etc...
Hit ENTER to use Default Location:
----------------------------------
Default:  /discover/nobackup/USER/GEOSgcm/install

After these it will ask you for a group ID -- the default for this writer is g0620 (GMAO modeling group). Enter whatever is appropriate, as necessary.

 Current GROUPS: g0620 gmaoint
Enter your GROUP ID for Current EXP: (Default: g0620)
-----------------------------------

The script will produce some messages and create an experiment directory (EXPDIR) in your space as /discover/nobackup/USERID/EXPID, which contains the files and sub-directories:

• AGCM.rc -- resource file with specifications of boundary conditions, initial conditions, parameters, etc.
• archive/ -- contains job script for archiving output
• CAP.rc -- resource file with run job parameters
• convert -- contains job script that converts restarts (initial condition files) from older model versions
• ExtData.rc -- sample resource file for external data, not used
• forecasts/ -- contains scripts used for data assimilation mode
• fvcore_layout.rc --
• gcm_run.j -- run script
• GEOSgcm.x -- model executable
• HISTORY.rc -- resource file specifying the fields in the model that are output as data
• plot/ -- contains plotting job script template and .rc file
• post/ -- contains the script template and .rc file for post-processing model output
• RC/ -- contains resource files for various components of the model
• regress/ -- contains scripts for doing regression testing of model
• src -- directory with a tarball of the model version's source code

The post-processing script will generate the archiving and plotting scripts as it runs. The setup script that you ran also creates an experiment home directory (HOMDIR) as either in ~USERID/geos5/EXPID (if you accepted the default) or in /discover/nobackup/USERID/EXPID (if you followed the above advice) containing the run scripts and GEOS resource (.rc) files.

Running GEOS-5

Before running the model, there is some more setup to be completed. The run scripts need some environment variables set in ~/.cshrc (regardless of which login shell you use -- the GEOS-5 scripts use csh). Here are the minimum contents of a .cshrc:

umask 022
unlimit
limit stacksize unlimited
set arch = `uname`

The umask 022 is not strictly necessary, but it will make the various files readable to others, which will facilitate data sharing and user support. Your home directory ~USERID is also inaccessible to others by default; running chmod 755 ~ is helpful.

Copy the restart (initial condition) files and associated cap_restart into EXPDIR. You can get an arbitrary set of restarts by copying the contents of the directory /discover/nobackup/mathomp4/Restarts-I30/nc4/Reynolds/c48, containing 2-degree cubed sphere restarts from April 14, 2000, and their corresponding cap_restart.

The script you submit, gcm_run.j, is in HOMEDIR. It should be ready to go as is. The parameter END_DATE in CAP.rc can be set to the date you want the run to stop. Submit the job with sbatch gcm_run.j. You can keep track of it with squeue or squeue -u USERID, or follow stdout with tail -f EXPDIR/slurm-JOBID.out, JOBID being returned by sbatch and displayed with squeue. Jobs can be killed with scancel JOBID.

Output and Plots

During a normal run, the gcm_run.j script will run the model for the segment length (current default is 15 days in model time). The model creates output files (with an nc4 extension), also called collections (of output variables), in EXPDIR/scratch directory. After each segment, the script moves the output to the EXPDIR/holding and spawns a post-processing batch job which partitions and moves the output files within the holding directory to their own distinct collection directory, which is again partitioned into the appropriate year and month. The post processing script then checks to see if a full month of data is present. If not, the post-processing job ends. If there is a full month, the script will then run the time-averaging executable to produce a monthly mean file in EXPDIR/geosgcm_*. The post-processing script then spawns a new batch job which will archive the data onto the mass-storage drives (/archive/u/USERID/GEOS5.0/EXPID).

If a monthly average file was made, the post-processing script will also check to see if it should spawn a plot job. Currently, our criteria for plotting are: 1) if the month created was February or August, AND 2) there are at least 3 monthly average files, then a plotting job for the seasons DJF or JJA will be issued. The plots are created as gifs in EXPDIR/plots_CLIM.

The post-processing script can be found in: GEOSagcm/src/GMAO_Shared/GEOS_Util/post/gcmpost.script. The nc4 output files can be opened and plotted with grads -- see http://www.iges.org/grads/gadoc/tutorial.html for a tutorial, but use sdfopen instead of open.

The contents of the output files (including which variables get saved) may be configured in the HOMEDIR/HISTORY.rc -- a good description of this file may be found at http://modelingguru.nasa.gov/clearspace/docs/DOC-1190 .

---

Back to Documentation for GEOS GCM v10

Contact Matthew Thompson at GMAO with questions and comments