GEOS-5 Checkout and Build Instructions (Fortuna)
These instructions presume checking out the PIESA group tag.
How to Check Out and Build the Code
Find a place to store and build the code
The GEOS-5 (AGCM) source code checks out at about 200 MB of space. Once compiled, the complete package is about 1.3 GB. Your home space on discover may not be sufficient for checking out and building the code. You should consider either 1) requesting a larger quota in your home space (call the tag x6-9120 and ask, telling them you are doing GEOS-5 development work) or 2) building in your (larger) nobackup space (recommended). But consider, nobackup is not backed up. So be careful...
One strategy I like is to check the code out to my nobackup space, but then make symlink from him home space back to that. For example, if I have my code stored at $NOBACKUP/GEOSagcm, I would make a symlink in my home space to point to that like:
% ln -s $NOBACKUP/GEOSagcm GEOSagcm
Setup your environment and check out code
Let's get ourselves ready to check out the code. We'll be using the cvs command to check out the code. The basic syntax is:
% cvs -d $CVSROOT checkout -d DIRECTORY -r TAGNAME MODULENAME
Here, $CVSROOT specifies the CVS repository we'll be getting the code from, DIRECTORY is the name of the directory you would like to create to hold the code, MODULENAME is the particular module (set of code) we'll be checking out, and TAGNAME is a particular version of that module. Let's fill in the blanks:
% cvs -d :ext:pcolarco@progressdirect:/cvsroot/esma co -d GEOSagcm -r piesa-Fortuna-2_1_p1-m1 Fortuna
So our module is Fortuna and the tag is piesa-Fortuna-2_1_p1-m1. The code will check-out to a newly created directory call GEOSagcm. Note that I substituted the shortcut co for checkout in the above command.
The above command is generally valid. You ought to be able to execute it and checkout some code. If you don't have your ssh keys setup on progress then you should be prompted for your progress password. The assumption here is that your username on progress is the same as on the machine you are checking the code out on.
Here's a short cut. So that you don't have to type in the -d :ext:pcolarco@progressdirect:/cvsroot/esma business all the time, you can add the following lines to your, e.g., .cshrc file:
setenv CVSROOT ':ext:pcolarco@progressdirect:/cvsroot/esma' setenv CVS_RSH ssh
Modify as appropriate to put in your username in or if you use a different shell (i.e., put the analog of these lines into your .bashrc file or whatever). Or if you are on a different machine than discover, the path to the progress repository may be somewhat different.
If you set that up, you should be able now to type in:
% cvs co -d GEOSagcm -r piesa-Fortuna-2_1_p1-m1 Fortuna
and the code will check out.
Setup your environment and build the code
Now you've checked out the code. Your should have a directory called GEOSagcm in front of you. You're almost ready to build the code at this point.
The first thing to do is to set up your shell environment. First, we need to define the environment variable ESMADIR, e.g.,
% setenv ESMADIR $NOBACKUP/GEOSagcm
assuming you put the source code in your $NOBACKUP directory.
Then:
% cd GEOSagcm/src % source g5_modules
This loads the relevant "modules" for this version of the model. The "modules" specify the version of compiler, MPI libraries, and scientific libraries assumed by the code. This step also sets the environment variable BASEDIR, which specifies the version of the so-called "Baselibs" the code requires. The "Baselibs" contain, e.g., the netcdf and ESMF libraries.
By the way, I have an alias in my .cshrc file that actually accomplishes the above tasks. It looks like this:
% alias g5 'setenv ESMADIR $HOME/GEOSagcm;cd $ESMADIR/src;source g5_modules'
Now, assuming you're in the source directory, you can build the issuing the following command:
% gmake install
Note: If you want to use the special PIESA resource files build like this:
% gmake RCDIR=PIESA install
If you do that, go away and take a coffee break. A long one. This may take an hour or more to build. There are a couple of ways to speed this process up. One way is to build the code without optimization:
% gmake install FOPT=-g
The code builds faster in this instance, but be warned that without optimization any generated code will run very slowly.
A better way is to do a parallel build. To do this, start an interactive queue (on discover):
% qsub -I -W group_list=g0604 -N g5Debug -l select=2:ncpus=4,walltime=03:00:00 -S /bin/tcsh -V -j eo
Note that the string following "group_list=" is your group-id code. It's the project that gets charged for the computer time you use. If you're not on "g0604" that's okay, the queue system will let you know and it won't start your job. To find out which group you belong to, issue the following command:
% getsponsor
and you'll get a table of sponsor codes available to you. Enter one of those codes as the group_list string and try again.
Wait, what have we done here? We've started an interactive queue (interactive in the sense that you have a command line) where we've now go 8 cpus allocated to us (and us alone!) for the next 3 hours. We can use all those 8 cpus to speed up our build as follows:
% gmake --jobs=8 pinstall
The syntax here is that "--jobs=" specifies the number of cpus to use (up to the 8 we've requested in our queue) and "pinstall" means to do a parallel install. Don't worry, the result should be the same as "gmake install" above but take a fraction of the time.
What if something goes wrong? Sometimes the build just doesn't go right. It's useful to save the output that scrolls by on the screen to a file so you can analyze it later. Modify any of the build examples above as follows to capture the text to a file called "make.log":
% gmake --jobs=8 pinstall |& tee make.log
and now you have a record of how the build progressed. When the build completes (successfully or otherwise) you can analyze the build results by issuing the following command:
% Config/gmh.pl -v make.log
and you'll be given a list of what compiled and didn't, which will hopefull allow you to go in and find any problems. If there are problems indicated the first pass through, try the build step again and see if they are cleared up.
If all goes well, you should have a brand-new build of GEOS-5. Step back up out of the src directory you should see the following sub-directories:
Config CVS Linux src
In the Linux directory you'll find:
bin Config doc etc include lib
The executables are in the bin directory.
In this example, the directory GEOSagcm is the root directory everything ends up under. You can specify another location by setting the environment variable ESMADIR to some other location and installing again.
How to Setup and Run and Experiment
Now that you've built the code, let's try to run it. In the exercise that follows, we will create a fresh experiment.
In what follows I will assume we are working on the NCCS computer discover.
Before we get going, let's make some light edits to your .cshrc file. First, near the top of your .cshrc file add the word:
unlimit
We're not sure what this means, but Arlindo says it is important.
Let's make sure that your binaries from your compiled GEOS-5 code are in your path. Include the following line somewhere in your .cshrc file:
setenv PATH .:$NOBACKUP/GEOSagcm/Linux/bin:$PATH
where obviously you replace the particular path to the binaries with your path. Note what this implies: it won't be a good idea to move or clean this directory while the model is running!
Setup Experiment using gcm_setup
From your model build src directory, go into Applications/GEOSgcm_App. Then run gcm_setup.
% cd Applications/GEOSgcm_App % ./gcm_setup
Your will be prompted to answer some questions:
experiment ID 1-line description of the experiment model resolution (IM JM) aero provider (PCHEM or GOCART) HOME directory EXP directory group ID
Upon completion, you will have two directories, the HOMEDIR which is in your home space (something like ~/geos5/EXPID) and the EXPDIR which is in your nobackup space.
Go to the HOMEDIR. You need to edit a couple of files to get the experiment to run the way you want. e.g., edit AGCM.rc to setup model run time parameters e.g., edit HISTORY.rc to setup model history output e.g., edit CAP.rc to control the model run e.g., edit gcm_run.j to modify the run script
For example, my AGCM.rc file has the following lines characteristic (to run GOCART with climatological aerosol forcing):
GOCART_INTERNAL_RESTART_FILE: gocart_internal_rst GOCART_INTERNAL_RESTART_TYPE: binary GOCART_INTERNAL_CHECKPOINT_FILE: gocart_internal_checkpoint GOCART_INTERNAL_CHECKPOINT_TYPE: binary AEROCLIM: ExtData/AeroCom/L72/aero_clm/gfedv2.aero.eta.%y4%m2clm.nc AEROCLIMDEL: ExtData/AeroCom/L72/aero_clm/gfedv2.del_aero.eta.%y4%m2clm.nc AEROCLIMYEAR: 2002 DIURNAL_BIOMASS_BURNING: no RATS_PROVIDER: PCHEM # options: PCHEM, GMICHEM, STRATCHEM (Radiatively active tracers) AERO_PROVIDER: PCHEM # options: PCHEM, GOCART (Radiatively active aerosols)
To run with interactive aerosol forcing, modify the appropriate lines above to look like:
#AEROCLIM: ExtData/AeroCom/L72/aero_clm/gfedv2.aero.eta.%y4%m2clm.nc #AEROCLIMDEL: ExtData/AeroCom/L72/aero_clm/gfedv2.del_aero.eta.%y4%m2clm.nc #AEROCLIMYEAR: 2002 AERO_PROVIDER: GOCART # options: PCHEM, GOCART (Radiatively active aerosols)
--I do this: copy gcm_run.j -> gcm_run.1 (you want to edit the END_DATE and STOP_DATE)
Now create a file called "cap_restart" which has contents of the model start date in the form YYYYMMDD HHMMSS
Copy cap_restart to your EXPDIR
% cp cap_restart /discover/nobackup/USERNAME/EXPID
Now go to your EXPDIR You might want to edit the resource files in the RC sub-directory (e.g., Chem_Registry.rc to turn things on/off) You also need some restarts to run the model. For Fortuna at "b" resolution you can copy a set (valid nominally at 20071001) from /discover/nobackup/pcolarco/restart0/fortuna/b72. You can just copy these files into your experiment directory, but what I like to is make a "restart0" directory in my EXPDIR and copy the files there, and then copy back up to the EXPDIR.
At this point you should be good to go. Go back to your HOMDIR and qsub gcm_run.1
Populate the Experiment Directory
At the beginning of the experiment, the experiment directory will contain a few things:
- Resource Files
- Restart Files
- The GEOSgcm.x executable (the model itself)
I've provided a sample experiment directory (/discover/nobackup/colarco/dragnet). There are a number of files that end in ".rc". These are the resource files. Copy these to your experiment directory.
I've prepared a set of restarts based on a previous GEOS-5 run that will serve for this example. You can find these files in /discover/nobackup/colarco/restart0/b72/. Copy all the "*rst" files to your experiment directory, e.g.,
% cp /discover/nobackup/colarco/restart0/b72/*rst /discover/nobackup/colarco/dragnet
Finally, copy your GEOSgcm.x executable from wherever you built the model to your experiment directory, e.g.,
% cp /discover/nobackup/colarco/GEOSagcm/Linux/bin/GEOSgcm.x /discover/nobackup/colarco/dragnet
There's one other file you need: in my experiment directory find a file called cap_restart. Copy this over to your experiment directory. Take a look at this file. It has one line with two numbers:
20021001 210000
These are the YYYYMMDD and HHMMSS timestamp I am using to start the model run. Note that I choose these dates to correspond to the date at which the restarts I gave you are valid.
I haven't told you anything much about the resource files. Suffice to say for now that this group I am providing here will differ from the resource files in your source code collection. I've modified these to run aerosols and use aerosol source files that will work for the time period we will simulate.
Populate the Home Directory
Now go into your home directory and copy the contents of my sample into yours:
% cd /home/USERNAME/geos5/YOUR_EXPID % cp /home/colarco/geos5/dragnet/* .
There are several files here:
AGCM.tmpl CAP.tmpl HISTORY.tmpl gcm_run.j gcm_post.j gcm_post.script gcm_regress.j
For the moment, we don't need to do anything to AGCM.tmpl or CAP.tmpl. They are filled in with information from the run script and written as AGCM.rc and CAP.rc in your experiment directory. (As a rule, if you have AGCM.rc and CAP.rc and HISTORY.rc in your experiment directory, you might want to remove them before starting an experiment so that you know you are using the *.tmpl files from your home directory.)
We have to make several edits to the remaining files. In particular, we need to change the group ID code used to charge your jobs to PBS, we need to change the experiment ID to match the ID you've chosen for your experiment, and we need to modify various path variables to point to your user ID (not mine!). You can accomplish these changes by using grep on the files. For example, look for instances of my group ID r0605:
% grep r0605 *
on these files find the following:
gcm_post.j:#PBS -W group_list=r0605 gcm_regress.j:#PBS -W group_list=r0605 gcm_run.j:#PBS -W group_list=r0605
Unless you have permission to charge to r0605 you need to change these numbers. Use your favorite editor to do this. Recall, to find your group ID, use getsponsor.
Here's the result of grepping on my username:
% grep colarco *
AGCM.tmpl:#REPLAY_FILE: /discover/nobackup/colarco/replay_b/winds/b5_merrasc_jan92.ana.eta.%y4%m2%d2_%h2z.hdf
gcm_post.j:setenv GEOSUTIL /home/colarco/GEOS_Util
gcm_post.j:setenv SOURCE /discover/nobackup/colarco/$EXPID
gcm_post.j:setenv ARCHIVE /archive/u/colarco/GEOS5.0/$EXPID
gcm_post.j:setenv HOMDIR /home/colarco/geos5/$EXPID
gcm_regress.j:setenv GEOSUTIL /home/colarco/GEOS_Util
gcm_regress.j:setenv EXPDIR /discover/nobackup/colarco/$EXPID
gcm_regress.j:setenv HOMDIR /home/colarco/geos5/$EXPID
gcm_run.j:setenv GEOSUTIL /home/colarco/GEOS_Util
gcm_run.j:setenv EXPDIR /discover/nobackup/colarco/$EXPID
gcm_run.j:setenv HOMDIR /home/colarco/geos5/$EXPID
gcm_run.j:/bin/ln -s /discover/nobackup/colarco/SST/dataoceanfile_MERRA_sst_1971-current.${OGCM_IM}x${OGCM_JM}.LE sst.data
gcm_run.j:/bin/ln -s /discover/nobackup/colarco/SST/dataoceanfile_MERRA_ice_temperature_1971-current.${OGCM_IM}x${OGCM_JM}.LE sstsi.data
gcm_run.j:/bin/ln -s /discover/nobackup/colarco/SST/dataoceanfile_MERRA_fraci_1971-current.${OGCM_IM}x${OGCM_JM}.LE fraci.data
Ignore the entry in AGCM.tmpl for now, it's been commented out. Ignore also the last three entries (pertaining to the location of the SST files); for whatever reason there were incompatible SST files on discover and these were copied by hand over from palm. For now, we'll use these. All the other instances of username "colarco" should be modified.
Finally, here's the results of grepping on my experiment ID:
% grep dragnet * gcm_post.j:setenv EXPID dragnet gcm_regress.j:#PBS -N dragnet gcm_regress.j:setenv EXPID dragnet gcm_run.j:#PBS -N dragnet gcm_run.j:setenv EXPID dragnet HISTORY.tmpl:EXPID: dragnet
Your should change all instances of dragnet here to reflect your user ID. Note that lines that look like #PBS -N dragnet are really just the name of the experiment that PBS queueing system reports. This does not have to replicate your experiment ID at all; it is merely a string that helps you identify your job running in PBS.
Run the Simulation
At this point we're ready to take a crack at running the model. Go into your experiment home directory (e.g., /home/colarco/geos5/dragnet). You can start the model job by issuing:
% qsub gcm_run.j
And check the progress by issuing:
% qstat | grep YOUR_USERNAME