Fortuna 2.5 User's Guide: Difference between revisions

(27 intermediate revisions by the same user not shown)

Line 9:

cvs co -r ''TAGNAME'' -d ''DIRECTORY'' Fortuna

where ''TAGNAME'' is the model "tag" (version). A tag in <code>cvs</code> marks the various versions of the source files in the repository that together make up a particular version of the model. A sample release tag is <code>Fortuna-~~2_5~~</code>, indicating the version Fortuna 2.5 ~~(the latest version)~~. ''DIRECTORY'' is the directory that the source code tree will be created. If you are using a stock model tag it is reasonable to name the directory the same as the tag. This directory determines which model in presumably your space a particular experiment is using. Some scripts use the environment variable <code>ESMADIR</code>, which should be set to the absolute (full) pathname of this directory.

where ''TAGNAME'' is the model "tag" (version). A tag in <code>cvs</code> marks the various versions of the source files in the repository that together make up a particular version of the model. A sample release tag is <code>Fortuna-2_5_p6</code>, indicating the latest patch of version Fortuna 2.5. ''DIRECTORY'' is the directory that the source code tree will be created. If you are using a stock model tag it is reasonable to name the directory the same as the tag. This directory determines which model in presumably your space a particular experiment is using. Some scripts use the environment variable <code>ESMADIR</code>, which should be set to the absolute (full) pathname of this directory.

When a modified version of some component of the model is saved to the repository, the tag it uses -- different from the standard model tag -- is supposed to be applied at most only to the directories with modified files. This means that if you need to use some variant tag of a gridded component, you will have to <code>cd</code> to that directory and update to the variant tag. So, for example, if you needed to apply updates to the SatSim gridded component, you would have to <code>cd</code> several levels down to the directory <code>GEOSsatsim_GridComp</code> and run

Line 18:

Once the checkout from the repository is completed, you are ready to compile. <code>cd</code> to the <code>src</code> directory at the top of the source code directory tree and from a <code>csh</code> shell run <code>source g5_modules</code>. This will load the appropriate modules and create the necessary environment for compiling and running. It is tailored to the individual systems that GEOS-5 usually runs on, so it probably won't work elsewhere. After that you can run <code>make install</code>, which will create the necessary executables in the directory <code>''ARCH''/bin</code>, where ''ARCH'' is the local architecture (most often <code>Linux</code>).

== Setting up a Global Model Run ==

The following describes how to set up a global model run. The prodcedure to set up a single column model run is described in [[Fortuna 2.5 Single Column Model]].

=== Using <code>gcm_setup</code> ===

The setup script for global runs, <code>gcm_setup</code>, is in the directory <code>src/Applications/GEOSgcm_App</code>. The following is an example of a session with the setup script, with commentary. :

<pre>

Enter the Experiment ID:

</pre>

Enter a name and hit return. For this example we'll set the experiment ID to "myexp42". Experiment IDs need to have no whitespace and not start with a digit, since it will be the prefix of job names and PBS imposes certain limits on job names.

<pre>

Enter a 1-line Experiment Description:

</pre>

This should be short but descriptive, since it will be used to label plots. It can have spaces, though the string will be stored with underscores for the spaces. Provide a description and hit return.

<pre>

Enter the Lat/Lon Horizontal Resolution: IM JM

or ..... the Cubed-Sphere Resolution: cNN

</pre>

The lat/lon option allows four resolutions: 144x91, 288x181, 576x361 or 1152x721 corresponding roughly to 2, 1, 1/2 and 1/4 degree resolutions. Enter a resolution like so:

<pre>

144 91

</pre>

and hit enter.

<pre>

Enter the Model Vertical Resolution: LM (Default: 72)

</pre>

The current standard is 72 levels, and unless you know what you are doing you should stick with that.

<pre>

Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE)

</pre>

You probably don't, so hit enter.

<pre>

Do you wish to run GOCART? (Default: NO or FALSE)

</pre>

GOCART is the interactive chemistry package, as opposed to prescribed chemistry. It incurs a significant performance cost, so unless you know you want it, you should go with the default. The following assumes that you have entered "y". Otherwise, skip two steps to "Enter the tag..."

<pre>

Enter the GOCART Emission Files to use: "CMIP" (Default), "PIESA", or "OPS":

</pre>

Select your favorite emission files here.

<pre>

Enter the AERO_PROVIDER: GOCART (Default) or PCHEM:

</pre>

Here you get to choose again to use interactive or prescribed aerosols.

<pre>

Enter the tag or directory (/filename) of the HISTORY.AGCM.rc.tmpl to use

(To use HISTORY.AGCM.rc.tmpl from current build, Type: Current )

-------------------------------------------------------------------------

Hit ENTER to use Default Tag/Location: (Current)

</pre>

This provides a default HISTORY.rc (output specification) file. The initial default will be the tag of the build in which you are running <code>gcm_setup</code>. The idea is that you can save a custom <code>HISTORY.rc</code> to the repository and have it checked out for your experiments.

<pre>

Enter Desired Location for HOME Directory (to contain scripts and RC files)

Hit ENTER to use Default Location:

----------------------------------

Default: /discover/nobackup/aeichman/myexp42

</pre>

This option determines where the experiment's home directory is located -- where the basic job scripts and major RC files (<code>AGCM.rc</code>, <code>CAP.rc</code> and <code>HISTORY.rc</code>) will be located. The first time you run the script it will default to a subdirectory under your account's home directory named <code>geos5</code>, remember what you decide (in <code>~/.HOMDIRroot</code>) and use that as a default in subsequent times the script is run. This initial default is fine, though another possibility is to enter your nobackup space, as shown here. This will place all of the HOME directory files of the experiment together with the rest of them.

<pre>

Enter Desired Location for EXPERIMENT Directory (to contain model output and restart files)

Hit ENTER to use Default Location:

----------------------------------

Default: /discover/nobackup/aeichman/myexp42

</pre>

This determines the experiment directory, where restart files and various job output is stored. These are the storage-intensive parts and so default to the <code>nobackup</code> space.

<pre>

Enter Location for Build directory containing: src/ Linux/ etc...

Hit ENTER to use Default Location:

----------------------------------

Default: /discover/nobackup/aeichman/Fortuna-2_5_p6

</pre>

This determines which of your local builds is used to create the experiment. It defaults to the build of the script you are running, which is generally a good idea.

<pre>

Current GROUPS: g0620

Enter your GROUP ID for Current EXP: (Default: g0620)

</pre>

This is used for by the job accounting system. If you are not in the default group, you will probably have been informed.

<pre>

sending incremental file list

GEOSgcm.x

sent 50848492 bytes received 31 bytes 33899015.33 bytes/sec

total size is 50842191 speedup is 1.00

Creating gcm_run.j for Experiment myexp42 ...

Creating gcm_post.j for Experiment myexp42 ...

Creating gcm_plot.tmpl for Experiment myexp42 ...

Creating gcm_archive.j for Experiment myexp42 ...

Creating gcm_regress.j for Experiment myexp42 ...

Creating AGCM.rc for Experiment myexp42 ...

Creating CAP.rc for Experiment myexp42 ...

Creating HISTORY.rc for Experiment myexp42 ...

Done!

-----

Build Directory: /discover/nobackup/aeichman/Fortuna-2_5_p6

----------------

The following executable has been placed in your Experiment Directory:

----------------------------------------------------------------------

/discover/nobackup/aeichman/Fortuna-2_5_p6/Linux/bin/GEOSgcm.x

You must now copy your Initial Conditions into:

-----------------------------------------------

/discover/nobackup/aeichman/myexp42

discover28: /discover/nobackup/aeichman/Fortuna-2_5_p6/src/Applications/GEOSgcm_App >

</pre>

And the experiment is set up. After you copy initial condition files (aka restarts) to the experiment directory, you can submit your job.

===Do not copy old experiments===

When creating related experiments, you will be tempted to copy the experiment directory tree of an older experiment. '''Do ''not'' copy old experiments, run <code>gcm_setup</code> instead.''' There are numerous instances where an experiment-specific directory is used in the run scripts created from templates by <code>gcm_setup</code> and they will wreak subtle and pervasive havoc if executed in an unexpected environment. This warning is especially true between model versions. A useful and relatively safe exception to this rule is to copy previously used examples of <code>HISTORY.rc</code>. However, you need to change the lines labeled <code>EXPID</code> and <code>EXPDSC</code> to the values in your automatically-generated <code>HISTORY.rc</code> or the plotting will fail.

=== Using restart files ===

Restart files provide the initial conditions for a run, and a set needs to be copied into a fresh experiment directory before running. This includes the file <code>cap_restart</code>, which provides the model starting date and time in text. Restart files themselves are resolution-specific and sometimes change between model versions. As of the current model version, they are flat binary files with no metadata, so they tend to be stored together with restarts of the same provinance with the date either embedded in the filename or in an accompanying <code>cap_restart</code>, typically under a directory indicating the model version.

A cleanly completed model run will leave a set of restarts and the corresponding <code>cap_restart</code> in its experiment directory. Another source is <code>/archive/u/aeichman/restarts</code>. Restarts are also left during runs in date-labeled tarballs in the <code>restarts</code> directory under the experiment directory before being transferred to the user's <code>/archive</code> space. You may have to create the <code>cap_restart</code>, which is simply one line of text with the date of the restart files in the format ''YYYYMMDD HHMMSS'' (with a space).

Failing the above sources, you can convert restarts from different resolutions and model versions, including MERRA, as described in [[Regridding Restarts for Fortuna 2.5]].

== What Happens During a Run ==

When the script <code>gcm_run.j</code> starts running, it creates a directory called <code>scratch</code> and copies or links into it the model executable, rc files, restarts and boundary conditions necessary to run the model. It also creates a directory for each of the output collections (in the default setup with the suffix <code>geosgcm_</code>) in the directory <code>holding</code> for before post-processing, and in the experiment directory for after post-processing. It also tars the restarts and moves the tarball to the <code>restarts</code> directory.

Then the executable <code>GEOSgcm.x</code> is run in the <code>scratch</code> directory, starting with the date in <code>cap_restart</code> and running for the length of a segment. A segment is the length of model time that the model integrates before returning, letting <code>gcm_run.j</code> do some housekeeping and then running another segment. A model job will typically run a number of segments before trying to resubmit itself, hopefully before the allotted wallclock time of the job runs out.

The processing that the various batch jobs perform is illustrated below:

[[Image:F2.5-job-diagram002.png]]

Each time a segment ends, <code>gcm_run.j</code> submits a post-processing job before starting a new segment or exiting. The post-processing job moves the model output from the <code>scratch</code> directory to the respective collection directory under <code>holding</code>. Then it determines whether there is a enough output to create a monthly or seasonal mean, and if so, creates them and moves them to the collection directories in the experiment directory, and then tars up the daily output and submits an archiving job. The archiving job tries to move the tarred daily output, the monthly and seasonal means and any tarred restarts to the user's space in <code>archive</code> filesystem. The post-processing script also determines (assuming the default settings) whether enough output exists to create plots; if so, a plotting job is submitted to the queue. The plotting script produces a number of pre-determined plots as <code>.gif</code> files in the <code>plot_CLIM</code> directory in the experiment directory.

As explained above, the contents of the <code>cap_restart</code> file determine the start of the model run in model time, which determines boundary conditions and the times stamps of the output. The end time may be set in <code>CAP.rc</code> with the property <code>END_DATE</code> (format ''YYYYMMDD HHMMSS'', with a space), though integration is usually leisurely enough that one can just kill the job or rename the run script <code>gcm_run.j</code> so that it is not resubmitted to the job queue.

===Tuning a run===

Most of the other properties in <code>CAP.rc</code> are discussed elsewhere, but two that are important for understanding how the batch jobs work are <code>JOB_SGMT</code>, the length of the segment, and <code>NUM_SGMT</code>, the number of segments that the job tries to run before resubmitting itself and exiting. <code>JOB_SGMT</code> is in the format of ''YYYYMMDD HHMMSS'' (but usually expressed in days) and <code>NUM_SGMT</code> as an integer, so the multiple of the two is the total model time that a job will attempt to run. It may be tempting to just run one long segment, but much housekeeping is done between segments, such as saving state in the form of restarts and spawning archiving jobs that keep your account from running over disk quota. So to tune for the maximum number of segments in a job, it is usually best to manipulate <code>JOB_SGMT</code>.

== Determining Output: <code>HISTORY.rc</code> ==

The contents of the the file <code>HISTORY.rc</code> (in your experiment <code>HOME</code> directory) tell the model what and how to output its state and diagnostic fields. The default <code>HISTORY.rc</code> provides many fields as is, but you may want to modify it to suit your needs.

===File format===

The top of a default <code>HISTORY.rc</code> will look something like this:

<pre>

EXPID: myexp42

EXPDSC: this_is_my_experiment

COLLECTIONS: 'geosgcm_prog'

'geosgcm_surf'

'geosgcm_moist'

'geosgcm_turb'

</pre>

[....]

The attribute <code>EXPID</code> must match the name of the experiment <code>HOME</code> directory; this is only an issue if you copy the <code>HISTORY.rc</code> from a different experiment. The <code>EXPDSC</code> attribute is used to label the plots. The <code>COLLECTIONS</code> attribute contains list of strings indicating the output collections to be created. The content of the individual collections are determined after this list. Individual collections can be "turned off" by commenting the relevant line with a <code>#</code>.

The following is an example of a collection specification:

<pre>

geosgcm_prog.template: '%y4%m2%d2_%h2%n2z.nc4',

geosgcm_prog.archive: '%c/Y%y4',

geosgcm_prog.format: 'CFIO',

geosgcm_prog.frequency: 060000,

geosgcm_prog.resolution: 144 91,

geosgcm_prog.vscale: 100.0,

geosgcm_prog.vunit: 'hPa',

geosgcm_prog.vvars: 'log(PLE)' , 'DYN' ,

geosgcm_prog.levels: 1000 975 950 925 900 875 850 825 800 775 750 725 700 650 600 550 500 450 400 350 300 250 200 150 100 70 50 40 30 20 10 7 5 4 3 2 1 0.7 0.5 0.4 0.3 0.2

0.1 0.07 0.05 0.04 0.03 0.02,

geosgcm_prog.fields: 'PHIS' , 'AGCM' ,

'T' , 'DYN' ,

'PS' , 'DYN' ,

'ZLE' , 'DYN' , 'H' ,

'OMEGA' , 'DYN' ,

'Q' , 'MOIST' , 'QV' ,

::

</pre>

The individual collection attributes are described below, but what users modify the most are the <code>fields</code> attribute. This determines which exports are saved in the collection. Each field record is a string with the name of an export from the model followed by a string with the name of the gridded component which exports it, separated by a comma. The entries with a third column determine the name by which that export in saved in the collection file when the name is different from that of the export.

===What exports are available?===

To add export fields to the <code>HISTORY.rc</code> you will need to know what fields the model provides, which gridded component provides them, and their name. The most straightforward way to do this is to use <code>PRINTSPEC</code>. The setting for <code>PRINTSPEC</code> is in the file <code>CAP.rc</code>. By default the line looks like so:

PRINTSPEC: 0 # (0: OFF, 1: IMPORT & EXPORT, 2: IMPORT, 3: EXPORT)

Setting <code>PRINTSPEC</code> to 3 will make the model send to standard output a list of exports available to <code>HISTORY.rc</code> in the model's current configuration, and then exit without integrating. The list includes each export's gridded component and short name (both necessary to include in <code>HISTORY.rc</code>), long (descriptive) name, units, and number of dimensions. Note that run-time options can affect the exports available, so see to it that you have those set as you intend. The other <code>PRINTSPEC</code> values are useful for debugging.

While you can set <code>PRINTSPEC</code>, submit <code>qsub gcm_run.j</code>, and get the export list as part of PBS standard output, there are quicker ways of obtaining the list. One way is to run it as a single column model on a single processor, as explained in [[Fortuna 2.5 Single Column Model]]. Another way is to run it in an existing experiment. In the <code>scratch</code> directory of an experiment that has already run, change <code>PRINTSPEC</code> in <code>CAP.rc</code> as above. Then, in the file <code>AGCM.rc</code>, change the values of <code>NX</code> and <code>NY</code> (near the beginning of the file) to 1. Then, from an interactive job (one processor will suffice), run the executable <code>GEOSgcm.x</code> in <code>scratch</code>. You will need to run <code>source src/g5_modules</code> in the model's build tree to set up the environment. The model executable will simply output the export list to <code>stdout</code>.

== Special Requirements ==

=== Perpetual ("Groundhog Day") mode ===

GEOS-5 Fortuna 2.5 and later can be run in "perpetual mode", automatically running with the same forcings for a time period delineated as a calendar year, month or day. The time period desired is set in <code>CAP.rc</code> with the parameters <code>PERPETUAL_YEAR</code>, <code>PERPETUAL_MONTH</code> and <code>PERPETUAL_DAY</code>. Set all three to run with the forcings for a particular day, and <code>NUM_SGMT</code> to how many times you wish to run it -- the history collection files will be appended with dates starting with the one in <code>cap_restart</code> and generally incrementing for the number of days in <code>NUM_SGMT</code>.

=== Saving restarts during a segment ===

=== post.rc ===

@@ Line 9: / Line 9: @@
    cvs co -r  ''TAGNAME'' -d ''DIRECTORY'' Fortuna
-where ''TAGNAME'' is the model "tag" (version).  A tag in <code>cvs</code> marks the various versions of the source files in the repository that together make up a particular version of the model.  A sample release tag is <code>Fortuna-2_5</code>, indicating the version Fortuna 2.5 (the latest version). ''DIRECTORY'' is the directory that the source code tree will be created.  If you are using a stock model tag it is reasonable to name the directory the same as the tag.  This directory determines which model in presumably your space a particular experiment is using.  Some scripts use the environment variable <code>ESMADIR</code>, which should be set to the absolute (full) pathname of this directory.
+where ''TAGNAME'' is the model "tag" (version).  A tag in <code>cvs</code> marks the various versions of the source files in the repository that together make up a particular version of the model.  A sample release tag is <code>Fortuna-2_5_p6</code>, indicating the latest patch of version Fortuna 2.5. ''DIRECTORY'' is the directory that the source code tree will be created.  If you are using a stock model tag it is reasonable to name the directory the same as the tag.  This directory determines which model in presumably your space a particular experiment is using.  Some scripts use the environment variable <code>ESMADIR</code>, which should be set to the absolute (full) pathname of this directory.
 When a modified version of some component of the model is saved to the repository, the tag it uses -- different from the standard model tag -- is supposed to be applied at most only to the directories with modified files.  This means that if you need to use some variant tag of a gridded component, you will have to <code>cd</code> to that directory and update to the variant tag.  So, for example, if you needed to apply updates to the SatSim gridded component, you would have to <code>cd</code> several levels down to the directory <code>GEOSsatsim_GridComp</code> and run
@@ Line 18: / Line 18: @@
 Once the checkout from the repository is completed, you are ready to compile.  <code>cd</code> to the <code>src</code> directory at the top of the source code directory tree and from a <code>csh</code> shell run <code>source g5_modules</code>.  This will load the appropriate modules and create the necessary environment for compiling and running.  It is tailored to the individual systems that GEOS-5 usually runs on, so it probably won't work elsewhere.  After that you can run <code>make install</code>, which will create the necessary executables in the directory <code>''ARCH''/bin</code>, where ''ARCH'' is the local architecture (most often <code>Linux</code>).
+== Setting up a Global Model Run ==
+The following describes how to set up a global model run.  The prodcedure to set up a single column model run is described in [[Fortuna 2.5 Single Column Model]].
+=== Using <code>gcm_setup</code> ===
+The setup script for global runs, <code>gcm_setup</code>, is in the directory <code>src/Applications/GEOSgcm_App</code>.  The following is an example of a session with the setup script, with commentary.  :
+<pre>
+ Enter the Experiment ID:
+</pre>
+Enter a name and hit return.  For this example we'll set the experiment ID to "myexp42".  Experiment IDs need to have no whitespace and not start with a digit, since it will be the prefix of job names and PBS imposes certain limits on job names.
+<pre>
+Enter a 1-line Experiment Description:
+</pre>
+This should be short but descriptive, since it will be used to label plots.  It can have spaces, though the string will be stored with underscores for the spaces.  Provide a description and hit return.
+<pre>
+Enter the Lat/Lon Horizontal Resolution: IM JM
+   or ..... the Cubed-Sphere Resolution: cNN
+</pre>
+The lat/lon option allows four resolutions: 144x91, 288x181, 576x361 or 1152x721 corresponding roughly to 2, 1, 1/2 and 1/4 degree resolutions.  Enter a resolution like so:
+<pre>
+91
+</pre>
+and hit enter.
+<pre>
+Enter the Model Vertical Resolution: LM (Default: 72)
+</pre>
+The current standard is 72 levels, and unless you know what you are doing you should stick with that.
+<pre>
+Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE)
+</pre>
+You probably don't, so hit enter.
+<pre>
+Do you wish to run GOCART? (Default: NO or FALSE)
+</pre>
+GOCART is the interactive chemistry package, as opposed to prescribed chemistry.  It incurs a significant performance cost, so unless you know you want it, you should go with the default.  The following assumes that you have entered "y".  Otherwise, skip two steps to "Enter the tag..."
+<pre>
+Enter the GOCART Emission Files to use: "CMIP" (Default), "PIESA", or "OPS":
+</pre>
+Select your favorite emission files here.
+<pre>
+Enter the AERO_PROVIDER: GOCART (Default) or PCHEM:
+</pre>
+Here you get to choose again to use interactive or prescribed aerosols.
+<pre>
+Enter the tag or directory (/filename) of the HISTORY.AGCM.rc.tmpl to use
+(To use HISTORY.AGCM.rc.tmpl from current build, Type:  Current         )
+-------------------------------------------------------------------------
+Hit ENTER to use Default Tag/Location: (Current)
+</pre>
+This provides a default HISTORY.rc (output specification) file.  The initial default will be the tag of the build in which you are running <code>gcm_setup</code>.  The idea is that you can save a custom <code>HISTORY.rc</code> to the repository and have it checked out for your experiments.
+<pre>
+Enter Desired Location for HOME Directory (to contain scripts and RC files)
+Hit ENTER to use Default Location:
+----------------------------------
+Default:  /discover/nobackup/aeichman/myexp42
+</pre>
+This option determines where the experiment's home directory is located -- where the basic job scripts and major RC files (<code>AGCM.rc</code>, <code>CAP.rc</code> and <code>HISTORY.rc</code>) will be located.  The first time you run the script it will default to a subdirectory under your account's home directory named <code>geos5</code>, remember what you decide (in <code>~/.HOMDIRroot</code>) and use that as a default in subsequent times the script is run.  This initial default is fine, though another possibility is to enter your nobackup space, as shown here.  This will place all of the HOME directory files of the experiment together with the rest of them.
+<pre>
+Enter Desired Location for EXPERIMENT Directory (to contain model output and restart files)
+Hit ENTER to use Default Location:
+----------------------------------
+Default:  /discover/nobackup/aeichman/myexp42
+</pre>
+This determines the experiment directory, where restart files and various job output is stored.  These are the storage-intensive parts and so default to the <code>nobackup</code> space.
+<pre>
+Enter Location for Build directory containing:  src/ Linux/ etc...
+Hit ENTER to use Default Location:
+----------------------------------
+Default:  /discover/nobackup/aeichman/Fortuna-2_5_p6
+</pre>
+This determines which of your local builds is used to create the experiment.  It defaults to the build of the script you are running, which is generally a good idea.
+<pre>
+Current GROUPS: g0620
+Enter your GROUP ID for Current EXP: (Default: g0620)
+</pre>
+This is used for by the job accounting system.  If you are not in the default group, you will probably have been informed.
+<pre>
+sending incremental file list
+GEOSgcm.x
+sent 50848492 bytes  received 31 bytes  33899015.33 bytes/sec
+total size is 50842191  speedup is 1.00
+Creating gcm_run.j for Experiment myexp42 ...
+Creating gcm_post.j for Experiment myexp42 ...
+Creating gcm_plot.tmpl for Experiment myexp42 ...
+Creating gcm_archive.j for Experiment myexp42 ...
+Creating gcm_regress.j for Experiment myexp42 ...
+Creating AGCM.rc for Experiment myexp42 ...
+Creating CAP.rc for Experiment myexp42 ...
+Creating HISTORY.rc for Experiment myexp42 ...
+Done!
+-----
+Build Directory: /discover/nobackup/aeichman/Fortuna-2_5_p6
+----------------
+The following executable has been placed in your Experiment Directory:
+----------------------------------------------------------------------
+/discover/nobackup/aeichman/Fortuna-2_5_p6/Linux/bin/GEOSgcm.x
+You must now copy your Initial Conditions into:
+-----------------------------------------------
+/discover/nobackup/aeichman/myexp42
+discover28: /discover/nobackup/aeichman/Fortuna-2_5_p6/src/Applications/GEOSgcm_App >
+</pre>
+And the experiment is set up.  After you copy initial condition files (aka restarts) to the experiment directory, you can submit your job.
+===Do not copy old experiments===
+When creating related experiments, you will be tempted to copy the experiment directory tree of an older experiment.  '''Do ''not'' copy old experiments, run <code>gcm_setup</code> instead.'''  There are numerous instances where an experiment-specific directory is used in the run scripts created from templates by <code>gcm_setup</code>  and they will wreak subtle and pervasive havoc if executed in an unexpected environment.  This warning is especially true between model versions.  A useful and relatively safe exception to this rule is to copy previously used examples of <code>HISTORY.rc</code>.  However, you need to change the lines labeled <code>EXPID</code> and <code>EXPDSC</code> to the values in your automatically-generated  <code>HISTORY.rc</code> or the plotting will fail.
+=== Using restart files ===
+Restart files provide the initial conditions for a run, and a set needs to be copied into a fresh experiment directory before running.  This includes the file <code>cap_restart</code>, which provides the model starting date and time in text.  Restart files themselves are resolution-specific and sometimes change between model versions.  As of the current model version, they are flat binary files with no metadata, so they tend to be stored together with restarts of the same provinance with the date either embedded in the filename or in an accompanying <code>cap_restart</code>, typically under a directory indicating the model version.
+A cleanly completed model run will leave a set of restarts and the corresponding <code>cap_restart</code> in its experiment directory.  Another source is <code>/archive/u/aeichman/restarts</code>.  Restarts are also left during runs in date-labeled tarballs in the <code>restarts</code> directory under the experiment directory before being transferred to the user's <code>/archive</code> space.  You may have to create the <code>cap_restart</code>, which is simply one line of text with the date of the restart files in the format ''YYYYMMDD HHMMSS'' (with a space).
+Failing the above sources, you can convert restarts from different resolutions and model versions, including MERRA, as described in  [[Regridding Restarts for Fortuna 2.5]].
+== What Happens During a Run ==
+When the script <code>gcm_run.j</code> starts running, it creates a directory called  <code>scratch</code> and copies or links into it the model executable, rc files, restarts and boundary conditions necessary to run the model.  It also creates a directory for each of the output collections (in the default setup with the suffix <code>geosgcm_</code>) in the directory <code>holding</code> for before post-processing, and in the experiment directory for after post-processing.  It also tars the restarts and moves the tarball to the <code>restarts</code> directory.
+Then the executable  <code>GEOSgcm.x</code> is run in the <code>scratch</code> directory, starting with the date in  <code>cap_restart</code> and running for the length of a segment.  A segment is the length of model time that the model integrates before returning, letting <code>gcm_run.j</code> do some housekeeping and then running another segment.  A model job will typically run a number of segments before trying to resubmit itself, hopefully before the allotted wallclock time of the job runs out.
+The processing that the various batch jobs perform is illustrated below:
+[[Image:F2.5-job-diagram002.png]]
+Each time a segment ends, <code>gcm_run.j</code> submits a post-processing job before starting a new segment or exiting.  The post-processing job moves the model output from the  <code>scratch</code> directory to the respective collection directory under  <code>holding</code>.  Then it determines whether there is a enough output to create a monthly or seasonal mean, and if so, creates them and moves them to the collection directories in the experiment directory, and then tars up the daily output and submits an archiving job.  The archiving job tries to move the tarred daily output, the monthly and seasonal means and any tarred restarts to the user's space in <code>archive</code> filesystem.  The post-processing script also determines (assuming the default settings) whether enough output exists to create plots; if so, a plotting job is submitted to the queue.  The plotting script produces a number of pre-determined plots as <code>.gif</code> files in the <code>plot_CLIM</code> directory in the experiment directory.
+As explained above, the contents of the <code>cap_restart</code> file determine the start of the model run in model time, which determines boundary conditions and the times stamps of the output.  The end time may be set in <code>CAP.rc</code> with the property <code>END_DATE</code>  (format ''YYYYMMDD HHMMSS'', with a space), though integration is usually leisurely enough that one can just kill the job or rename the run script <code>gcm_run.j</code> so that it is not resubmitted to the job queue.
+===Tuning a run===
+Most of the other properties in <code>CAP.rc</code> are discussed elsewhere, but two that are important for understanding how the batch jobs work are <code>JOB_SGMT</code>, the length of the segment, and <code>NUM_SGMT</code>, the number of segments that the job tries to run before resubmitting itself and exiting.  <code>JOB_SGMT</code> is in the format of ''YYYYMMDD HHMMSS'' (but usually expressed in days) and <code>NUM_SGMT</code> as an integer, so the multiple of the two is the total model time that a job will attempt to run.  It may be tempting to just run one long segment, but much housekeeping is done between segments, such as saving state in the form of restarts and spawning archiving jobs that keep your account from running over disk quota.  So to tune for the maximum number of segments in a job, it is usually best to manipulate <code>JOB_SGMT</code>.
+== Determining Output: <code>HISTORY.rc</code> ==
+The contents of the the file <code>HISTORY.rc</code> (in your experiment <code>HOME</code> directory) tell the model what and how to output its state and diagnostic fields.  The default <code>HISTORY.rc</code> provides many fields as is, but you may want to modify it to suit your needs.
+===File format===
+The top of a default <code>HISTORY.rc</code> will look something like this:
+<pre>
+EXPID:  myexp42
+EXPDSC: this_is_my_experiment
+COLLECTIONS: 'geosgcm_prog'
+             'geosgcm_surf'
+             'geosgcm_moist'
+             'geosgcm_turb'
+</pre>
+[....]
+The attribute <code>EXPID</code> must match the name of the experiment <code>HOME</code> directory; this is only an issue if you copy the  <code>HISTORY.rc</code> from a different experiment.  The <code>EXPDSC</code> attribute is used to label the plots.  The <code>COLLECTIONS</code> attribute contains list of strings indicating the output collections to be created.  The content of the individual collections are determined after this list.  Individual collections can be "turned off" by commenting the relevant line with a <code>#</code>.
+The following is an example of a collection specification:
+<pre>
+  geosgcm_prog.template:  '%y4%m2%d2_%h2%n2z.nc4',
+  geosgcm_prog.archive:   '%c/Y%y4',
+  geosgcm_prog.format:    'CFIO',
+  geosgcm_prog.frequency:  060000,
+  geosgcm_prog.resolution: 144 91,
+  geosgcm_prog.vscale:     100.0,
+  geosgcm_prog.vunit:     'hPa',
+  geosgcm_prog.vvars:     'log(PLE)' , 'DYN'          ,
+  geosgcm_prog.levels:     1000 975 950 925 900 875 850 825 800 775 750 725 700 650 600 550 500 450 400 350 300 250 200 150 100 70 50 40 30 20 10 7 5 4 3 2 1 0.7 0.5 0.4 0.3 0.2
+.1 0.07 0.05 0.04 0.03 0.02,
+  geosgcm_prog.fields:    'PHIS'     , 'AGCM'         ,
+                          'T'        , 'DYN'          ,
+                          'PS'       , 'DYN'          ,
+                          'ZLE'      , 'DYN'          , 'H'   ,
+                          'OMEGA'    , 'DYN'          ,
+                          'Q'        , 'MOIST'        , 'QV'  ,
+                          ::
+</pre>
+The individual collection attributes are described below, but what users modify the most are the <code>fields</code> attribute.  This determines which exports are saved in the collection.  Each field record is a string with the name of an export from the model followed by a string with the name of the gridded component which exports it, separated by a comma.  The entries with a third column determine the name by which that export in saved in the collection file when the name is different from that of the export.
+===What exports are available?===
+To add export fields to the <code>HISTORY.rc</code> you will need to know what fields the model provides, which gridded component provides them, and their name.  The most straightforward way to do this is to use <code>PRINTSPEC</code>.  The setting for  <code>PRINTSPEC</code> is in the file <code>CAP.rc</code>.  By default the line looks like so:
+ PRINTSPEC: 0  # (0: OFF, 1: IMPORT & EXPORT, 2: IMPORT, 3: EXPORT)
+Setting <code>PRINTSPEC</code> to  3 will make the model send to standard output a list of exports available to <code>HISTORY.rc</code> in the model's current configuration, and then exit without integrating. The list includes each export's gridded component and short name (both necessary to include in <code>HISTORY.rc</code>), long (descriptive) name, units, and number of dimensions.  Note that run-time options can affect the exports available, so see to it that you have those set as you intend.  The other <code>PRINTSPEC</code> values are useful for debugging.
+While you can set  <code>PRINTSPEC</code>, submit <code>qsub gcm_run.j</code>, and get the export list as part of PBS standard output, there are quicker ways of obtaining the list.  One way is to run it as a single column model on a single processor, as explained in [[Fortuna 2.5 Single Column Model]].  Another way is to run it in an existing experiment.  In the <code>scratch</code> directory of an experiment that has already run, change <code>PRINTSPEC</code> in  <code>CAP.rc</code> as above.  Then, in the file <code>AGCM.rc</code>, change the values of <code>NX</code> and <code>NY</code> (near the beginning of the file) to 1.  Then, from an interactive job (one processor will suffice), run the executable <code>GEOSgcm.x</code> in <code>scratch</code>.  You will need to run <code>source src/g5_modules</code> in the model's build tree to set up the environment.  The model executable will simply output the export list to <code>stdout</code>.
+== Special Requirements ==
+=== Perpetual ("Groundhog Day") mode  ===
+GEOS-5 Fortuna 2.5 and later can be run in "perpetual mode", automatically running with the same forcings for a time period delineated as a calendar year, month or day.  The time period desired is set in <code>CAP.rc</code> with the parameters <code>PERPETUAL_YEAR</code>, <code>PERPETUAL_MONTH</code> and <code>PERPETUAL_DAY</code>.  Set all three to run with the forcings for a particular day, and <code>NUM_SGMT</code> to how many times you wish to run it -- the history collection files will be appended with dates starting with the one in <code>cap_restart</code> and generally incrementing for the number of days in <code>NUM_SGMT</code>.
+=== Saving restarts during a segment ===
+=== post.rc ===