Heracles 4.0 User's Guide: Difference between revisions

(11 intermediate revisions by the same user not shown)

Line 7:

Most of the time for longer runs you will be using a release version of the model, perhaps compiled with a different version of one or more of the model's gridded components, as defined by subdirectories in the source code. This process starts with checking out the stock model from the repository using the command

cvs co -r ''TAGNAME'' -d ''DIRECTORY'' ~~Heracles~~ GEOSagcm

cvs co -r ''TAGNAME'' -d ''DIRECTORY'' GEOSagcm

where ''TAGNAME'' is the model "tag" (version). A tag in <code>cvs</code> marks the various versions of the source files in the repository that together make up a particular version of the model. A sample release tag is <code>~~Ganymed~~-~~4_1~~</code>, indicating the latest patch of version Heracles 4.0 for general use. ''DIRECTORY'' is the directory that the source code tree will be created. If you are using a stock model tag it is reasonable to name the directory the same as the tag. This directory determines which model in presumably your space a particular experiment is using. Some scripts use the environment variable <code>ESMADIR</code>, which should be set to the absolute (full) pathname of this directory.

where ''TAGNAME'' is the model "tag" (version). A tag in <code>cvs</code> marks the various versions of the source files in the repository that together make up a particular version of the model. A sample release tag is <code>Heracles-4_0</code>, indicating the latest patch of version Heracles 4.0 for general use. ''DIRECTORY'' is the directory that the source code tree will be created. If you are using a stock model tag it is reasonable to name the directory the same as the tag. This directory determines which model in presumably your space a particular experiment is using. Some scripts use the environment variable <code>ESMADIR</code>, which should be set to the absolute (full) pathname of this directory.

When a modified version of some component of the model is saved to the repository, the tag it uses -- different from the standard model tag -- is supposed to be applied at most only to the directories with modified files. This means that if you need to use some variant tag of a gridded component, you will have to <code>cd</code> to that directory and update to the variant tag. So, for example, if you needed to apply updates to the SatSim gridded component, you would have to <code>cd</code> several levels down to the directory <code>GEOSsatsim_GridComp</code> and run

Line 23:

=== Using <code>gcm_setup</code> ===

The setup script for global runs, <code>gcm_setup</code>, is in the directory <code>src/Applications/GEOSgcm_App</code>. The following is an example of a session with the setup script, with commentary. :

~~<pre>~~

Enter the Experiment ID:

~~</pre>~~

Enter a name and hit return. For this example we'll set the experiment ID to "myexp42". Experiment IDs need to have no whitespace and not start with a digit, since it will be the prefix of job names and PBS imposes certain limits on job names.

Enter a name and hit return. For this example we'll set the experiment ID to "myexp42". Experiment IDs need to have no whitespace and not start with a digit, since it will be the prefix of job names and the queue system imposes certain limits on job names.

Enter a 1-line Experiment Description:

~~<pre>~~

~~Enter a 1-line Experiment Description:~~

~~</pre>~~

This should be short but descriptive, since it will be used to label plots. It can have spaces, though the string will be stored with underscores for the spaces. Provide a description and hit return.

Enter an Experiment Source Tag for History (Default: Heracles-4_0):

The setup script extracts the repository tag of the model you checked out into your sandbox, checks out that tag from the repository, and creates a tarball of the source tree to keep with the experiment. It also <code>diff</code>s your sandbox against the tag it checks out and records those differences with the tarball in <code><i>EXPDIR</i>/src</code>. This is where you can change the default tag extracted from your sandbox.

Do you wish to CLONE an old experiment? (Default: NO or FALSE)

This gives you an opportunity to copy the setup of an existing experiment while changing what needs to be changed to make it work.

<pre>

Line 58:

Line 70:

The options b/c/d/e select a resolution with lat/lon, and c48-c1440 select one with the cubed sphere. Enter a resolution like so:

~~<pre>~~

c48

~~</pre>~~

and hit enter.

~~<pre>~~

Enter the Atmospheric Model Vertical Resolution: LM (Default: 72)

Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE)

~~</pre>~~

Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE)

You probably don't, so hit enter.

Line 79:

Line 90:

Unless you are using a higher-resolution experiment, the default will suffice.

~~<pre>~~

Do you wish to run GOCART? (Default: NO or FALSE)

~~</pre>~~

GOCART is the interactive chemistry package, as opposed to prescribed chemistry. It incurs a significant performance cost, so unless you know you want it, you should go with the default. The following assumes that you have entered "y". Otherwise, skip two steps to "Enter the tag..."

Enter the GOCART Emission Files to use: MERRA2 (Default), PIESA, CMIP, NR, MERRA2-DD or OPS:

~~<pre>~~

~~Enter the GOCART Emission Files to use: "CMIP" (Default), "PIESA", or "OPS":~~

~~</pre>~~

Select your favorite emission files here.

~~<pre>~~

Enter the AERO_PROVIDER: GOCART (Default) or GOCART.data:

Enter the AERO_PROVIDER: GOCART (Default) or ~~PCHEM~~:

~~</pre>~~

Here you get to choose again to use interactive or prescribed aerosols.

Line 107:

Line 112:

This provides a default HISTORY.rc (output specification) file. The initial default will be the tag of the build in which you are running <code>gcm_setup</code>. The idea is that you can save a custom <code>HISTORY.rc</code> to the repository and have it checked out for your experiments.

~~<pre>~~

Enter Desired Location for HOME Directory (to contain scripts and RC files)

Hit ENTER to use Default Location:

Enter Desired Location for HOME Directory (to contain scripts and RC files)

----------------------------------

Hit ENTER to use Default Location:

Default: /discover/nobackup/aeichman/myexp42

----------------------------------

Default: /discover/nobackup/aeichman/myexp42

~~</pre>~~

This option determines where the experiment's home directory is located -- where the basic job scripts and major RC files (<code>AGCM.rc</code>, <code>CAP.rc</code> and <code>HISTORY.rc</code>) will be located. The first time you run the script it will default to a subdirectory under your account's home directory named <code>geos5</code>, remember what you decide (in <code>~/.HOMDIRroot</code>) and use that as a default in subsequent times the script is run. This initial default is fine, though another possibility is to enter your nobackup space, as shown here. This will place all of the HOME directory files of the experiment together with the rest of them.

~~<pre>~~

Enter Desired Location for EXPERIMENT Directory (to contain model output and restart files)

Hit ENTER to use Default Location:

----------------------------------

Default: /discover/nobackup/aeichman/myexp42

~~</pre>~~

This determines the experiment directory, where restart files and various job output is stored. These are the storage-intensive parts and so default to the <code>nobackup</code> space.

~~<pre>~~

Enter Location for Build directory containing: src/ Linux/ etc...

Hit ENTER to use Default Location:

----------------------------------

Default: /discover/nobackup/aeichman/~~Ganymed~~-~~4_1~~

Default: /discover/nobackup/aeichman/Heracles-4_0

~~</pre>~~

This determines which of your local builds is used to create the experiment. It defaults to the build of the script you are running, which is generally a good idea.

Line 156:

Line 155:

Creating gcm_convert.j for Experiment: myexp42

Creating gcm_plot.tmpl for Experiment: myexp42

Creating gcm_quickplot.csh for Experiment: myexp

Creating gcm_moveplot.j for Experiment: myexp

Creating gcm_stats.j for Experiment: myexp

Creating gcm_forecast.tmpl for Experiment: myexp42

Creating gcm_forecast.setup for Experiment: myexp42

Line 173:

Line 175:

The following executable has been placed in your Experiment Directory:

----------------------------------------------------------------------

/discover/nobackup/aeichman/~~Ganymed~~-~~4_1~~/Linux/bin/GEOSgcm.x

/discover/nobackup/aeichman/Heracles-4_0/Linux/bin/GEOSgcm.x

Line 192:

Line 194:

=== Using restart files ===

Restart files provide the initial conditions for a run, and a set needs to be copied into a fresh experiment directory before running. This includes the file <code>cap_restart</code>, which provides the model starting date and time in text. Restart files themselves are resolution-specific and sometimes change between model versions. As of the current model version, they are flat binary files ~~with no metadata~~, so they tend to be stored together with restarts of the same provinance with the date either embedded in the filename or in an accompanying <code>cap_restart</code>, typically under a directory indicating the model version.

Restart files provide the initial conditions for a run, and a set needs to be copied into a fresh experiment directory before running. This includes the file <code>cap_restart</code>, which provides the model starting date and time in text. Restart files themselves are resolution-specific and sometimes change between model versions. As of the current model version, they are now netcdf files with metadata instead of the previous flat binary files but typically have the same naming convention, so they tend to be stored together with restarts of the same provinance with the date either embedded in the filename or in an accompanying <code>cap_restart</code>, typically under a directory indicating the model version. If you have restarts from a previous model version it may be worthwhile to try running with them as the model will check to see if they are in netcdf format, and if not, convert them.

A cleanly completed model run will leave a set of restarts and the corresponding <code>cap_restart</code> in its experiment directory. Another source is <code>/archive/u/aeichman/restarts</code>. Restarts are also left during runs in date-labeled tarballs in the <code>restarts</code> directory under the experiment directory before being transferred to the user's <code>/archive</code> space. You may have to create the <code>cap_restart</code>, which is simply one line of text with the date of the restart files in the format ''YYYYMMDD HHMMSS'' (with a space).

Line 273:

Line 275:

Setting <code>PRINTSPEC</code> to 3 will make the model send to standard output a list of exports available to <code>HISTORY.rc</code> in the model's current configuration, and then exit without integrating. The list includes each export's gridded component and short name (both necessary to include in <code>HISTORY.rc</code>), long (descriptive) name, units, and number of dimensions. Note that run-time options can affect the exports available, so see to it that you have those set as you intend. The other <code>PRINTSPEC</code> values are useful for debugging.

While you can set <code>PRINTSPEC</code>, submit <code>~~qsub~~ gcm_run.j</code>, and get the export list as part of PBS standard output, there are quicker ways of obtaining the list. One way is to run it as a single column model on a single processor, as explained in [[Heracles 4.0 Single Column Model]]. Another way is to run it in an existing experiment. In the <code>scratch</code> directory of an experiment that has already run, change <code>PRINTSPEC</code> in <code>CAP.rc</code> as above. Then, in the file <code>AGCM.rc</code>, change the values of <code>NX</code> and <code>NY</code> (near the beginning of the file) to 1. Then, from an interactive job (one processor will suffice), run the executable <code>GEOSgcm.x</code> in <code>scratch</code>. You will need to run <code>source src/g5_modules</code> in the model's build tree to set up the environment. The model executable will simply output the export list to <code>stdout</code>.

While you can set <code>PRINTSPEC</code>, submit <code>sbatch gcm_run.j</code>, and get the export list as part of PBS standard output, there are quicker ways of obtaining the list. One way is to run it as a single column model on a single processor, as explained in [[Heracles 4.0 Single Column Model]]. Another way is to run it in an existing experiment. In the <code>scratch</code> directory of an experiment that has already run, change <code>PRINTSPEC</code> in <code>CAP.rc</code> as above. Then, in the file <code>AGCM.rc</code>, change the values of <code>NX</code> and <code>NY</code> (near the beginning of the file) to 1. Then, from an interactive job (one processor will suffice), run the executable <code>GEOSgcm.x</code> in <code>scratch</code>. You will need to run <code>source src/g5_modules</code> in the model's build tree to set up the environment. The model executable will simply output the export list to <code>stdout</code>.

===Outputting Derived Fields===

@@ Line 7: / Line 7: @@
 Most of the time for longer runs you will be using a release version of the model, perhaps compiled with a different version of one or more of the model's gridded components, as defined by subdirectories in the source code.  This process starts with checking out the stock model from the repository using the command
-   cvs co -r  ''TAGNAME'' -d ''DIRECTORY'' Heracles GEOSagcm
+   cvs co -r  ''TAGNAME'' -d ''DIRECTORY''  GEOSagcm
-where ''TAGNAME'' is the model "tag" (version).  A tag in <code>cvs</code> marks the various versions of the source files in the repository that together make up a particular version of the model.  A sample release tag is <code>Ganymed-4_1</code>, indicating the latest patch of version Heracles 4.0 for general use. ''DIRECTORY'' is the directory that the source code tree will be created.  If you are using a stock model tag it is reasonable to name the directory the same as the tag.  This directory determines which model in presumably your space a particular experiment is using.  Some scripts use the environment variable <code>ESMADIR</code>, which should be set to the absolute (full) pathname of this directory.
+where ''TAGNAME'' is the model "tag" (version).  A tag in <code>cvs</code> marks the various versions of the source files in the repository that together make up a particular version of the model.  A sample release tag is <code>Heracles-4_0</code>, indicating the latest patch of version Heracles 4.0 for general use. ''DIRECTORY'' is the directory that the source code tree will be created.  If you are using a stock model tag it is reasonable to name the directory the same as the tag.  This directory determines which model in presumably your space a particular experiment is using.  Some scripts use the environment variable <code>ESMADIR</code>, which should be set to the absolute (full) pathname of this directory.
 When a modified version of some component of the model is saved to the repository, the tag it uses -- different from the standard model tag -- is supposed to be applied at most only to the directories with modified files.  This means that if you need to use some variant tag of a gridded component, you will have to <code>cd</code> to that directory and update to the variant tag.  So, for example, if you needed to apply updates to the SatSim gridded component, you would have to <code>cd</code> several levels down to the directory <code>GEOSsatsim_GridComp</code> and run
@@ Line 23: / Line 23: @@
 === Using <code>gcm_setup</code> ===
 The setup script for global runs, <code>gcm_setup</code>, is in the directory <code>src/Applications/GEOSgcm_App</code>.  The following is an example of a session with the setup script, with commentary.  :
-<pre>
   Enter the Experiment ID:
-</pre>
-Enter a name and hit return.  For this example we'll set the experiment ID to "myexp42".  Experiment IDs need to have no whitespace and not start with a digit, since it will be the prefix of job names and PBS imposes certain limits on job names.
+Enter a name and hit return.  For this example we'll set the experiment ID to "myexp42".  Experiment IDs need to have no whitespace and not start with a digit, since it will be the prefix of job names and the queue system imposes certain limits on job names.
+ Enter a 1-line Experiment Description:
-<pre>
-Enter a 1-line Experiment Description:
-</pre>
 This should be short but descriptive, since it will be used to label plots.  It can have spaces, though the string will be stored with underscores for the spaces.  Provide a description and hit return.
+ Enter an Experiment Source Tag for History (Default: Heracles-4_0):
+The setup script extracts the repository tag of the model you checked out into your sandbox, checks out that tag from the repository, and creates a tarball of the source tree to keep with the experiment. It also <code>diff</code>s your sandbox against the tag it checks out and records those differences with the tarball in <code><i>EXPDIR</i>/src</code>. This is where you can change the default tag extracted from your sandbox.
+ Do you wish to CLONE an old experiment? (Default: NO or FALSE)
+This gives you an opportunity to copy the setup of an existing experiment while changing what needs to be changed to make it work.
 <pre>
@@ Line 58: / Line 70: @@
 The options b/c/d/e select a resolution with lat/lon, and c48-c1440 select one with the cubed sphere. Enter a resolution like so:
-<pre>
+ c48
-c48
-</pre>
 and hit enter.
-<pre>
+ Enter the Atmospheric Model Vertical Resolution: LM (Default: 72)
-Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE)
-</pre>
+ Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE)
 You probably don't, so hit enter.
@@ Line 79: / Line 90: @@
 Unless you are using a higher-resolution experiment, the default will suffice.
-<pre>
+ Do you wish to run GOCART? (Default: NO or FALSE)
-Do you wish to run GOCART? (Default: NO or FALSE)
-</pre>
 GOCART is the interactive chemistry package, as opposed to prescribed chemistry.  It incurs a significant performance cost, so unless you know you want it, you should go with the default.  The following assumes that you have entered "y".  Otherwise, skip two steps to "Enter the tag..."
+ Enter the GOCART Emission Files to use: MERRA2 (Default), PIESA, CMIP, NR, MERRA2-DD or OPS:
-<pre>
-Enter the GOCART Emission Files to use: "CMIP" (Default), "PIESA", or "OPS":
-</pre>
 Select your favorite emission files here.
-<pre>
+ Enter the AERO_PROVIDER: GOCART (Default) or GOCART.data:
-Enter the AERO_PROVIDER: GOCART (Default) or PCHEM:
-</pre>
 Here you get to choose again to use interactive or prescribed aerosols.
@@ Line 107: / Line 112: @@
 This provides a default HISTORY.rc (output specification) file.  The initial default will be the tag of the build in which you are running <code>gcm_setup</code>.  The idea is that you can save a custom <code>HISTORY.rc</code> to the repository and have it checked out for your experiments.
-<pre>
+  Enter Desired Location for HOME Directory (to contain scripts and RC files)
+ Hit ENTER to use Default Location:
-Enter Desired Location for HOME Directory (to contain scripts and RC files)
+ ----------------------------------
-Hit ENTER to use Default Location:
+ Default:  /discover/nobackup/aeichman/myexp42
-----------------------------------
-Default:  /discover/nobackup/aeichman/myexp42
-</pre>
 This option determines where the experiment's home directory is located -- where the basic job scripts and major RC files (<code>AGCM.rc</code>, <code>CAP.rc</code> and <code>HISTORY.rc</code>) will be located.  The first time you run the script it will default to a subdirectory under your account's home directory named <code>geos5</code>, remember what you decide (in <code>~/.HOMDIRroot</code>) and use that as a default in subsequent times the script is run.  This initial default is fine, though another possibility is to enter your nobackup space, as shown here.  This will place all of the HOME directory files of the experiment together with the rest of them.
-<pre>
+ Enter Desired Location for EXPERIMENT Directory (to contain model output and restart files)
-Enter Desired Location for EXPERIMENT Directory (to contain model output and restart files)
+ Hit ENTER to use Default Location:
-Hit ENTER to use Default Location:
+ ----------------------------------
-----------------------------------
+ Default:  /discover/nobackup/aeichman/myexp42
-Default:  /discover/nobackup/aeichman/myexp42
-</pre>
 This determines the experiment directory, where restart files and various job output is stored.  These are the storage-intensive parts and so default to the <code>nobackup</code> space.
-<pre>
-Enter Location for Build directory containing:  src/ Linux/ etc...
+ Enter Location for Build directory containing:  src/ Linux/ etc...
-Hit ENTER to use Default Location:
+ Hit ENTER to use Default Location:
-----------------------------------
+ ----------------------------------
-Default:  /discover/nobackup/aeichman/Ganymed-4_1
+ Default:  /discover/nobackup/aeichman/Heracles-4_0
-</pre>
 This determines which of your local builds is used to create the experiment.  It defaults to the build of the script you are running, which is generally a good idea.
@@ Line 156: / Line 155: @@
 Creating gcm_convert.j for Experiment: myexp42
 Creating gcm_plot.tmpl for Experiment: myexp42
+Creating gcm_quickplot.csh for Experiment: myexp
+Creating gcm_moveplot.j for Experiment: myexp
+Creating gcm_stats.j for Experiment: myexp
 Creating gcm_forecast.tmpl for Experiment: myexp42
 Creating gcm_forecast.setup for Experiment: myexp42
@@ Line 173: / Line 175: @@
 The following executable has been placed in your Experiment Directory:
 ----------------------------------------------------------------------
-/discover/nobackup/aeichman/Ganymed-4_1/Linux/bin/GEOSgcm.x
+/discover/nobackup/aeichman/Heracles-4_0/Linux/bin/GEOSgcm.x
@@ Line 192: / Line 194: @@
 === Using restart files ===
-Restart files provide the initial conditions for a run, and a set needs to be copied into a fresh experiment directory before running.  This includes the file <code>cap_restart</code>, which provides the model starting date and time in text.  Restart files themselves are resolution-specific and sometimes change between model versions.  As of the current model version, they are flat binary files with no metadata, so they tend to be stored together with restarts of the same provinance with the date either embedded in the filename or in an accompanying <code>cap_restart</code>, typically under a directory indicating the model version.
+Restart files provide the initial conditions for a run, and a set needs to be copied into a fresh experiment directory before running.  This includes the file <code>cap_restart</code>, which provides the model starting date and time in text.  Restart files themselves are resolution-specific and sometimes change between model versions.  As of the current model version, they are now netcdf files with metadata instead of the previous flat binary files but typically have the same naming convention, so they tend to be stored together with restarts of the same provinance with the date either embedded in the filename or in an accompanying <code>cap_restart</code>, typically under a directory indicating the model version.  If you have restarts from a previous model version it may be worthwhile to try running with them as the model will check to see if they are in netcdf format, and if not, convert them.
 A cleanly completed model run will leave a set of restarts and the corresponding <code>cap_restart</code> in its experiment directory.  Another source is <code>/archive/u/aeichman/restarts</code>.  Restarts are also left during runs in date-labeled tarballs in the <code>restarts</code> directory under the experiment directory before being transferred to the user's <code>/archive</code> space.  You may have to create the <code>cap_restart</code>, which is simply one line of text with the date of the restart files in the format ''YYYYMMDD HHMMSS'' (with a space).
@@ Line 273: / Line 275: @@
 Setting <code>PRINTSPEC</code> to  3 will make the model send to standard output a list of exports available to <code>HISTORY.rc</code> in the model's current configuration, and then exit without integrating. The list includes each export's gridded component and short name (both necessary to include in <code>HISTORY.rc</code>), long (descriptive) name, units, and number of dimensions.  Note that run-time options can affect the exports available, so see to it that you have those set as you intend.  The other <code>PRINTSPEC</code> values are useful for debugging.
-While you can set  <code>PRINTSPEC</code>, submit <code>qsub gcm_run.j</code>, and get the export list as part of PBS standard output, there are quicker ways of obtaining the list.  One way is to run it as a single column model on a single processor, as explained in [[Heracles 4.0 Single Column Model]].  Another way is to run it in an existing experiment.  In the <code>scratch</code> directory of an experiment that has already run, change <code>PRINTSPEC</code> in  <code>CAP.rc</code> as above.  Then, in the file <code>AGCM.rc</code>, change the values of <code>NX</code> and <code>NY</code> (near the beginning of the file) to 1.  Then, from an interactive job (one processor will suffice), run the executable <code>GEOSgcm.x</code> in <code>scratch</code>.  You will need to run <code>source src/g5_modules</code> in the model's build tree to set up the environment.  The model executable will simply output the export list to <code>stdout</code>.
+While you can set  <code>PRINTSPEC</code>, submit <code>sbatch gcm_run.j</code>, and get the export list as part of PBS standard output, there are quicker ways of obtaining the list.  One way is to run it as a single column model on a single processor, as explained in [[Heracles 4.0 Single Column Model]].  Another way is to run it in an existing experiment.  In the <code>scratch</code> directory of an experiment that has already run, change <code>PRINTSPEC</code> in  <code>CAP.rc</code> as above.  Then, in the file <code>AGCM.rc</code>, change the values of <code>NX</code> and <code>NY</code> (near the beginning of the file) to 1.  Then, from an interactive job (one processor will suffice), run the executable <code>GEOSgcm.x</code> in <code>scratch</code>.  You will need to run <code>source src/g5_modules</code> in the model's build tree to set up the environment.  The model executable will simply output the export list to <code>stdout</code>.
 ===Outputting Derived Fields===