Regridding Restarts for Heracles 5.3

Here are basic instructions on how to create restart files for Heracles 5.3. Due to changes in the tiling it is not possible to use restarts from older versions of GEOS-5 without converting them with the regrid.pl script.

Back to GEOS-5 Documentation for Heracles 5.3

Using the `regrid.pl` Script

Restarts can be easily created from MERRA2 data or other model-generated sources using the regrid.pl script in Linux/bin/ (above src/) -- this script is different from, and is specifically a wrapper for, the regrid script used in earlier versions. The regrid.pl script is created when make install is run. You might have to update the regrid.pl script by going to src/GMAO_Shared/GEOS_Util/post, running cvs upd -r HEAD regrid.pl, and running make install. The same script can regrid restarts between different versions of Fortuna 1.4 to Heracles 5.3 at any resolution.

Running regrid.pl -h (after running source g5_modules) -- with the -h for help -- tells the story:


NAME
    regrid.pl

PURPOSE
    Wrapper script for programs which regrid GCM restarts

SYNOPSIS

    regrid.pl yyyymmdd hr gridID outdir [options]

REQUIRED INPUTS
   -----------------------------
   as ordered runtime parameters
   -----------------------------
   yyyymmdd          8-digit date of the restarts being regridded
   hr                2-digit hour of the restarts being regridded
   grOUT             grid ID of the output restarts (see GRID IDENTIFIERS below)
   outdir            directory location for output restarts

   --------------------
   or as flagged values
   --------------------
   -ymd      yyyymmdd
   -hr       hr
   -grout    grOUT
   -outdir   outdir

REQUIRED OPTION FOR MERRA INPUTS
   -merra             (same as -merra1)
   -merra1            get input restarts from OPS MERRA-1 data archives
   -merra2            get input restarts from OPS MERRA-2 data archives

REQUIRED OPTIONS FOR NON-MERRA INPUTS
   -d        rstdir   location of input restart files
   -expid    expid    experiment ID of input restart files

INTERACTIVE OPTION
   -i                 prompt for inputs that are not supplied
                      (this is the default if no inputs are supplied)

OTHER OPTIONS
   -oceanin  oceanIN  ocean horizontal grid of inputs
                      =c : 1-deg (360x180); e.g. Reynolds
                      =e : 1/4-deg (1440x720); e.g. MERRA-2
                      =f : 1/8-deg (2880x1440); e.g. OSTIA
                      defaults to 'c'
   -oceanout oceanOUT ocean horizontal grid of outputs (see -oceanIN)
   -esmabin  ESMABIN  location of build's scripts and programs; defaults to location
                      of regrid.pl script
   -iceDT     dtime   datetime for alternate landice rst input if regridding to
                      'Ganymed-2_0' from earlier tag; dtime should have the
                      following format: -iceDT yyyymmdd_hh
                      if dtime is not provided or if no restarts can be found to
                      match dtime, then script will prompt user from list of
                      available datetimes.
                      if dtime eq '0', then alternate landice rst will not be used
   -newid    newid    label to replace expid in output restart names;
                      defaults to 'n-expid' where n is OUTPUT atmosphere grid ID
   -tagin    tagIN    GCM or DAS tag associated with inputs (see TAGS below);
                      defaults to Ganymed-4_0
   -tagout   tagOUT   GCM or DAS tag associated with outputs (see TAGS below);
                      defaults to Ganymed-4_0
   -rs       flag     flag indicating which restarts to regrid
                      =1 for upper-air restarts only
                      =2 for land-surface restarts only
                      =3 for both upper-air and land-surface restarts (default)

   -[no]bkg           copy and rename input bkg + satbang/bias files
   -[no]lbl           label final restarts with 'tagID.gridID' extension
   -[no]lcv           create rst.lcv file for final restarts
   -gcm               gcm mode; equivalent to -nobkg, -lbl, and -nolcv flags

   -np                no prompt; take defaults for all prompts
   -db                (debug mode) Do not clean work directory after running programs;
   -dbh               (debug hash) Show contents of hashes: %IN and %OUT
   -v                 verbose mode
   -h[elp]            print usage message

GRID IDENTIFIERS

   Atmosphere Horizontal Grids
   ===========================
   latlon grids         im        jm
   ------------         --        --
   a = 4x5              72        46
   b = 2x2.5           144        91
   c = 1x1.25          288       181
   D = 0.5x0.666       540       361
   d = 0.5x0.625       572       361
   E = 0.25x0.3330    1080       721
   e = 0.25x0.3125    1152       721

   cubed-sphere grids
   ------------------
   Cn, where n = {24, 48, 90, 180, 360, 500, 720, 1000, 1440, 2000, 2880}

   Ocean Horizontal Grids
   ======================
   c = 1-deg   (360x180);   e.g. Reynolds
   e = 1/4-deg (1440x720) ; e.g. MERRA-2
   f = 1/8-deg (2880x1440); e.g. OSTIA

TAGS
   Use GCM or DAS tag names with -tagin and -tagout flags

      Sample GCM tags
      ---------------
      F14  : Fortuna-1_4  ............  Fortuna-1_4_p1
      F21  : Fortuna-2_1  ............  Fortuna-2_5_pp2
      G10  : Ganymed-1_0  ............  Ganymed-1_0_BETA4
      G10p : Ganymed-1_0_p1  .........  Ganymed-1_0_p6
      G20  : Ganymed-2_0  ............  Ganymed-2_1_p6
      G30  : Ganymed-3_0  ............  Ganymed-3_0_p1
      G40  : Ganymed-4_0  ............  Heracles-4_0

      Sample DAS tags
      ---------------
      214  : GEOSdas-2_1_4  ..........  GEOSdas-2_1_4-m4
      540  : GEOSadas-5_4_0  .........  GEOSadas-5_5_3
      561  : GEOSadas-5_6_1  .........  GEOSadas-5_7_3_p2
      580  : GEOSadas-5_8_0  .........  GEOSadas-5_9_1
      591p : GEOSadas-5_9_1_p1  ......  GEOSadas-5_9_1_p9
      5A0  : GEOSadas-5_10_0  ........  GEOSadas-5_10_0_p1
      5B0  : GEOSadas-5_10_0_p2  .....  GEOSadas-5_11_0
      512  : GEOSadas-5_12_2  ........  GEOSadas-5_13_1

AUTHOR
   Joe Stassi, SAIC (joe.stassi@nasa.gov)

So the following will create restarts in C180 (roughly equivalent to ½-degree lat-lon) resolution, on the ¼-degree ocean from MERRA2 on the first day of 2012 and put them in /discover/nobackup/mathomp4/Regrid:

./regrid.pl -ymd 20120101 -hr 21 -grout C180 -outdir /discover/nobackup/mathomp4/Regrid -merra -tagout Heracles-4_0 -oceanout e

The above presents the minimum options to get a working set of restarts -- anything lacking will be requested interactively. Your selections will be presented for your review before the script does any heavy lifting. The script may appear to stall while it waits for the relevant MERRA or MERRA-2 files to be retrieved from the /archive filesystem.

For sources other than MERRA/MERRA-2, you will have to specify the location of a regrid resource file that contains the model version, resolution and location of the source restarts, and the resolution of the output restarts. A sample regrid resource file is regrid.rc in the same directory as the regrid script. Note that regrid only works for creating restarts for its model tag, and does not work at all for making restarts for tags before Heracles 5.3. Restarts for earlier model tags have to be created "manually".

Renaming Restart Files

At this point you will have restart files with a fair bit of cruft in the filenames. As described above, they will have to be renamed for the model to recognize them. There are a number of scripts with different techniques for completing this tedious chore; here is one example:

#!/usr/bin/env python

import re
import os


# Open file for reading
dirlist = os.listdir(".")

for file in dirlist:

   if( re.search(".*rst.*",file) ):
      print file
# the concept is to match ??????.something_rst.??????? OR something_rst.????????
# and extract "something_rst"
      newfile = re.sub(r'(^.+\.|^)([a-z_]+rst)\..+$',r'\2',file)
      print newfile
      if cmp(newfile,file):
         os.symlink(file,newfile)

It will grab anything in the current working directory resembling the common forms of restart filename generated by the regridding scripts and other utilities and create symbolic links to them in the same directory with the superfluous parts of the filenames stripped. In an experiment directory the links will be overwritten by the model with "fresh" restart files at the end of a run segment. Alternatively, you can easily copy the restart files with cp *rst

Required Restarts

A few of the restart files are critical and need to be present for the model to function, while others can be "bootstrapped": the model is started without them and then it generates the restarts in order to save state for the next restart.

The following restart files are critical and need to be regridded:

fvcore_internal_rst (atmosphere)
moist_internal_rst (atmosphere)
catch_internal_rst (surface)

The following surface restarts should be regridded, but can be bootstrapped:

vegdyn_internal_rst
landice_internal_rst
lake_internal_rst
saltwater_internal_rst

Optional (boot-strappable) restarts:

pchem_internal_rst
gocart_internal_rst
turb_internal_rst
saltwater_import_rst
surf_import_rst
turb_import_rst
moist_import_rst

For DAS runs, you may also interpolate:

agcm_import_rst

The restarts irrad_internal_rst and solar_internal_rst should not be regridded from versions earlier than Fortuna 2.5, as they must be bootstrapped.

Note: if the grid dimensions are the same SOME restarts from earlier model versions may be used , while other may not.

fvcore_internal_rst and moist_internal_rst may be used directly from earlier versions.
catch_internal_rst, lake_internal_rst, landice_internal_rst, and saltwater_internal_rst must be re-gridded due to new land surface tile data
All other restarts from earlier versions should be boot-strapped.

Where to Find MERRA Restarts

If you lack a set of restarts for the appropriate date for an older version of Fortuna, you can derive yours from MERRA. MERRA restarts are available on the NCCS discover cluster under the following directories:

/archive/g_proj5/production/GEOSdas-2_1_4/d5_merra_jan79/rs/
/archive/g_proj5/production/GEOSdas-2_1_4/d5_merra_jan89/rs/
/archive/g_proj5/production/GEOSdas-2_1_4/d5_merra_jan98/rs/

Restarts are in subdirectories according to their respective year and month. The are in the form (as an example) d5_merra_jan79.catch_internal_rst.19910128_21z.bin, which you would otherwise know as catch_internal_rst (with the appropriate cap_restart). Be aware that a cp from /archive may appear to hang initially -- sometimes for hours -- while the appropriate tape is queued for mounting. Alternatively, run dmget on the files before copying them: this will retrieve the files and have them ready for copying once the command returns.

Regridding Restarts Manually

If you need restarts for a model tag before Fortuna 2.5, you will have to regrid them manually. This section shows you how.

Regridding Upper-Air Related Restarts

Within your build, go to the directory:

 src/GMAO_Shared/GEOS_Util/post

Set up the build environment by running source g5_modules and setting the evironment variable ESMADIR to the directory path above the src directory. Then run make rs_hinterp.x to produce the executable:

rs_hinterp.x

Running rs_hinterp.x shows the usage:

> ./rs_hinterp.x
Usage:

rs_hinterp_$ARCH.x -dyn      DYNRST
                   -moist    MOISTRST
                   -other    OtherRST1 OtherRST2 OtherRST3 ...
                   -topo_old TOPO_OLD
                   -topo_new TOPO_NEW
                   -im       IM_OUT
                   -jm       JM_OUT

where:

  -dyn         DYNRST:  Filename for DYNAMICS_INTERNAL_RESTART
  -moist     MOISTRST:  Filename for    MOIST_INTERNAL_RESTART
  -other     OtherRST:  Filename for Other Flat Binary_RESTART
  -topo_old  TOPO_OLD:  Filename for OLD Topography (associated with INPUT  resolution)
  -topo_new  TOPO_NEW:  Filename for NEW Topography (associated with OUTPUT resolution)
  -im          IM_OUT:  IM Dimension for Output
  -jm          JM_OUT:  JM Dimension for Output

So, as an example, to regrid from an existing 2x2.5-deg set of restarts to 1x1.25-deg, you have:

> rs_hinterp.x -dyn fvcore_internal_rst -moist moist_internal_rst -other pchem_internal_rst  gocart_internal_rst -topo_old TOPO_OLD 
-topo_new TOPO_NEW -im 288 -jm 181

where TOPO_OLD is linked to:

/discover/nobackup/ltakacs/bcs/Fortuna-2_1/144x91/topo_DYN_ave_144x91_DC.data

and TOPO_NEW is linked to:

/discover/nobackup/ltakacs/bcs/Fortuna-2_1/288x181/topo_DYN_ave_288x181_DC.data

Topography files for different Fortuna versions can be found under /discover/nobackup/ltakacs/bcs/. For regridding from MERRA use Fortuna 1.4 540x361 topography files.

Running the above will produce:

fvcore_internal_rst.0288x0181
moist_internal_rst.0288x0181
pchem_internal_rst.0288x0181
gocart_internal_rst.0288x0181

These are the finished restarts. They will have to be renamed -- truncated after rst -- in order to be recognized by the model in its standard configuration.

Note that even if you have no restarts that fall under the -other option, you still have to include that option (with nothing after it) in the command.

Regridding Land-Surface Related Restarts

Within your build, go to:

src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/ ...
GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts

At this point it is best to update your directory to the HEAD of CVS to be sure you have the latest datasets:

cvs upd -A

Under the mk_restarts directory you have the directories:

InData
OutData

The basic idea is:

Put your existing land-related restarts (catch_internal_rst, vegdyn_internal_rst, landice_internal_rst, lake_internal_rst, and saltwater_internal_rst) in the InData directory
Put the appropriate tile file for your current resolution in the InData directory
Put your target resolution tile file and appropriate clsm directory in the OutData directory

Consider soft linking files instead of copying, as they can be large. Where to find the appropriate tile file depends on the Fortuna versions you are regridding from and to, and is detailed below.

Run mk_Restarts. This will create the executables (if needed; remember to source g5_modules) and run them. If the executables are run without the proper setup, they will crash and create a zero-length file in OutData with the name *.til (with the asterisk in the filename); you will need to delete it before running again.

Your regridded restarts will be produced in the OutData directory.

Old and New Tile Files

There are two possible scenarios for regridding land-surface related restarts:

Converting existing restarts to resolutions: 144x91, 288x181, 540x361, 1080x721 (old tile files for Fortuna 1.4 and earlier)
Converting existing restarts to resolutions: 144x91, 288x181, 576x361, 1152x721 (new tile files for Fortuna 1.5 and later)

The old tile files are currently archived in:

/discover/nobackup/ltakacs/bcs/Fortuna-1_4/144x91/FV_144x91_DC_360x180_DE.til
/discover/nobackup/ltakacs/bcs/Fortuna-1_4/288x181/FV_288x181_DC_360x180_DE.til
/discover/nobackup/ltakacs/bcs/Fortuna-1_4/540x361/FV_540x361_DC_360x180_DE.til
/discover/nobackup/ltakacs/bcs/Fortuna-1_4/1080x721/FV_1080x721_DC_360x180_DE.til

The new tile files and clsm directories are located in:

/discover/nobackup/ltakacs/bcs/Fortuna-2_1/144x91/FV_144x91_DC_360x180_DE.til
/discover/nobackup/ltakacs/bcs/Fortuna-2_1/288x181/FV_288x181_DC_360x180_DE.til
/discover/nobackup/ltakacs/bcs/Fortuna-2_1/576x361/FV_576x361_DC_360x180_DE.til
/discover/nobackup/ltakacs/bcs/Fortuna-2_1/1152x721/FV_1152x721_DC_360x180_DE.til

(These work for Fortuna 2.4 as well.)

Depending on which tile file you are regridding to, there are two procedures:

Target tile file is from OLD set (model tags `Fortuna-1_4` and earlier)

Since your target tile file is OLD (and not consistent with the new mosaic_veg_typs_fracs data), you have to remove the mosaic_veg_typs_fracs file from the InData direcory (or simply rename it, eg: mosaic_veg_typs_fracs.hold). In its place you have to copy the OLD nirdf.dat, visdf.dat, and lai_grn_clim boundary condition datasets into the InData directory. These can be found in the corresponding /discover/nobackup/ltakacs/bcs/Fortuna-1_4 directories. Note, the names that the Fortran program is looking for are not the same names in our bcs directory. Therefore you need to link the old datasets to the proper names. For example, regridding FROM the old 540x361 resolution (eg. MERRA), you should have:

nirdf.dat -> /discover/nobackup/ltakacs/bcs/Fortuna-1_4/540x361/nirdf_540x361_DC.dat

visdf.dat -> /discover/nobackup/ltakacs/bcs/Fortuna-1_4/540x361/visdf_540x361_DC.dat

lai_grn_clim -> /discover/nobackup/ltakacs/bcs/Fortuna-1_4/540x361/lai_green_clim_540x361_DC.data

Target tile file is from NEW set (model tags `Fortuna-1_5` and later)

In this case, the target tile files are consistent with the mosaic_veg_typs_fracs dataset which resides in the InData directory. Therefore you do not have link in the old nirdf.dat, visdf.dat, and lai_grn_clim datasets. For Fortuna 2.2 you do have to copy or link the clsm directory in directory with the appropriately-dimensioned tile files.

Bootstrapping for a Complete Set of Restarts

The above procedure produces all the necessary restart files for the model to run, but others are needed for proper function. The model can produce them starting from the minimal set created by the regridding procedure; this requires some modification of the AGCM.rc file in your experiment home directory.

In AGCM.rc, using turb_import_rst as an example, add a "-" to the beginning of the filename of the restarts to be bootstrapped:

TURBULENCE_IMPORT_RESTART_FILE:         turb_import_rst
TURBULENCE_IMPORT_RESTART_TYPE:         binary 
TURBULENCE_IMPORT_CHECKPOINT_FILE:      turb_import_checkpoint
TURBULENCE_IMPORT_CHECKPOINT_TYPE:      binary

TURBULENCE_IMPORT_RESTART_FILE:         -turb_import_rst
TURBULENCE_IMPORT_RESTART_TYPE:         binary
TURBULENCE_IMPORT_CHECKPOINT_FILE:      turb_import_checkpoint
TURBULENCE_IMPORT_CHECKPOINT_TYPE:      binary

This will have the effect that when the model starts, it will look for that restart file and use it if it finds it, and will otherwise bootstrap the restart file. Running the model for a model day using this modified AGCM.rc will produce a full set of restart files, which can be used with the default model configuration.

Back to GEOS-5 Documentation for Heracles 5.3

Regridding Restarts for Heracles 5.3

Contents

Using the `regrid.pl` Script

Renaming Restart Files

Required Restarts

Where to Find MERRA Restarts

Regridding Restarts Manually

Regridding Upper-Air Related Restarts

Regridding Land-Surface Related Restarts

Old and New Tile Files

Target tile file is from OLD set (model tags `Fortuna-1_4` and earlier)

Target tile file is from NEW set (model tags `Fortuna-1_5` and later)

Bootstrapping for a Complete Set of Restarts

Navigation menu

Regridding Restarts for Heracles 5.3

Using the regrid.pl Script

Renaming Restart Files

Required Restarts

Where to Find MERRA Restarts

Regridding Restarts Manually

Regridding Upper-Air Related Restarts

Regridding Land-Surface Related Restarts

Old and New Tile Files

Target tile file is from OLD set (model tags Fortuna-1_4 and earlier)

Target tile file is from NEW set (model tags Fortuna-1_5 and later)

Bootstrapping for a Complete Set of Restarts

Navigation menu

Search

Using the `regrid.pl` Script

Target tile file is from OLD set (model tags `Fortuna-1_4` and earlier)

Target tile file is from NEW set (model tags `Fortuna-1_5` and later)