Running ADAS

DAS assimilation and forecast runs

Select DAS run and corresponding build

First, we choose a DAS run to simulate. For a list of runs

$ ls ~dao_ops 

on NCCS discover. As an example, we choose the run d591_fpit

Look for ATAG in file ~dao_ops/d591_fpit/run/FVDAS_Run_Config

$ grep ATAG ~dao_ops/d591_fpit/run/FVDAS_Run_Config

    setenv ATAG    GEOSadas-5_9_1_p4
    setenv FVARCH   /archive/u/gmao_ops/GEOS-5.9.1/${ATAG}/ #include trailing colon

This gives us the tag and location of the restart files [for an older build, it is best to checkout the model tag (given by ATAG) and rebuild the model].

Move to the bin directory (of the existing build) to setup experiment.

$ cd ~dao_ops/GEOSadas-5_9_1_p4/GEOSadas/Linux/bin

Setup experiment

Run fvsetup from the bin directory with the following options (default value is chosen for the ones not mentioned): [For now we choose the default value for FVICS as well]

AGCM Resolution? [b72]
> d72

OGCM Resolution? [c34]
> 

EXPID? [u000_d72]
> g591p4

OBSERVING SYSTEM CLASSES?
> "ncep_prep_bufr"

CHECKING OBSYSTEM? [2]
> 3

After saving these inputs, move to the experiment directory

$ cd $NOBACKUP/g591p4/run

At this point, one can edit CAP.rc to run 12-hour experiment. Also, one can turn on MAPL timers and memutils

JOB_SGMT:     0 120000
NUM_SGMT:     1

MAPL_ENABLE_TIMERS: YES
MAPL_ENABLE_MEMUTILS: YES

Data Assimilation experiment

$ cd $NOBACKUP/g591p4_run/run
$ qsub g5das.j

g5das creates a scratch directory fvwork.xxxx under $NOBACKUP and runs the experiment there. After the initial setup, GSIsa.x is run with output written to ana.log followed by GEOSgcm.x which writes its output to stdout and fvpsas.log.

Forecast experiment

Later.

Simulating DAS forecast run

The DAS forecast run can be simulated using the corresponding AGCM tag.

Checkout and build the model

Grab the corresponding AGCM tag (Check out GEOSadas-5_9_1_p4 and look for GCM in the file src/Applications/GEOSdas_App/ChangeLog).

 $ cvs co -r GEOSadas-5_7_2_p5_m1 GEOSadas-5_7
 $ cd GEOSadas/src
 $ source g5_modules
 $ gmake install

Or, one can checkout the corresponding AGCM model Fortuna-2_5_p6 (module Fortuna) and build it.

Note: For GEOSadas-5_7_2_p5_m1 in the example above, g5_modules will need to be edited in order to compile successfully on discover (due to SP1). You should change the discover section to be (where the bold items are changes, and mod5 is removed):

  set basedir = /discover/nobackup/projects/gmao/share/gmao_ops/Baselibs/v3.2.0_buildSLES11SP1
  set mod1 = comp/intel-11.0.083
  set mod2 = mpi/impi-3.2.2.006
  set mod3 = lib/mkl-10.0.3.020
  set mod4 = other/SIVO-PyD/spd_1.6.0_gcc-4.3.4-sp1
  
  set mods = ( $mod1 $mod2 $mod3 $mod4 )
  set modinit = /usr/share/modules/init/csh
  set loadmodules = 0 

2. Experiment setup:

To simulate a forecast run we can use gcm_setup to setup our experiment (instead of fvsetup). Run ./gcm_setup and provide the required answers

 $ cd Applications/GEOSgcm_App/
 $ ./gcm_setup

Cubed-sphere was not fully implemented in this model. Choose Lat/Lon with resolution 1152x721x72 for the 'e'-run. The other options are

 GOCART: yes
 GOCART Emission Files to use: OPS
 AERO_PROVIDER: GOCART
 HISTORY: Current

3. AGCM.rc, CAP.rc, HISTORY.rc

Edit these files to match with the corresponding .tmpl files in

 ~dao_ops/e572p5_fp/run

Then, change the restart reading behavior to pbinary:

 $ sed -i -e '/RESTART_TYPE:/ s/binary/pbinary/' AGCM.rc

4. gcm_run.j

Ensure that the correct boundary datasets are linked to. Compare the links in this file to that of

 ~dao_ops/e572p5_fp/run/lnbcs

Also, be sure to add an "exit" after you run the executable (if, say, you want to run one segment).

5. RC files

Copy the .rc files over

 $ find ~dao_ops/e572p5_fp/run -name "*.rc" -exec cp {} /path/to/home/dir/RC \;

6. restart files

Locate the dir containing restarts by querying FVARCH in FVDAS_Run_Config

 $ grep ATAG ~dao_ops/e572p5_fp/run/FVDAS_Run_Config

   setenv ATAG    GEOSadas-5_7_2_p5_m1
   setenv ATAG    $GTAG
   setenv FVARCH   /archive/u/gmao_ops/GEOS-5.7.2/${ATAG}/ #include trailing colon

Pick a date and time: say 2012/05/15, 21z

 $ cd /archive/u/gmao_ops/GEOS-5.7.2/GEOSadas-5_7_2_p5_m1/e572p5_fp/rs/Y2012/M05
 $ ls *_rst.20120515_21z.bin

Copy these files over from ARCHIVE to experiment directory and the rename them:

 $ cd /path/to/experiment
 $ rename _rst.20120515_21z.bin _rst *_rst.20120515_21z.bin
 $ rename e572p5_fp. '' *rst

Also, create a cap_restart file:

 $ echo '20120515 210000' > cap_restart

7. Run job

 $ qsub gcm_run.j