Using the ExtData component

Overview of the ExtData Component

MAPL_ExtDataGridCompMod is an internal MAPL gridded component used to fulfill imports fields in a MAPL hierarchy from netcdf files on disk. It is usually one of the three gridded components in the cap or main program of a MAPL application, the others being the root of the MAPL hierarchy it is servicing and MAPL_HistoryGridCompMod. It is instanciated and all its registered methods are run automatically by the CAP. MAPL_ExtDataGridCompMod will provide data to fields in the Import states of MAPL components that are not satisfied by a MAPL_AddConnectivity call in the MAPL hierarchy. In a MAPL application fields added to the Import state of a component are passed up the MAPL hierarchy looking for a connectivity to another component that will provide data to fill the import. If a connectivity is not found these fields will eventually reach the cap. At this point any fields that have not have their connectivity satisfied are passed to the MAPL_ExtDataGridCompMod through its Export state. MAPL_ExtDataGridCompMod is in essence a provider of last resort for Import fields that need to be filled with data. Like other components it has a run method that gets called every step in your MAPL application. What actually happens when it is run is determined by the resource file. The graphic below illustrates where ExtData fits in the MAPL application.

The user provides a resource file available to the MAPL_ExtDataGridCompMod GC. At its heart this resource file provides a connection between a field name and a variable name in a netcdf file on disk. The component receives a list of fields that need to be filled and parses the resource file to determine if MAPL_ExtDataGridCompMod can fill a variable of that name. We will refer to each field name-file variable combination as a primary export. Each primary export is an announcment that MAPL_ExtDataGridCompMod is capable of filling a field named A with data contained in variable B on file xyz. Note that the field name in each primary export does not need to actually be a field that needs to be filled by the model. The component only processes primary exports that are needed The resource file should be viewed as an anncountment of what MAPL_ExtDataGridCompMod can provide. In addition to simply announcing what MAPL_ExtDataGridCompMod can provide the user can specify other information such as how frequently to update the data from disk. This could be at every step, just once when starting the model run, or at a particular time each days. MAPL_ExtDataGridCompMod also allows data to be shifted and scaled.

ExtData Resource File

Primary Exports

The ExtData.rc file itself can have up to 3 parts. The first is the primary exports. The example below is an example of a primary export block. This may have as many entries as the user desires. Note that only the last three entries in a line are optional, even though the units are not currently used (this may change someday).

PrimaryExports::
# ---------|---------|-----|-----|------|----------------------|--------|-------|---------------------------------|-----------------------------------------------------------|
#  Export  |         |     | V   |      |_______ Refresh ______|____ Factors ___|________ External File __________|______________________External File Time Data______________|
#  Name    | Units   | Dim | Loc | Clim |     Time Template    | Offset | Scale | Variable |      Template        |  Reference Time    | File Frequency Unit | File Frequency |
# ---------|---------|-----|-----|------|----------------------|--------|-------|----------|----------------------|--------------------|---------------------|----------------|
ALBNF           NA      xy    c      N             0               0.0      1.0     ALBNF    myfile.%y4%m2%d2.nc4   2000-04-15T00:00:00           days               1
du001           NA      xyz   c      N             0               0.0      1.0     du001    /dev/null
::

The following is an explanation of each entry in a line.

Export Name - This is the actual name of the export in the application to fill

Units - Units not currently used

Dimensions - xy for 2D or xyz for 3D

Vertical location - c for center, e for edge, if 2d enter c or e but this will obviously not be used

Clim - enter Y if the file is a climatology file, otherwise enter no. If you specify it is a climatology ExtData will assume all data is on one file.

Refresh template - you have 3 choices

1.) Enter '-'. In this case the field will only be updated once the first time ExtData runs

2.) Enter a refresh template of the form %y4-%m2-%h2T%h2:%n2:00 to set the recurring time to update the file. For example to update the file once a day at 21Z enter %y4-%m2-%d2T21:00:00, as another example update the file the first of the month at 21Z enter %y4-%m2-01T21:0000

3.) Enter '0' to update the variable at every step. ExtData will do a linear interpolation using the available times on the file. This option allows the user more sophisticated option for breaking the data up into different files. See the discussion further down about the file frequency.

Offset - This is a factor the variable will be shifted by

Scale - This is a factor the variable will be scaled by

Variable - This is the name of the variable ON THE FILE. It need not be the same as the export name. You can also set this to /dev/null to set the

File Template - this is a grads style template, if there are no tokens in the template name ExtData will assume that all the data is on one file. Note that if the data on file is at a different resolution that the application grid, the underlying I/O library ExtData uses will regrid the data to the application grid.

The next 3 keywords are optional but all 3 must be supplied if used. Also note if your data is on one file then it makes no sense to worry about these.

Reference Time - When trying to find a file to get data from, the times tried when applying the template will be some multiple of a time interval from this reference time.

File Frequency Units - The frequency units for the file, options are years, months, days, hours, minutes.

File Frequency - An integer specifying the frequency in the units given in the previous keyword.

Refresh Template

The refresh template allows quite a bit of flexibility in how the data can be distributed in the files and when the data is updated.

If you enter '-' the field will be updated only the first time ExtData runs. It will take the current model time, apply that to the file template provided and finally try to find data on the file at the current model time to fill the variable. It will not do time interpolation.

If you enter a refresh template to update at a certain time each day, month, etc ... as described above there are a few considerations. When ExtData trieds to fill the field at the time described by the template it will once again apply the current model time to the file template, and then try to find data on file at that time. It will not do time interpolation.

The most complex case is if you enter '0' for the refresh template. In this case the field gets updated every step and ExtData will try to interpolate to that time using the data on file and can interpolate between data points in different files. In order to accomplish this if you select a refresh template of '0' ExtData tries to find two times that bracket the start time the first time ExtData runs. These get updated as time advances in your application. But how does it know what time to apply to the file template if the data is spread across multiple files? This is where the idea of a file frequency and reference time comes in. The user can specify a file frequency and a reference time either with the 3 optional key words to the primary export or by letting ExtData "guess" from the file template. The idea is that starting from a reference time and the given frequency ExtData tries to find the bracketing data on the file with the time closest to the current time. If it does not find the data there it checks the next time. The user can specify a reference time and the frequency. If these are not given it tries to guess this from the file template. For example if the last token in the template is an hour then the frequency is an hour and the reference time is whatever the current data is with anything after the hour set to zero. If the file template is myfile_%y4%m2%d2_%h2.nc4 and ExtData will assume the frequency is 1 hour and the reference time is the top of the hour on whatever date you start the application. The following picture illustrates another example.