Using the ExtData component: Difference between revisions

MAPL_ExtDataGridCompMod is an internal MAPL gridded component used to fulfill imports fields in a MAPL hierarchy from NetCDF files containing gridded, geospatial data on disk. It is usually one of the three gridded components in the cap or main program of a MAPL application, the others being the root of the MAPL hierarchy it is servicing and MAPL_HistoryGridCompMod. It is instantiated and all its registered methods are run automatically by the CAP. MAPL_ExtDataGridCompMod will provide data to fields in the Import states of MAPL components that are not satisfied by a MAPL_AddConnectivity call in the MAPL

hierarchy. In a MAPL application fields added to the Import state of a component are passed up the MAPL hierarchy looking for a connectivity to another component that will provide data to fill the import. If a connectivity is not found these fields will eventually reach the cap. At this point any fields that have not have their connectivity satisfied are passed to the MAPL_ExtDataGridCompMod through its Export state. MAPL_ExtDataGridCompMod is in essence a provider of last resort for Import fields that need to be filled with data. Like other components, it has a run method that gets called every step in your MAPL application. What actually happens when it is run is determined by the resource file. The graphic below illustrates where ExtData fits in the MAPL application.

a NetCDF file on disk. The component receives a list of fields that need to be supplied with data

and parses the resource file to determine if MAPL_ExtDataGridCompMod can supply a variable of that name.

We will refer to each field name-file variable combination as a primary export of ExtData.

announcement that MAPL_ExtDataGridCompMod is capable of filling a field named A with data contained in variable B on file xyz.

The resource file should be viewed as an announcement of what MAPL_ExtDataGridCompMod can provide, not what it necessarily will provide.

disk and how the data is organized. This update could be at every step, just once when starting the model run, or at a particular

time each days. It also allows tremendous flexibility as to how the user chooses to organize the data files. MAPL_ExtDataGridCompMod also allows data to be shifted, scaled, and control what method is used to regrid the file to the application grid.

=== Primary Exports - Before Heracles-5_4 ===

The first table the user can define is a PrimaryExports table. Below is an example of a PrimaryExport table. This may have as many entries as the user desires. Note that only the last entry in a line is optional.

# ---------|---------|-----|-----|------|------------ |----------------------|--------|-------|------------------------------------|-----------------------------------------------------------|

#  Export  |        |    | V  |      |            |_______ Refresh ______|____ Factors ___|________ External File _____________|______________________External File Time Data______________|

#  Name    | Units  | Dim | Loc | Clim |Conservative |    Time Template    | Offset | Scale | Variable |      Template           |  Reference Time and frequency                            |

# ---------|---------|-----|-----|------|------------ |----------------------|--------|-------|----------|-------------------------|-----------------------------------------------------------|

ALBNF          NA      xy    c      N        N              0                0.0      1.0    ALBNF    myfile.%y4%m2%d2_%h2z.nc4  2000-04-15T00:00:00P03:00

:Export Name - This is the actual name of the import in the application this entry will satisfy.

:Units - Units not currently used, just a placeholder for now, hopefully someday it might be used for automatic unit conversion for example.

::2.) Enter a refresh template of the form %y4-%m2-%h2T%h2:%n2:00 to set the recurring time to update the file. The file will be updated when the evaluated template changes. For example a template of the form %y4-%m2-%d2T12:00:00 will cause the variable to be updated at the start of a new day. Note that ExtData will use the evaluated template as the working time for reading the file and will try to interpolate to that time. So in the example of %y4-%m2-%d2T12:00:00, when the clock hits 2007-08-02T00:00:00 it will update the variable but the time it will use for reading and interpolation is 2007-08-02T12:00:00

::3.) Enter '0' to update the variable at every step. ExtData will do a linear interpolation to the current time using the available data.

:File Template - this is a grads style template describing the time structure of your data if it is broken into multiple files. If there are no tokens in the template name ExtData will assume that all the data is on one file. Note that if the data on file is at a different resolution that the application grid, the underlying I/O library ExtData uses will regrid the data to the application grid. The user can enter /dev/null to simply fill the import with zero or enter /dev/null:300.0 to set the import to a non-zero constant value.

<span style="color:#FF0000">ExtData always tries to interpolate to the current model time or evaluated time from the template using the data files you provide via the file template. To do the interpolation at any time ExtData must be able to find two times on the data series that bracket this time. These need not be on the same file as it can interpolate between data on different files. Also you should not have time on a file that lies outside the range indicated by the file name. For example if the yearly files so your file template looks like myfile_%y4.nc4 and you have a 2008 file, the 2008 file should not have any time in it that falls outside of 2008. This defeats the entire purpose of ExtData and will cause unpredictable behaviour. This as it is key to understanding how ExtData functions.</span>

This bracketing data gets updated as time advances in your application. But how does it know where to find this, especially if the data is spread across multiple files? If your file_template has no grads tokens then all data you need to span the time span your application will run on needs to be in the file. More likely if you have multiple files you specify a grads style file template. When ExtData tries to find the bracketing times, it takes the time it is trying to interpolate to and applies that to the file template. It looks for data at two times that bracket the current time in this file. If it does not find either the left or right bracket time it looks in the previous file (for the left) or the next file (for the right). How does it know what the next file is. If the user does not specify a reference time and frequency it looks at the right most token and assumes that is the frequency of the files. For example if the right most token is %d2 it assumes that files are daily so to find the right bracket time, it will advance the time by one day, re-evaluate the file template, and try to look in that file to update the right bracket time. Now what if you have a file every 3 hours for example, this logic would not work. In this case you must specify a reference time and frequency using the optional keyword after the file template to tell it explicitly when the files start and how to advance the time to find the bracketing data if the data can not be found on the current file.

Sometimes data represents a yearly climatology in that you have data that spans a year and you want ExtData to recycle this year of data. ExtData supports 12 month climatologies, i.e. one time per month. If the climatology keyword is set to yes you are telling ExtData that you have a file with 12 times spanning at year or 12 files, one per month. In either case ExtData does not use the application clock's year but replaces it with the year in the file for interpolation and wraps the data around at the end of the year.

 

 

# ---------|------------|-------------|

#  Export  | Name on    |             |

#  Name    |  File    | File       |

# ---------|------------|-------------|

The user can also specify Derived Export lists. These are import fields that are satisfied via an arithmetic expression or function using fields from the primary or mask export lists.  The user specifies an import to satisfy, an expression involving mask or derived exports, and a refresh template. The import is updated with the result of the expression. The expression can be an arithmetic expression using fields from the Primary Export lists and is evaluated using the MAPL parser component. Although it is currently not checked, the refresh template on the derived export should be the same as the primary exports in the expression. Also, the primary exports in the expression need not actually be needed to fulfil an import but can still be used in a derived expression. In this case the ExtData component adds the Primary Export to the list of fields it needs to fulfil and allocates space for it. In addition to arithmetic expressions the there two functions that are currently supported for derived expressions.

# ---------|-----------------------------------|---------------------|

#  Import |           Expression            | Refresh            |

#  Name    |                                   | Template            |

# ---------|-----------------------------------|---------------------|

CO_CH4nbeu regionmask(CO_CH4,CO_regionMask;3,9) %y4-%m2-%d2t12:00:00

CO_CH4bbbo zonemask(CO_CH4,45,90)              %y4-%m2-%d2t12:00:00

CO_CH4bbbo  boxmask(CO_CH4,minLat,maxlat,minLon,maxLon) %y4-%m2-%d2t12:00:00

UVMAG       U^2+V^2                              0