Using the ExtData component: Difference between revisions

(4 intermediate revisions by one other user not shown)

Line 1:

== Overview of the ExtData Component ==

MAPL_ExtDataGridCompMod is an internal MAPL gridded component used to fulfill imports fields in a MAPL hierarchy from NetCDF files containing gridded, geospatial data on disk. It is usually one of the three gridded components in the cap or main program of a MAPL application, the others being the root of the MAPL hierarchy it is servicing and MAPL_HistoryGridCompMod. It is instantiated and all its registered methods are run automatically by the CAP. MAPL_ExtDataGridCompMod will provide data to fields in the Import states of MAPL components that are not satisfied by a MAPL_AddConnectivity call in the MAPL

Line 55:

Line 57:

=== Primary Exports - Heracles-5_4 on ===

The first table the user can define is a PrimaryExports table. Below is an example of a PrimaryExport table. This may have as many entries as the user desires. Note that only the last entry in a line is optional.

<pre>

PrimaryExports%%

# ---------|---------|------|------------ |----------------------|--------|-------|------------------------------------|-----------------------------------------------------------|

# ---------|---------|--------------------|----------------------|--------|-------|----------|-------------------------|-----------------------------------------------------------|

ALBNF NA N N 0 0.0 1.0 ALBNF myfile.%y4%m2%d2_%h2z.nc4 2000-04-15T00:00:00P03:00

du001 NA N N 0 0.0 1.0 du001 /dev/null

%%

</pre>

The following is an explanation of each entry in a line.

:Export Name - This is the actual name of the import in the application this entry will satisfy.

:Units - Units not currently used, just a placeholder for now, hopefully someday it might be used for automatic unit conversion for example.

:Dimensions - xy for 2D or xyz for 3D

:Vertical location - c for center, e for edge, if 2d enter c or e but this will obviously not be used

:Clim - enter Y if the file is a 12 month climatology, if you specify it is a climatology ExtData the data can be on either one file or 12 files if they are templated appropriately with one per month. Enter N if the data is not a climatological data set. Enter a year (2008 for example) if you have a years worth of data you want to use as a climatology, but is not 12 months, for example you had multiple years worth of data but wanted to use 1 particular year worth as a climatology

:Conservative - Enter Y the data should be regridded in a mass conserving fashion through a tiling file. Enter N to use the non-conervative bilinear regridding. Enter V to regrid using the conservative voting option. Enter F;integer to regrid using the fractional conservative option.

:Refresh template - you have 3 choices

::1.) Enter '-'. In this case the field will only be updated once the first time ExtData runs

::2.) Enter a refresh template of the form %y4-%m2-%h2T%h2:%n2:00 to set the recurring time to update the file. The file will be updated when the evaluated template changes. For example a template of the form %y4-%m2-%d2T12:00:00 will cause the variable to be updated at the start of a new day. Note that ExtData will use the evaluated template as the working time for reading the file and will try to interpolate to that time. So in the example of %y4-%m2-%d2T12:00:00, when the clock hits 2007-08-02T00:00:00 it will update the variable but the time it will use for reading and interpolation is 2007-08-02T12:00:00

::3.) Enter '0' to update the variable at every step. ExtData will do a linear interpolation to the current time using the available data.

::4.) Enter Phr:mn where hr is a two digit hour and mn is a two digit minute or enter Pyear-mm-ddThr:mn where now year, mm, and dd are a year, month and day, this is an interval at which to update the variable from the start time of the clock used in the program.

:Offset - This is a factor the variable will be shifted by, if you enter "none", no shifting will be performed, [[If you do not want to shift do not put 0.0 as you will be wasting time since it will multiply by 0.0 instead of skipping the shifting]].

:Scale - This is a factor the variable will be scaled by, if you enter "none", no scaling will be performed, [[If you do not want to scale do not put 1.0 as you will be wasting time since it will multiply by 1.0 instead of skipping the scaling]].

:Variable - This is the name of the variable ON THE FILE. It need not be the same as the export name.

:File Template - this is a grads style template describing the time structure of your data if it is broken into multiple files. If there are no tokens in the template name ExtData will assume that all the data is on one file. Note that if the data on file is at a different resolution that the application grid, the underlying I/O library ExtData uses will regrid the data to the application grid. The user can enter /dev/null to simply fill the import with zero or enter /dev/null:300.0 to set the import to a non-zero constant value.

:File reference time and frequency - this keyword is optional. Also note if your data is on one file then it makes there is no point to this keyword. This entry is the time and time-interval that describes the start time and frequency of the file template you provided and has the form %y4-%m2-%d2T%h2%n2P%y4-%m2-%d2T%h2%n2 where the time before the P is a reference time and the time after the P is a time interval. Note that the year, month, and day can be left off as a unit in the time interval and it will assume that these are zero. This keyword says that the first time that the file template is good for is the reference time and there will be files using the supplied file template at the interval provided. This provides a direct way to specify if ExtData can not determine the file frequency from the template. For example if you have data every half hour to ingest, this can not be "guessed" from a file template that has a last token of a minute or you had files every 3 hours. For example, a valid entry would be 2012-01-01T21:00P03:00. This says you have a file every 3 hours from starting at 21z on 01/01/2012.

==== Refresh Template ====

Line 104:

Line 135:

==== Climatologies ====

Sometimes data represents a yearly climatology. ExtData ~~currently only~~ supports 12 month climatologies, i.e. one time per month. If the climatology keyword is set to yes you are telling ExtData that you have a file with 12 times spanning at year or 12 files, one per month. In either case ExtData does not use the application clock's year but replaces it with the year in the file for interpolation and wraps the data around at the end of the year.

Sometimes data represents a yearly climatology in that you have data that spans a year and you want ExtData to recycle this year of data. ExtData supports 12 month climatologies, i.e. one time per month. If the climatology keyword is set to yes you are telling ExtData that you have a file with 12 times spanning at year or 12 files, one per month. In either case ExtData does not use the application clock's year but replaces it with the year in the file for interpolation and wraps the data around at the end of the year.

Note in Heracles-5_4 onward this has been expanded so that you can have a climatological data set that is not just 12 timesteps. For example, suppose you had several years of daily files and you wanted to use one particular year. You can now specify a year in the keyword telling it to use that year for the interpolation. Essentially it replaces the clock year with the year you specify and correctly wraps back to the first piece of data in the year when you cross into the new year.

==== Constant Files ====

Occasionally the user might want to satisfy a variable with a data set that is constant (not in the sense that you want to set the import to the same value but you have some geospatial data with only one time on a file that has no tokens) and should not be time interpolated. If the user specifies "-" for the refresh template (update once), ExtData finds that the file template has no tokens, and that the file itself only has one time, ExtData will note this and update the variable once with this set of data.

Line 140:

Line 174:

=== Masks ===

NOTE AS OF HERACLES-5_4 THE MASK SECTION HAS BEEN ELIMINATED. YOU CAN REPLICATE THIS FUNCTIONALITY BY CHOOSING V FOR THE REGRIDDING KEYWORLD AND USING A - FOR THE REFRESH TEMPLATE.

<pre>

Line 186:

Line 223:

CO_CH4nbeu regionmask(CO_CH4,CO_regionMask;3,9) %y4-%m2-%d2t12:00:00

CO_CH4bbbo zonemask(CO_CH4,45,90) %y4-%m2-%d2t12:00:00

CO_CH4bbbo boxmask(CO_CH4,minLat,maxlat,minLon,maxLon) %y4-%m2-%d2t12:00:00

UVMAG U^2+V^2 0

%%

</pre>

If you have any issues or questions, please email the GMAO SI Team at siteam_AT_gmao.gsfc.nasa.gov

@@ Line 1: / Line 1: @@
 == Overview of the ExtData Component ==
 MAPL_ExtDataGridCompMod is an internal MAPL gridded component used to fulfill imports fields in a MAPL hierarchy from NetCDF files containing gridded, geospatial data on disk. It is usually one of the three gridded components in the cap or main program of a MAPL application, the others being the root of the MAPL hierarchy it is servicing and MAPL_HistoryGridCompMod. It is instantiated and all its registered methods are run automatically by the CAP. MAPL_ExtDataGridCompMod will provide data to fields in the Import states of MAPL components that are not satisfied by a MAPL_AddConnectivity call in the MAPL
@@ Line 55: / Line 57: @@
+=== Primary Exports - Heracles-5_4 on ===
+The first table the user can define is a PrimaryExports table. Below is an example of a PrimaryExport table. This may have as many entries as the user desires. Note that only the last entry in a line is optional.
+<pre>
+PrimaryExports%%
+# ---------|---------|------|------------ |----------------------|--------|-------|------------------------------------|-----------------------------------------------------------|
+#  Export  |         |      |             |_______ Refresh ______|____ Factors ___|________ External File _____________|______________________External File Time Data______________|
+#  Name    | Units   | Clim |Conservative |     Time Template    | Offset | Scale | Variable |      Template           |  Reference Time and frequency                             |
+# ---------|---------|--------------------|----------------------|--------|-------|----------|-------------------------|-----------------------------------------------------------|
+ALBNF           NA       N        N               0                0.0      1.0     ALBNF     myfile.%y4%m2%d2_%h2z.nc4   2000-04-15T00:00:00P03:00
+du001           NA       N        N               0                0.0      1.0     du001     /dev/null
+%%
+</pre>
+The following is an explanation of each entry in a line.
+:Export Name - This is the actual name of the import in the application this entry will satisfy.
+:Units - Units not currently used, just a placeholder for now, hopefully someday it might be used for automatic unit conversion for example.
+:Dimensions - xy for 2D or xyz for 3D
+:Vertical location - c for center, e for edge, if 2d enter c or e but this will obviously not be used
+:Clim - enter Y if the file is a 12 month climatology, if you specify it is a climatology ExtData the data can be on either one file or 12 files if they are templated appropriately with one per month. Enter N if the data is not a climatological data set. Enter a year (2008 for example) if you have a years worth of data you want to use as a climatology, but is not 12 months, for example you had multiple years worth of data but wanted to use 1 particular year worth as a climatology
+:Conservative - Enter Y the data should be regridded in a mass conserving fashion through a tiling file. Enter N to use the non-conervative bilinear regridding. Enter V to regrid using the conservative voting option. Enter F;integer to regrid using the fractional conservative option.
+:Refresh template - you have 3 choices
+::1.) Enter '-'. In this case the field will only be updated once the first time ExtData runs
+::2.) Enter a refresh template of the form %y4-%m2-%h2T%h2:%n2:00 to set the recurring time to update the file. The file will be updated when the evaluated template changes. For example a template of the form %y4-%m2-%d2T12:00:00 will cause the variable to be updated at the start of a new day. Note that ExtData will use the evaluated template as the working time for reading the file and will try to interpolate to that time. So in the example of %y4-%m2-%d2T12:00:00, when the clock hits 2007-08-02T00:00:00 it will update the variable but the time it will use for reading and interpolation is 2007-08-02T12:00:00
+::3.) Enter '0' to update the variable at every step. ExtData will do a linear interpolation to the current time using the available data.
+::4.) Enter Phr:mn where hr is a two digit hour and mn is a two digit minute or enter Pyear-mm-ddThr:mn where now year, mm, and dd are a year, month and day, this is an interval at which to update the variable from the start time of the clock used in the program.
+:Offset - This is a factor the variable will be shifted by, if you enter "none", no shifting will be performed, [[If you do not want to shift do not put 0.0 as you will be wasting time since it will multiply by 0.0 instead of skipping the shifting]].
+:Scale - This is a factor the variable will be scaled by, if you enter "none", no scaling will be performed, [[If you do not want to scale do not put 1.0 as you will be wasting time since it will multiply by 1.0 instead of skipping the scaling]].
+:Variable - This is the name of the variable ON THE FILE. It need not be the same as the export name.
+:File Template - this is a grads style template describing the time structure of your data if it is broken into multiple files. If there are no tokens in the template name ExtData will assume that all the data is on one file. Note that if the data on file is at a different resolution that the application grid, the underlying I/O library ExtData uses will regrid the data to the application grid. The user can enter /dev/null to simply fill the import with zero or enter /dev/null:300.0 to set the import to a non-zero constant value.
+:File reference time and frequency - this keyword is optional. Also note if your data is on one file then it makes there is no point to this keyword. This entry is the time and time-interval that describes the start time and frequency of the file template you provided and has the form %y4-%m2-%d2T%h2%n2P%y4-%m2-%d2T%h2%n2 where the time before the P is a reference time and the time after the P is a time interval. Note that the year, month, and day can be left off as a unit in the time interval and it will assume that these are zero. This keyword says that the first time that the file template is good for is the reference time and there will be files using the supplied file template at the interval provided. This provides a direct way to specify if ExtData can not determine the file frequency from the template. For example if you have data every half hour to ingest, this can not be "guessed" from a file template that has a last token of a minute or you had files every 3 hours. For example, a valid entry would be 2012-01-01T21:00P03:00. This says you have a file every 3 hours from starting at 21z on 01/01/2012.
 ==== Refresh Template ====
@@ Line 104: / Line 135: @@
 ==== Climatologies ====
-Sometimes data represents a yearly climatology. ExtData currently only supports 12 month climatologies, i.e. one time per month. If the climatology keyword is set to yes you are telling ExtData that you have a file with 12 times spanning at year or 12 files, one per month. In either case ExtData does not use the application clock's year but replaces it with the year in the file for interpolation and wraps the data around at the end of the year.
+Sometimes data represents a yearly climatology in that you have data that spans a year and you want ExtData to recycle this year of data. ExtData  supports 12 month climatologies, i.e. one time per month. If the climatology keyword is set to yes you are telling ExtData that you have a file with 12 times spanning at year or 12 files, one per month. In either case ExtData does not use the application clock's year but replaces it with the year in the file for interpolation and wraps the data around at the end of the year.
+Note in Heracles-5_4 onward this has been expanded so that you can have a climatological data set that is not just 12 timesteps. For example, suppose you had several years of daily files and you wanted to use one particular year. You can now specify a year in the keyword telling it to use that year for the interpolation. Essentially it replaces the clock year with the year you specify and correctly wraps back to the first piece of data in the year when you cross into the new year.
 ==== Constant Files ====
 Occasionally the user might want to satisfy a variable with a data set that is constant (not in the sense that you want to set the import to the same value but you have some geospatial data with only one time on a file that has no tokens) and should not be time interpolated. If the user specifies "-" for the refresh template (update once), ExtData finds that the file template has no tokens, and that the file itself only has one time, ExtData will note this and update the variable once with this set of data.
@@ Line 140: / Line 174: @@
 === Masks ===
+NOTE AS OF HERACLES-5_4 THE MASK SECTION HAS BEEN ELIMINATED. YOU CAN REPLICATE THIS FUNCTIONALITY BY CHOOSING V FOR THE REGRIDDING KEYWORLD AND USING A - FOR THE REFRESH TEMPLATE.
 <pre>
@@ Line 186: / Line 223: @@
 CO_CH4nbeu  regionmask(CO_CH4,CO_regionMask;3,9) %y4-%m2-%d2t12:00:00
 CO_CH4bbbo  zonemask(CO_CH4,45,90)               %y4-%m2-%d2t12:00:00
+CO_CH4bbbo  boxmask(CO_CH4,minLat,maxlat,minLon,maxLon) %y4-%m2-%d2t12:00:00
 UVMAG       U^2+V^2                              0
 %%
 </pre>
+If you have any issues or questions, please email the GMAO SI Team at siteam_AT_gmao.gsfc.nasa.gov