Using the ExtData component: Difference between revisions

Line 28:

<pre>

PrimaryExports%%

# ---------|---------|-----|-----|------|------------ |----------------------|--------|-------|---------------------------------|-----------------------------------------------------------|

# ---------|---------|-----|-----|------|------------ |----------------------|--------|-------|------------------------------------|-----------------------------------------------------------|

# ---------|---------|-----|-----|------|------------ |----------------------|--------|-------|----------|----------------------|-----------------------------------------------------------|

# ---------|---------|-----|-----|------|------------ |----------------------|--------|-------|----------|-------------------------|-----------------------------------------------------------|

ALBNF NA xy c N N 0 0.0 1.0 ALBNF myfile.%y4%m2%d2.nc4 2000-04-15T00:00:00P03:00

ALBNF NA xy c N N 0 0.0 1.0 ALBNF myfile.%y4%m2%d2_%h2z.nc4 2000-04-15T00:00:00P03:00

du001 NA xyz c N N 0 0.0 1.0 du001 /dev/null

%%

Line 51:

:Scale - This is a factor the variable will be scaled by, if you enter "none", no scaling will be performed, [[If you do not want to scale do not put 1.0 as you will be wasting time since it will multiply by 1.0 instead of skipping the scaling]].

:Variable - This is the name of the variable ON THE FILE. It need not be the same as the export name.

:File Template - this is a grads style template, if there are no tokens in the template name ExtData will assume that all the data is on one file. Note that if the data on file is at a different resolution that the application grid, the underlying I/O library ExtData uses will regrid the data to the application grid. The user can enter /dev/null to simply fill the import with zero or enter /dev/null:300.0 to set the import to a non-zero constant value.

:File Template - this is a grads style template describing the time structure of your data if it is broken into multiple files. If there are no tokens in the template name ExtData will assume that all the data is on one file. Note that if the data on file is at a different resolution that the application grid, the underlying I/O library ExtData uses will regrid the data to the application grid. The user can enter /dev/null to simply fill the import with zero or enter /dev/null:300.0 to set the import to a non-zero constant value.

:File reference time and frequency - this keyword is optional. Also note if your data is on one file then it makes there is no point to this keyword. This entry is the time and time-interval that describes the start time and frequency of the file template you provided and has the form %y4-%m2-%d2T%h2%n2P%y4-%m2-%d2T%h2%n2 where the time before the P is a reference time and the time after the P is a time interval. Note that the year, month, and day can be left off as a unit in the time interval and it will assume that these are zero. This keyword says that the first time that the file template is good for is the reference time and there will be files using the supplied file template at the interval provided. This provides a direct way to specify if ExtData can not determine the file frequency from the template. For example if you have data every half hour to ingest, this can not be "guessed" from a file template that has a last token of a minute or you had files every 3 hours. For example, a valid entry would be 2012-01-01T21:00P03:00. This says you have a file every 3 hours from starting at 21z on 01/01/2012.

Line 65:

If you enter '0' for the refresh template the field gets updated every step and ExtData will try to interpolate to the application time using the data on file.

<span style="color:#FF0000">ExtData always tries to interpolate to the current model time or evaluated time from the template using the data files you provide via the file template. To do the interpolation at any time ExtData must be able to find two times on the data series that bracket this time. These need not be on the same file as it can interpolate between data on different files. Also you should not have time on a file that lies outside the range indicated by the file name. For example if the yearly files so your file template looks like myfile_%y4.nc4 and you have a 2008 file, the 2008 file should not have any time in it that falls outside of 2008. This defeats the entire purpose of ExtData and will cause unpredictable ~~behaivor~~. This as it is key to understanding how ExtData functions.</span>

<span style="color:#FF0000">ExtData always tries to interpolate to the current model time or evaluated time from the template using the data files you provide via the file template. To do the interpolation at any time ExtData must be able to find two times on the data series that bracket this time. These need not be on the same file as it can interpolate between data on different files. Also you should not have time on a file that lies outside the range indicated by the file name. For example if the yearly files so your file template looks like myfile_%y4.nc4 and you have a 2008 file, the 2008 file should not have any time in it that falls outside of 2008. This defeats the entire purpose of ExtData and will cause unpredictable behaviour. This as it is key to understanding how ExtData functions.</span>

This bracketing data gets updated as time advances in your application. But how does it know where to find this, especially if the data is spread across multiple files? If your file_template has no grads tokens ~~to evaluate~~ then all data you need to span the time span your application will run on needs to be in the file. More likely if you have multiple files you specify a grads style file template. When ExtData tries to find the bracketing times, it takes the time it is trying to interpolate to and applies that to the file template. It looks for data at two times that bracket the current time. If it does not find either the left or right bracket time it looks in the previous file (for the left) or the next file (for the right). How does it know what the next file is. If the user does not specify a reference time and frequency it looks at the right most token and assumes that is the frequency of the files. For example if the right most token is %d2 it assumes that files are daily so to find the right bracket time, it will advance the time by one day and try to look in that file to update the right bracket time. Now what if you have a file every 3 hours for example, this logic would not work. In this case you must specify a reference time and frequency using the optional keyword after the file template to tell it explicitly when the files start and how to advance the time to find the bracketing data.

This bracketing data gets updated as time advances in your application. But how does it know where to find this, especially if the data is spread across multiple files? If your file_template has no grads tokens then all data you need to span the time span your application will run on needs to be in the file. More likely if you have multiple files you specify a grads style file template. When ExtData tries to find the bracketing times, it takes the time it is trying to interpolate to and applies that to the file template. It looks for data at two times that bracket the current time in this file. If it does not find either the left or right bracket time it looks in the previous file (for the left) or the next file (for the right). How does it know what the next file is. If the user does not specify a reference time and frequency it looks at the right most token and assumes that is the frequency of the files. For example if the right most token is %d2 it assumes that files are daily so to find the right bracket time, it will advance the time by one day, re-evaluate the file template, and try to look in that file to update the right bracket time. Now what if you have a file every 3 hours for example, this logic would not work. In this case you must specify a reference time and frequency using the optional keyword after the file template to tell it explicitly when the files start and how to advance the time to find the bracketing data if the data can not be found on the current file.

The following picture illustrates a simple example of how this works for a simple case where you have data on hourly files.

@@ Line 28: / Line 28: @@
 <pre>
 PrimaryExports%%
-# ---------|---------|-----|-----|------|------------ |----------------------|--------|-------|---------------------------------|-----------------------------------------------------------|
+# ---------|---------|-----|-----|------|------------ |----------------------|--------|-------|------------------------------------|-----------------------------------------------------------|
-#  Export  |         |     | V   |      |             |_______ Refresh ______|____ Factors ___|________ External File __________|______________________External File Time Data______________|
+#  Export  |         |     | V   |      |             |_______ Refresh ______|____ Factors ___|________ External File _____________|______________________External File Time Data______________|
-#  Name    | Units   | Dim | Loc | Clim |Conservative |     Time Template    | Offset | Scale | Variable |      Template        |  Reference Time and frequency                             |
+#  Name    | Units   | Dim | Loc | Clim |Conservative |     Time Template    | Offset | Scale | Variable |      Template           |  Reference Time and frequency                             |
-# ---------|---------|-----|-----|------|------------ |----------------------|--------|-------|----------|----------------------|-----------------------------------------------------------|
+# ---------|---------|-----|-----|------|------------ |----------------------|--------|-------|----------|-------------------------|-----------------------------------------------------------|
-ALBNF           NA      xy    c      N        N               0                0.0      1.0     ALBNF     myfile.%y4%m2%d2.nc4   2000-04-15T00:00:00P03:00
+ALBNF           NA      xy    c      N        N               0                0.0      1.0     ALBNF     myfile.%y4%m2%d2_%h2z.nc4   2000-04-15T00:00:00P03:00
 du001           NA      xyz   c      N        N               0                0.0      1.0     du001     /dev/null
 %%
@@ Line 51: / Line 51: @@
 :Scale - This is a factor the variable will be scaled by, if you enter "none", no scaling will be performed, [[If you do not want to scale do not put 1.0 as you will be wasting time since it will multiply by 1.0 instead of skipping the scaling]].
 :Variable - This is the name of the variable ON THE FILE. It need not be the same as the export name.
-:File Template - this is a grads style template, if there are no tokens in the template name ExtData will assume that all the data is on one file. Note that if the data on file is at a different resolution that the application grid, the underlying I/O library ExtData uses will regrid the data to the application grid. The user can enter /dev/null to simply fill the import with zero or enter /dev/null:300.0 to set the import to a non-zero constant value.
+:File Template - this is a grads style template describing the time structure of your data if it is broken into multiple files. If there are no tokens in the template name ExtData will assume that all the data is on one file. Note that if the data on file is at a different resolution that the application grid, the underlying I/O library ExtData uses will regrid the data to the application grid. The user can enter /dev/null to simply fill the import with zero or enter /dev/null:300.0 to set the import to a non-zero constant value.
 :File reference time and frequency - this keyword is optional. Also note if your data is on one file then it makes there is no point to this keyword. This entry is the time and time-interval that describes the start time and frequency of the file template you provided and has the form %y4-%m2-%d2T%h2%n2P%y4-%m2-%d2T%h2%n2 where the time before the P is a reference time and the time after the P is a time interval. Note that the year, month, and day can be left off as a unit in the time interval and it will assume that these are zero. This keyword says that the first time that the file template is good for is the reference time and there will be files using the supplied file template at the interval provided. This provides a direct way to specify if ExtData can not determine the file frequency from the template. For example if you have data every half hour to ingest, this can not be "guessed" from a file template that has a last token of a minute or you had files every 3 hours. For example, a valid entry would be 2012-01-01T21:00P03:00. This says you have a file every 3 hours from starting at 21z on 01/01/2012.
@@ Line 65: / Line 65: @@
 If you enter '0' for the refresh template the field gets updated every step and ExtData will try to interpolate to the application time using the data on file.
-<span style="color:#FF0000">ExtData always tries to interpolate to the current model time or evaluated time from the template using the data files you provide via the file template. To do the interpolation at any time ExtData must be able to find two times on the data series that bracket this time. These need not be on the same file as it can interpolate between data on different files. Also you should not have time on a file that lies outside the range indicated by the file name. For example if the yearly files so your file template looks like myfile_%y4.nc4 and you have a 2008 file, the 2008 file should not have any time in it that falls outside of 2008. This defeats the entire purpose of ExtData and will cause unpredictable behaivor. This as it is key to understanding how ExtData functions.</span>
+<span style="color:#FF0000">ExtData always tries to interpolate to the current model time or evaluated time from the template using the data files you provide via the file template. To do the interpolation at any time ExtData must be able to find two times on the data series that bracket this time. These need not be on the same file as it can interpolate between data on different files. Also you should not have time on a file that lies outside the range indicated by the file name. For example if the yearly files so your file template looks like myfile_%y4.nc4 and you have a 2008 file, the 2008 file should not have any time in it that falls outside of 2008. This defeats the entire purpose of ExtData and will cause unpredictable behaviour. This as it is key to understanding how ExtData functions.</span>
-This bracketing data gets updated as time advances in your application. But how does it know where to find this, especially if the data is spread across multiple files? If your file_template has no grads tokens to evaluate then all data you need to span the time span your application will run on needs to be in the file. More likely if you have multiple files you specify a grads style file template. When ExtData tries to find the bracketing times, it takes the time it is trying to interpolate to and applies that to the file template. It looks for data at two times that bracket the current time. If it does not find either the left or right bracket time it looks in the previous file (for the left) or the next file (for the right). How does it know what the next file is. If the user does not specify a reference time and frequency it looks at the right most token and assumes that is the frequency of the files. For example if the right most token is %d2 it assumes that files are daily so to find the right bracket time, it will advance the time by one day and try to look in that file to update the right bracket time. Now what if you have a file every 3 hours for example, this logic would not work. In this case you must specify a reference time and frequency using the optional keyword after the file template to tell it explicitly when the files start and how to advance the time to find the bracketing data.
+This bracketing data gets updated as time advances in your application. But how does it know where to find this, especially if the data is spread across multiple files? If your file_template has no grads tokens then all data you need to span the time span your application will run on needs to be in the file. More likely if you have multiple files you specify a grads style file template. When ExtData tries to find the bracketing times, it takes the time it is trying to interpolate to and applies that to the file template. It looks for data at two times that bracket the current time in this file. If it does not find either the left or right bracket time it looks in the previous file (for the left) or the next file (for the right). How does it know what the next file is. If the user does not specify a reference time and frequency it looks at the right most token and assumes that is the frequency of the files. For example if the right most token is %d2 it assumes that files are daily so to find the right bracket time, it will advance the time by one day, re-evaluate the file template, and try to look in that file to update the right bracket time. Now what if you have a file every 3 hours for example, this logic would not work. In this case you must specify a reference time and frequency using the optional keyword after the file template to tell it explicitly when the files start and how to advance the time to find the bracketing data if the data can not be found on the current file.
 The following picture illustrates a simple example of how this works for a simple case where you have data on hourly files.