GEOS-5

G5NR Data Access Guide: Difference between revisions

← Older edit
Pchakrab (talk | contribs)
1,048 edits
FTP is not supported anymore
 
(178 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{rightTOC}}
{{rightTOC}}
For questions or comments please send an email to g5nr at lists dot nasa dot gov.
For questions or comments please send an email to g5nr at lists dot nasa dot gov.


== Background ==
== G5NR background ==
 
==== File spec ====
 
The G5NR data files are organized into ''collection''s with each collection containing a specific group of variables (geophysical quantities). Each collection, as its name suggests, contains either ''instantaneous'' or ''time-averaged'' products, but not both. For more details, see [link].
 
The G5NR data files are generated using the NetCDF-4 library [link] which uses HDF-5 [link] as the underlying format.
 
==== Model config ====
 
== Getting data ==
 
==== ftp ====
 
[NOTE: THIS IS FOR TRIAL2 DATA]
 
The ftp location for G5NR data is ftp://ftp.nccs.nasa.gov/c1440_NR-Trial2
To access G5NR data via ftp one can either
* browser - use a browser to connect to the ftp site. The data is
 
==== Download tool ====
 
==== opendap ====
 
== Client access ==
 
In the following, we read the field 'T' (air temperature) from collection http://opendap.nccs.nasa.gov:9090/dods/OSSE/GEOS-5.12/BETA9/0.5000_deg/inst/inst01hr_3d_T_Cv.
 
==== Programming ====
 
These are simple programs to read the air temperature and compute its min/max. These codes require an '''OPeNDAP enabled NetCDF-4 library'''. The utility <code>nc-config</code> (<code>nf-config</code> for Fortran) bundled with the NetCDF-4 installation can be used to determine the necessary compiler flags. Individually downloaded files can be read similarly.


===== C =====
The GEOS-5 Nature Run (Ganymed release) is a 2-year global, non-hydrostatic mesoscale simulation for the period 2005-2006. In addition to standard meteorological parameters (wind, temperature, moisture, surface pressure), this simulation includes 15 aerosol tracers (dust, seasalt, sulfate, black and organic carbon), O3, CO and CO2. This model simulation is driven by prescribed sea-surface temperature and sea-ice, daily volcanic and biomass burning emissions, as well as high-resolution inventories of anthropogenic sources.


<nowiki>
GEOS-5 files are generated with the Network Common Data Form (NetCDF-4) library, which uses Hierarchical Data Format Version 5 (HDF-5) as the underlying format. NetCDF-4 is an open-source product of UCAR/Unidata (https://www.unidata.ucar.edu/software/netcdf/) and HDF-5 is developed by the HDF Group (http://www.hdfgroup.org/). One convenient method of reading GEOS-5 files is to use the netCDF library, but the HDF-5 library can also be used directly.
#include<stdio.h>
#include<stdlib.h>
#include<netcdf.h>  // for reading NR files
 
/* Handle errors by printing an error message and exiting with a
* non-zero status. */
#define ERRCODE 2
#define ERR(e) {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}
 
int main(void){
  // file name
  char* T_file = "http://opendap.nccs.nasa.gov:9090/dods/OSSE/GEOS-5.12/BETA9/0.5000_deg/inst/inst01hr_3d_T_Cv";
 
  // netCDF ID for the file and data variable
  int ncid, varid;
   
   
  // global 4D array: (time,lev,lat,lon), one time step
Each GEOS-5 file contains a '''collection''' of geophysical quantities that we will refer to as "fields" or "variables" as well as a set of coordinate variables that contain information about the grid coordinates. The variables as well as the complete structure of the file can be quickly listed using common utilities like <code>ncdump</code> or <code>h5dump</code>.
  const int IM = 720;
  const int JM = 361;
  const int LM = 72;
  const int asyz = 1*LM*JM*IM;
  float *T = NULL;
 
  // hypercube for reading one array for a given time
  size_t start[4] = {36, 0, 0, 0}; // time step 37
  size_t count[4] = {1, LM, JM, IM}; // 1 time step, 3D (lon,lat,lev) array
 
  // return code
  int rc;
 
  // min/max values
  float minval, maxval;
 
  // misc counters
  int ctr;
 
  // allocate memory for T
  T = malloc(asyz*sizeof(float));
 
  // read the data file
  printf("Reading T..."); fflush(stdout);
  if (rc = nc_open(T_file, NC_NOWRITE, &ncid)) ERR(rc);
  if (rc = nc_inq_varid(ncid, "t", &varid)) ERR(rc);
  if (rc = nc_get_vara_float(ncid, varid, start, count, T)) ERR(rc);
  printf("done.\n"); fflush(stdout);
 
  // min/max of T
  minval = 1.0e15;
  maxval = -1.0e15;
  for (ctr=0; ctr<asyz; ctr++){
    if (T[ctr]<minval){
      minval = T[ctr];
    }
    if (T[ctr]>maxval){
      maxval = T[ctr];
    }
  }
  printf("min(T): %f\n", minval);
  printf("max(T): %f\n", maxval);
 
  // free memory
  free(T);
 
  return 0;
}
</nowiki>
 
===== Fortran =====


<nowiki>
For more details about File Spec, please see [[File:G5NR-Ganymed-7km_FileSpec-ON6-V1.0.pdf]].
program g5nr_reader


  use netcdf          ! for reading the NR files                                                                             
For model configuration, please see [[File:GMAO-OfficeNote-5-V1-22Oct2014.pdf]].


  implicit none
== Download data files ==


  !  File name                                                                                                               
==== Global data ====
  !  ---------                                                                                                               
  character(len=256) :: T_file


  ! Global, 4D array: (lon,lat,lev,time)                                                                                     
<!--
  !  ------------------------------------  
===== [[Recipe: Retrieve (global) data from FTP server|FTP]] =====
  real, pointer :: T(:,:,:,:) => null()
-->


  !  Miscellaneous                                                                                                             
===== [[Recipe: Retrieve (global) data from HTTPS server|HTTPS]] =====
  !  -------------                                                                                                             
  integer :: ierr
  integer :: im, jm, lm
  integer :: ncid, varid
  integer :: start(4), count(4)


  !  For now hard code file name and dimensions                                                                               
==== Data subsets ====
  !  ------------------------------------------                                                                               
===== [[Recipe: Retrieve data subsets using download tool|Download tool]] =====
  im = 720
  jm = 361
  lm = 72
  T_file = "http://opendap.nccs.nasa.gov:9090/dods/OSSE/GEOS-5.12/BETA9/0.5000_deg/inst/inst01hr_3d_T_Cv"


  !  Allocate the Global 4-D array with only 1 time level                                                                     
== Read downloaded data files ==
  !  ----------------------------------------------------                                                                     
==== [[Recipe: Fortran program to read data from downloaded file|Fortran program]] ====
  allocate(T(im,jm,lm,1))
==== [[Recipe: C program to read data from downloaded file|C program]] ====
==== [[Recipe: Python program to read data from downloaded file|Python script]] ====
==== [[Recipe: Matlab program to read data from downloaded file|Matlab script]] ====
==== [[Recipe: IDL program to read data from downloaded file|IDL script]] ====
==== [[Recipe: Visualize downloaded data using Panoply|Panoply]] ====


  !  Hypercube for reading one 1 array for a given time                                                                       
== OPeNDAP access ==
  !  -------------------------------------------------                                                                         
  start = (/  1,  1,  1, 37 /)  ! time level 37                                                                               
  count = (/ im, jm, lm, 1  /)  ! 1 time level, 3D (lon,lat,lev) array                                                       


  !  Read the data file                                                                                                        
OPeNDAP is a data server architecture that allows users to use data files that are stored on remote computers with their favorite analysis and visualization tools. Opening an OPeNDAP file is as easy replacing the file name in the client software by an OPeNDAP URL. All G5NR collections that are provided by https/download-tool are also available on the OPeNDAP server
  !  ------------------                                                                                                       
  write(*,*)'Reading T'
  call check( nf90_open(T_file,NF90_NOWRITE,ncid), "opening T file")
  call check( nf90_inq_varid(ncid,"t",varid), "getting T varid")
  call check( nf90_get_var(ncid,varid,T,start=start,count=count), "reading T")
  call check( nf90_close(ncid), "closing T file")


  ! Orint min/max of arrays                                                                                                   
  https://opendap.nccs.nasa.gov/dods/OSSE/G5NR/Ganymed/7km
  !  -----------------------                                                                                                   
  write(*,*)'T: ', maxval(T),minval(T)


  !  All done                                                                                                                 
===== [[Recipe: Fortran program as OPeNDAP client|Fortran client]] =====
  !  --------                                                                                                                 
===== [[Recipe: C program as OPeNDAP client|C client]] =====
 
===== [[Recipe: Python program as OPeNDAP client|Python client]] =====
contains
===== [[Recipe: Matlab program as OPeNDAP client|Matlab client]] =====
 
===== [[Recipe: IDL program as OPeNDAP client|IDL client]] =====
  subroutine check(status, loc)
===== [[Recipe: Visualize OPeNDAP data using Panoply|Panoply]] =====
 
<!--
    integer, intent(in) :: status
    character(len=*), intent(in) :: loc
 
    if(status /= NF90_NOERR) then
      write (*,*) "Error at ", loc
      write (*,*) NF90_STRERROR(status)
    end if
 
  end subroutine check
 
end program g5nr_reader
</nowiki>


===== Shmem example =====
===== Shmem example =====
==== Free clients ====
In this section we read air temperature, compute it min/max (as with the 'programming' examples) and display the surface air temperature.
===== Python =====
====== netcdf4-python ======
If netcdf4-python module is available, the following script would read air temperature for the specified time, compute its min and max values and plot it.
<nowiki>
#!/usr/bin/env python                                                                               
import sys
import numpy as np
import netCDF4 as nc4
import matplotlib.pyplot as plt                                                                     
                                                                                                     
from mpl_toolkits.basemap import Basemap                                                             
rootgrp = nc4.Dataset('http://opendap.nccs.nasa.gov:9090/dods/OSSE/GEOS-5.12/BETA9/0.5000_deg/inst/inst01hr_3d_T_Cv', 'r')
# read air temperature                                                                               
print 'Reading T for time=37...',; sys.stdout.flush()
Ttime37 = rootgrp.variables['t'][36,:,:,:]
print 'done.'; sys.stdout.flush()
# min/max                                                                                           
print 'min(T):', np.min(Ttime37)
print 'max(T):', np.max(Ttime37)
# set up cylindrical map                                                                             
m = Basemap(                                                                                         
    projection='cyl',                                                                               
    llcrnrlat=-90, urcrnrlat=90,                                                                     
    llcrnrlon=-180, urcrnrlon=180,                                                                   
    resolution='c'                                                                                   
    )                                                                                               
m.drawcoastlines(linewidth=0.5)                                                                     
m.drawmapboundary()                                                                                 
                                                                                                     
# plot contour                                                                                       
level = 71                                                                                           
X = np.arange(-180.0, 180.0, .5)                                                                     
Y = np.arange(-90.0, 90.1, .5) # 90 is the last element                                             
cp = plt.contour(X, Y, T[0,level,:,:], 20, zorder=2)                                                 
plt.clabel(cp, inline=1, fontsize=9)                                                                 
plt.title('Air temperature at the surface')                                                         
plt.show()                                                                                           
</nowiki>
====== pygrads ======


===== R =====
===== R =====


This example requires the [http://cran.r-project.org/web/packages/ncdf4/index.html ncdf4] and [http://cran.r-project.org/web/packages/rworldmap/index.html rworldmap] packages.
This example requires the [https://cran.r-project.org/web/packages/ncdf4/index.html ncdf4] and [https://cran.r-project.org/web/packages/rworldmap/index.html rworldmap] packages.


<nowiki>
<syntaxhighlight lang="rsplus">
> library(ncdf4)
> library(ncdf4)
> library(rworldmap)
> library(rworldmap)
Line 244: Line 64:
> jm <- 361
> jm <- 361
> lm <- 72
> lm <- 72
> nc <- nc_open("http://opendap.nccs.nasa.gov:9090/dods/OSSE/GEOS-5.12/BETA9/0.5000_deg/inst/inst01hr_3d_T_Cv")
> nc <- nc_open("https://opendap.nccs.nasa.gov:9090/dods/OSSE/GEOS-5.12/BETA9/0.5000_deg/inst/inst01hr_3d_T_Cv")
< t <- ncvar_get(nc,"t",start=c(1,1,1,37),count=c(im,jm,lm,1))
< t <- ncvar_get(nc,"t",start=c(1,1,1,37),count=c(im,jm,lm,1))
> str(t)
> str(t)
Line 252: Line 72:
   179.7  220.9  241.7  243.4  265.3  316.7  
   179.7  220.9  241.7  243.4  265.3  316.7  
> mapGriddedData(t[1:im,1:jm,71])
> mapGriddedData(t[1:im,1:jm,71])
</nowiki>
</syntaxhighlight>


===== NCL =====
===== NCL =====
Line 258: Line 78:
===== IDV =====
===== IDV =====


[http://www.unidata.ucar.edu/software/idv/ IDV] is an OPeNDAP tool that can access and display the nature run data. In our OPenDAP server, all files are time aggregated, so they appear as a single dataset for each location.
[https://www.unidata.ucar.edu/software/idv/ IDV] is an OPeNDAP tool that can access and display the nature run data. In our OPenDAP server, all files are time aggregated, so they appear as a single dataset for each location.


This is an example to open and display the field 'T' (air temperature) from the collection 'inst01hr_3d_T_Cv'. The OPenDAP URL for this dataset is http://opendap.nccs.nasa.gov:80/dods/OSSE/GEOS-5.12/BETA9/0.5000_deg/inst/inst01hr_3d_T_Cv. The following steps are valid for IDV version 5.0u1 running on a Linux desktop.
This is an example to open and display the field 'T' (air temperature) from the collection 'inst01hr_3d_T_Cv'. The OPenDAP URL for this dataset is https://opendap.nccs.nasa.gov:80/dods/OSSE/GEOS-5.12/BETA9/0.5000_deg/inst/inst01hr_3d_T_Cv. The following steps are valid for IDV version 5.0u1 running on a Linux desktop.


From the 'Dashboard' panel
From the 'Dashboard' panel
Line 268: Line 88:
* Select Field Selector and choose the 3D field'air_temperature'. The 'Times' tab lists all the available levels and times for this data. At this point, one can select specific times, level and regions (subsetting) from the 'Times' and 'Level' and 'Region' tabs. Click on 'Create Display'.
* Select Field Selector and choose the 3D field'air_temperature'. The 'Times' tab lists all the available levels and times for this data. At this point, one can select specific times, level and regions (subsetting) from the 'Times' and 'Level' and 'Region' tabs. Click on 'Create Display'.


==== Proprietary clients ====
===== Matlab =====


===== IDL =====
-->
Retrieved from "https://geos5.org/wiki/index.php?title=G5NR_Data_Access_Guide"