GEOS-5

Using the datatank

Revision as of 12:31, 12 September 2014 by Bmauer (talk | contribs) (Created page with "Tutorial for datank and fln utilities ================== datank utility ================== Summary of "datank" ...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Tutorial for datank and fln utilities 

                   ==================
                     datank utility
                   ==================

Summary of "datank" commands:

These are the commands that a user will need to checkout and maintain staged data which are symbolic links to the DataTank. 

a) add:       Add file to be imported to DataTank
b) admin:     Run admin commands
c) annotate:  Add description to file log
d) commit:    Commit changed file back to DataTank
e) flco:      Wrapper for CVS checkout of FlexLinks 
f) help:      Display help information
g) log:       Display log information for a file
h) sync       Add new data from DataTank to staged directory
i) update:    Update a file from DataTank

Type "datank help 'command'" for command-specific options and arguments


Summary of "datank admin" commands:

These are the commands that an administrator will need to set up a DataTank repository. 

a) checkout:  Checkout latest version of DataTank files
b) flimp:     Wrapper for CVS import of FlexLinks 
c) help:      Display help information
d) import:    Import files and directories into DataTank
e) stage:     Stage data links from a tree of FlexLinks

Type "datank admin help 'command'" for command-specific options and arguments


Aliases for the datank commands: 

a) add:       ad
b) admin:     adm
c) annotate:  an, ann, anno
d) checkout:  co, get
e) commit:    com, comm
f) flco:      flget
g) flimp:     flim
h) help:      he
i) import:    im, imp
j) log:       lo
k) stage:     st, sta
l) update:    up, upd



                   ===============
                     fln utility
                   ===============

Summary of fln commands: 

a) create:  Create FlexLink tree from pre-existing directory tree
b) help:    Print help information
c) link:    Create a FlexLink to a remote file
d) stage:   Stage a directory tree from a tree of FlexLinks
e) update:  Update FlexLinks with changes to staged data

Type "fln help 'command'" for command-specific options and arguments

Aliases for the fln commands: 

a) create:  cr, cre
b) help:    he
c) link:    li, ln
d) stage:   st, sta
e) update:  up upd


Tutorial aliases and environment variables

These aliases and environment variables are needed in the tutorial examples which follow. 

     setenv BINDIR /home/jstassi/underconstruction/Datank
     alias datank $BINDIR/datank
     alias fln    $BINDIR/fln

     setenv FLHOME $HOME/FLNTEST
     setenv FLDIR  $FLHOME/FxLx
     setenv DLDIR  $NOBACKUP/fvInput

     setenv DATANKROOT $NOBACKUP/DATANKROOT

Note: The DATANKROOT environment variable can be set in the datank command with the -d flag.

Tutorial, Part I (Set up the DataTank)

Many commands include a -q flag option for quiet mode if you want to reduce the amount of verbiage going to standard output.

1. Import data from a pre-existing directory tree into the DataTank. We will 

  exclude *.hdf files from this import.  The command will prompt you for a log
  message to use for the import, or you can add a message on the command line
  using the -m flag.

     cd /discover/nobackup/projects/gmao/share/dao_ops/fvInput_nc3/g5chem/L72
     datank admin import g5chem/L72 -I \.hdf -m"imported from gmao/share"

2. Checkout data from the DataTank to the DTemp directory. This creates a 

  directory tree of symbolic links pointing to files in the DataTank.  Normally,
  this tree would be created from a set of flexlinks, but in this case we are
  going to use checked out symbolic links to create the initial set of
  flexlinks.  This is why we are calling it "DTemp".

     cd $NOBACKUP
     datank admin co -d DTemp

3. Create an initial set of flexlinks which point to the DataTank files. Use 

  the -A and -P flags (anchors and replacement patterns) to place the
  $DATANKROOT variable in the flexlink definition.  Put flexlinks in a
  directory named "FL".

     fln admin cr DTemp FLTemp -A $DATANKROOT -P \$DATANKROOT

4. Import the flexlinks into a CVS repository. 

     cd $NOBACKUP/FLTemp
     datank admin flimp -m"import flexlinks into CVS repository"

5. The set-up is complete. Clean the directories and check out a copy of the 

  flexlinks.

     cd $NOBACKUP
     \rm -rf DTemp FLTemp


Tutorial, Part II (Check out FlexLinks, Stage DataTank)

1. Check out the FlexLinks; Note that FLDIR is defined above. 

     mkdir -p $FLDIR
     datank flco -d $FLDIR

2. Stage the DataTank data to a directory named "fvInput". 

     datank stage $FLDIR $DLDIR


Tutorial, Part III (Add a data file to the DataTank)

1. Create a new file in the fvInput/g5chem/L72 directory. This is just a dummy 

  file created with an echo command, but it will be used to demonstrate how to
  add and change files in the DataTank.

     cd $DLDIR/g5chem/L72
     echo "this is a new file" > newfile.txt

2. Add the new file to the DataTank repository. 

     datank add newfile.txt
     datank commit newfile.txt

  You will be prompted to supply a log message.  This can also be given with
  the -m flag with the datank commit command.

  Note: newfile.txt in your directory is now a link pointing to the revision
  that was just placed in the DataTank. The original file has been renamed with
  a trailing tilde (~).

3. Show the file's log information. 

     datank log newfile.txt

4. Update flexlinks to include the new files. 

     fln update $DLDIR $FLDIR


Tutorial, Part IV (Make modifications to a file)

1. Create a new version of newfile.txt. Commit the change to the DataTank. 

     cd $DLDIR/g5chem/L72/
     \rm newfile.txt
     echo "new contents" > newfile.txt
     datank commit -m "file now contains new contents" newfile.txt
     datank log newfile.txt

2. Make another new version of newfile.txt 

     \rm newfile.txt
     echo "junk" > newfile.txt
     datank commit -m"file now contains junk" newfile.txt
     datank log newfile.txt

3. Update the local version of newfile.txt to revision 2. 

     datank update -r2 newfile.txt

4. Update flexlinks to reflect the latest changes in staged data directory. 

     fln upd $DLDIR $FLDIR


Tutorial, Part V (Add a new subdirectory to the DataTank)

1. Use the datank import command to import new files to a new subdirectory 

  into the DataTankRoot repository.

     cd /discover/nobackup/projects/gmao/share/dao_ops/fvInput_nc3/g5chem/sfc
     datank admin imp -m"imported from g5chem/sfc" g5chem/sfc/ gocart.dust_source.*.nc

2. Synchronize the staged data with the DataTank. This allows user to update new 

  directories and files in the staged data directory. All previously staged
  data will not be affected.

     cd $DLDIR
     datank sync

3. Add a log description for the new files. 

     cd g5chem/sfc
     datank log gocart*
     datank annotate -m"gocart related data" gocart*
     datank log gocart*

4. Update flexlinks to reflect the latest changes in staged data directory. 

     fln upd $DLDIR $FLDIR
Retrieved from "https://geos5.org/wiki/index.php?title=Using_the_datatank&oldid=2568"