TOC

Data Access

Data Flow

There are 2 independent data flows that work in conjunction throughout the entirety of the project.

Raw Data

The Raw Data (link to EOL's Definition) comprises all the engineering and low level bits assembled by the data logger, operator documentation, troubleshooting notes, pictures, and some chat logs.  This information is retained by EOL in order to properly Quality Control data, and apply needed meta data to soundings. Generally this is not dispensed to PIs, and is only used within EOL to create first pass data for use by others.

Basics

The raw data is shoveled around via a combination of rsync (when working with the aircraft) and Resillio (over the internet)

  • The AVAPS operation is responsible for renaming folder on 10.3.1.110 after the flight or tests are over. EG, if ground tests are performed on Aug 12, 2017, the folder "2017-08-12" needs to be renamed to "2017-08-12-ground-test" and moved into a "groundTest" folder.
  • rsync is running as a daemon on 10.3.1.110, which is the avaps data logger.  rsync://10.3.1.110/dropsondes provides read / write access to C:\AVAPSData.

  • A simple script and a shortcut icon on the desktop does a slightly fancier version of the following to pull data from 10.3.1.110 to the laptop.

    Laptop Sync Script
    export PROJECT=EPOCH-2017
rsync -rv rsync://10.3.1.110/dropsondes ~/AVAPSData-Projects/${PROJECT}/
  • The AVAPS Operator is responsible for collecting the AGS data off the computers in the GHOC and placing it in the correct flight folder.

  • There is a system service on the laptop and barolo.eol.ucar.edu that perform a two way sync.  Data added on the laptop will eventually be on barolo, and vise versa.

  • Barolo.eol.ucar.edu places the synced data into /net/isf/avaps-incoming

  • There is a system service running on isf-proc.eol.ucar.edu that perform the postflight:

    • Once every 5 minutes, it checks /net/isf/avaps-incoming for new data

    • If the there is new data, it either:

      • complains if there are errors (missing data, etc)

      • creates or re-creates new postflight plots

    • The created files eventually will be transfered to the laptop in the field

Diagram

epoch-raw-data-flow

There are 2 independent data flows that work in conjunction throughout the entirety of the project.

Operational Data

Operational Data (link to EOL's Definition) comprises the data that is used to compute first pass QC data and items that have been submitted to model runs. Usually this comprises a D_P files, and Aspen output EOL, TEMPDROP, and BUFR formats

Basics

Due to firewalls on both NCAR, NOAA, and NASA, this is a bit more convoluted, but does follow a linear path.

  • D_P files is created at the completion of a sounding
  • New D_P file is detected by ftp-agent on 10.3.1.110, and is ftp'd to ftp://130.30.161.70/avaps_data over Ku, directly to the so called 'linkserver'
  • On the linkserver, there is a simple script that rsyncs the contents of ~/avaps_data to avaps@asp-interface-2.arc.nasa.gov:/home/avaps/avaps_data every 30 seconds

    • This service must be restarted every time the linkserver is rebooted

    • This typically runs in a screen session
  • asp-interface-2.arc.nasa.gov runs a rsync instance that allows access utilizing SSH keys.
  • NOAA runs a script that does the following tasks:
    • Polls asp-interface-2.arc.nasa.gov for new D_P files
      • FTPs the new files to NCAR
      • Posts the D_P files somewhere inside the NOAA domain for HRD folks to process
    • Check for Aspen outputs:
      • FTP's outputs to NCAR
Diagram

operational-data-epoch

Recently Updated

  • No labels