Data Analysis Services Group - April 2013
News and Accomplishments
VAPOR Project
Project information is available at: http://www.vapor.ucar.edu
XD Vis Award:
With support from our ASD award John and Alan attended the European Geosciences Union meeting in Vienna and presented two talks and a poster on VAPOR.
We also continued planning for refactoring several components of the vaporgui code base.
KISTI Award:
Negotiations with KISTI for the Year Three award continued. The statement of work has been agreed upon by both parties. At KISTI's request we are now attempting to change the contract type from "contractual" to "collaborative", which would greatly improve our chances of continued funding beyond 2013. Meeting KISTI's needs for a collaborative agreement, which requires cost-sharing on CISL's part, has proved challenging due to NSF policy. However, with guidance from Al we have found a source of non-NSF funds that will allow us to meet the obligation. NCAR's contract office is now finalizing the agreement, which we hope to signed early in April.
Development:
We continue to work towards a patch release for VAPOR 2.2.0. The most significant component of this effort are revised MOM and ROMS ocean model data translators. The first generation of these translators were too restrictive in the forms of MOM and ROMS data that they would accept.
- The MOM data translator has been re-implemented and regression tested.
- Support for a variety of vertical coordinate systems was added to the CF compliant data reader. This was needed to support ROMS data.
- Work on refactoring the ROMS data translator has begun.
- We found that Sourceforge is not planning full support of Bazaar source control, so Alan moved all our recent source changes back to the CVS repository. We intend to convert this to a Subversion repository soon.
- We found that our planned conversion of Netcdf files (in vapor 2.2.2) requires support for unit conversion, such as is provided in the udunits package supported by Unidata. Unfortunately that package is not supported on Windows, so Alan has been evaluating the task of porting the udunits package to Windows.
- Alan is working on improvements to our Drupal documentation on the website that will eventually enable VAPOR users to access context-sensitive documentation directly from the vaporgui application.
We continue to respond to queries on the VAPOR mailing list, and to post and fix bugs.
Administrative:
An offer was made for our open SE II position. The applicant unfortunately declined. A new round of interviews is now underway.
Annual review "key activity" documents were completed.
John participated in the CISL strategic planning process, working with the Data Centric Working Group to develop strategies related to data handling.
Education and Outreach:
- Mel Shapiro showed Tom Bogdan the animations we have created from hurricane Sandy data. Bogdan presented this work to FEMA to demonstrate the value of higher accuracy weather prediction for emergency planning. We are waiting to see if FEMA is interested in funding a continuation of this effort.
- John and Alan presented a tutorial/demo of VAPOR at the Visualization Splinter Meeting at EGU 2013 in Vienna, and presented there a short talk entitled “Interactive analysis and visualization of massive earth system models using VAPOR”. John also presented a poster on Intelligent Data Storage for Structured Grids.
- John gave a talk at the DOECGF in Portland on VAPOR. He also was on a panel on "Big Data"
Software Research Projects
We (coauthors Pete Johnson, Mel, Shapiro, Mark Straka, Tom Galerno, Alan Norton) submitted an article for Supercomputing 2013 entitled “Petascale WRF simulation of Hurricane Sandy”, describing the computation performed on Blue Waters, visualized at NCAR using VAPOR. This article has also been submitted as an ACM Gordon Bell award finalist.
Production Visualization Services & Consulting
ASD Support
John continued working with Cornell's Peter Ireland, post-processing hundreds of TBs of particle and gridded data.
Publications, Papers & Presentations
- Paper: NCAR Storage Accounting and Analysis Possibilities, Hart, D., Gillman P., Thanhardt E. was accepted by the XSEDE 13 program committee.
Systems Projects
Data Services
- Continued work on scripts to manage Linux container templates and container instances. Installed four containers on the gladedm nodes to support the data access class of users. Reviewed data-access node documentation produced by CSG.
- Requested new X.509 host certificates for the GridFTP server hosts from SEG on April 8 due to the pending expiration of the current certificates on May 7.
Accounting & Statistics
- Created a new Glade disk usage graph for ganglia that is more accurate when looking at data back in time.
Security & Administration
- Removed an O(n^2) behavior from the AuthDB build process, speeding up the building of authorization domains from their input constituents quite a bit in certain cases.
- Updated administrative documents on the DASG wiki.
- Continued monitoring GPFS to understand normal and abnormal operating conditions and how that is reflected in the monitoring tools output.
- Continued improvement of the inactive account cleanup.
- Had to reduce the retention period for /glade/u/home backups to 3 weeks due to resource constraints on the data deduplication server. Updated GLADE user documentation to reflect this.
- Began to investigate the process for upgrading firmware on the DCS3700 units.
System Monitoring
- Began work on a script to monitor our Nagios state and report when certain combinations of outages can result in a GLADE downtime.
- Updated the Nagios configuration on dasgmon to take into account our recently decommissioned systems.
System Support
ML - Data Analysis & Visualization Clusters
- Handled the April power down for DASG production systems located at ML.- I generated new MATLAB licenses for SSG to use.
- Cleared the partition tables of the mirage systems in preparation for transferring them to a different group.
- Completed planning for the decommission of systems in the ML machine room.
- Completed planning for the re-deploy of systems in the ML machine room.
GLADE Storage Cluster
- Restored old /glade/home user directories upon request for users who failed to migrate their files to the new GLADE directories before the old GLADE storage was shutdown.
- On April 1st I unmounted the ML-GLADE filesystems from all clients and prevented them from being mounted ever again.
- Worked with VETS to move several of their GLADE clients to their new cluster at NWSC.
Data Transfer Cluster
- Notified CSG the data-access nodes are ready for testing.
Test Clusters
- Reconfigured storm1 into a Ubuntu test system for the VAPOR team. Renamed vapor1.ucar.edu
- Installed GPFS ptf20 on picnicbasket and provided the GPL-layer RPMs to SSG for testing.
Storage Usage Statistics
dasg:NWSC+GLADE+Usage+Report