Data Analysis Services Group - March 2011
News and Accomplishments
xxx
VAPOR Project
Project information is available at: http://www.vapor.ucar.edu
2.0.2 release: We released VAPOR 2.0.2 on Friday, March 4. There were a couple of last-minute bugs reported, but we generally found the code to be fairly stable and we were able to release as planned. We have seen a few problems reported by users of the release. However, so far, these have been minor and we have been able to provide workarounds.
User outreach: John and Alan met with Zbigniew Piotrosky and Andrzej (Andy) Wyszogrodzki. They described a number of capabilities that they use in Vis5d, features that we should consider providing in the next VAPOR release. We have been trying to arrange a computer classroom for the VAPOR tutorial at the June WRF workshop. We will probably need to use Linux this time because of setup constraints.
KISTI proposal: The invited KISTI proposal was completed and submitted to KISTI for review. The proposed work, a 30-month effort, would add support to VAPOR for supporting global ocean model data.
XD Vis Award: Alan completed the first implementation of the extensibility API, with documentation and example code. The example provides a working flow glyph renderer with manipulator control, implemented using the extension API. Team members have begun to review the design before making it public. There are extensive changes to the code base, so it is likely that a number of bugs will still need to be fixed. The documentation of the API was implemented using Doxygen.
Admin: Clyne authored the VAPOR section of the NSF 5-year review (SATR)
Quarterly reports were submitted for Q1FY11
TG GIG PY6 Award: Progress continued on a number of fronts:
- Cleaned up and extended PIOVDC library documentation and API
- Itegrated the PIOVDC library with GHOST source code
- Restructuring PIO and PIOVDC libraries (current activity)
- Creating fully portable build system for PIOVDC (current activity)
- Researched and began implementing simple VAPOR renderer (current activity)
Quarterly reports were submitted for Q1FY11
Misc:
Work is nearing completion on enhancements to VAPOR's color map translation tool to add support for reading colormaps generated by NCAR's NCL.
The VDF library and command line utilities were ported to the Cray.
Work began on prototyping an abstract data type to represent Cartesian gridded data in VAPOR. For performance reasons that are becoming obsolete with microprocessor advances, the current VAPOR implementation simply represents gridded data as arrays of floats. An C++ class object could greatly simplify numerous aspects of the code, improve reliability, and provide a clean path for handling missing data.
Greg thompson (from MMM) has used vapor to illustrate a paper he and Andy Heymsfeld wrote that was accepted for publication, showing the effect of an airplane travellling through a cloud. Greg is hoping that the vapor figures will be on the publication cover. Greg’s animation sequence can be seen at http://vis.ucar.edu/~alan/thompson/holecloud.mov
We selected two Siparcs interns, and one of them, Karamjeet Khalsa from CU, has accepted. He will be working on the ocean data requirements analysis project. The other intern, a Yale student from China, is not able to come because of inadequate work credentials.
Consulting: We continue to respond to VAPOR queries on the mailing list, and to field reported bugs.
Software Research Projects
Feature Tracking: John and Alan continued work with Pablo Mininni to improve they physically based techniques for feature tracking in CFD simulation data. Numerous refinements to the algorithm were required before reliable tracking could be demonstrated with a number of data sets. To compare the quality of our tracking algorithms with other techniques, we implemented an algorithm of Kwan-Liu Ma, and were able to show that our algorithm performs better. A manuscript entitled, Physically Based Feature Tracking for CFD Data, was submitted to Visualization 2011.
Climate data compression:
Data Analysis & Visualization Lab Projects
File System Space Management Project
- Continued design work on the Scrubber/Quota Enforcer/Accounting Data program starting focus on the initial implementation for generating accounting data. Work on this program was put on hold to implement KROLE support.
Visualization Test Bed Project
Accounting & Statistics Project
Security & Administration Projects
- Went back and forth on how to ask for KROLE principals with SEC, finally received first batch. Determined that George's KROLE scripts are not
designed with the DASG system configurations in mind, started writing DASG KROLE management utilities.
- Continued writing KROLE management utilities. Wrote basic C++ wrappers for Kerberos library objects. Used that to get Kerberized RPCs working based on the DCS software suite code. Started work on a DASG user information database that will be needed to manage user, group and KROLE information.
- Installed HPSS KROLE for Heo to test GridFTP/HPSS integration. Installed HPSS KROLE certificates for all DASG cluster users. Added KROLES as required for after the procedures were changed to automatically include requests for the creation of DAV island KROLEs when new users are added.
System Monitoring Project
- Iostat is running on the oasis nodes to measure performance to each LUN and to send that information to Ganglia.
CISL Projects
GLADE Project
Lustre Project
- Planned Lustre upgrade from the current 1.8.3 to 1.8.5 for stratus servers and MDS. The 1.8.5 version is the most up-to-date 1.8 series and will likely be the last official release from Oracle. In addition to support for the recent kernels for SLES11UP1, it has bug fixes for MDS hang.
- Performed the Lustre upgrade while SSG took extended PM on lynx. Patched kernels were installed on 5 Lustre servers and Lustre packages were upgraded to version 1.8.5. Basic tests were performed and integrity of old data were chcked after the upgrade. All configuration information was preserved except the quota status information. Re-initialized the quota on the file system and activated it (but not enforcing for now).
Data Transfer Services Project
- Helped Chi-fan run some basic network throughput tests from OSC to NCAR. Data rates were between 30-40 MB/s.
-
GridFTP/HPSS Interface
- Reconfigured the HSIRC configuration on gridftp nodes to accomodate the KROLE setup.
- Made updates on HSIRC configuration so that it reconizes the changed format of the role-based Kerberos CC files. Also modified the HSIRC on the mirage and storm nodes to use the role-based authentication for other users. Did functionality tests on it reproducing the results from previous runs.
- Performed extensive transfer tests from ORNL(NICS) to NCAR HPSS using the XIO driver we setup. By tuning the gridftp option parameters, successfully achieved ~150MB/s transfer speed for single gridftp target at NCAR. This means that we can realistically expect close to ~50TB/day of data transfer directly to HPSS. (from 150MB/s -> 540GB/hr -> 13 TB/day per gridftp target, we have four of them)
TeraGrid Project
Lynx Project
Batch Systems & Scheduler Project
NWSC Planning
- Preparation for RFP evaluations continues.
- Transition planning for NWSC continues.
Production Visualization Services & Consulting
Publications, Papers & Presentations
- Submitted for publication: Clyne, J., Mininni, P., and Norton, A. Physically Based Feature Tracking for CFD Data, IEEE Visualization.
- Clyne gave an invited talk on VAPOR to CU's Collaborative for Fluid and Energy Sciences
System Support
Data Analysis & Visualization Clusters
- Gcc 4.4 was installed on orthrus to fix an issue with a users code.
- All DCS software was removed after the HPSS switchover.
- A directory was restored from backups for Mariana Vertenstein
- The "fields" package for R was installed along with the latest version of Python, including numpy and the netcdf libraries.
- All dependency from blizzard was removed from the oasis nodes.
- Answered question about the DASG anonymous FTP server for John and Alan.
- Worked with to clear up a glitch with the /fs/blizzard/home file system. An fsck session on the live system with a temporary change of /etc/exports fixed it.
GLADE Storage Cluster
- Helped Nate Rini test GPFS 3.4 compatibility on Lynx with the Nomad GPFS test cluster.
- Helped SSG install GPFS3.3 on lynx DVS nodes by suggesting change in def.mcr configuration so that it can have the alternative lxtrace module without waiting for CRAY to provide to provide different kernel config.
- Replaced 2 failed DEM modules in the DCS9900 storage system. Monitored the LUN rebuilds until they completed.
- Power cycled an enclosure on the DCS9550 storage system to fix a chip lockup causing drives to fail.
- We recovered the data02 filesystem after bad block errors caused the LUNs to be inaccessible from the oasis systems.
- Rebooted G-channel enclosure for DDN9550 (which failed late afternoon on Mar 25) and affected LUNs were rebuilt within 24 hours.
- Started major cleanup of old project spaces and started creating new spaces allocated on April 1.
TeraGrid Cluster
Legacy MSS Systems
- Answered some questions about the Oracle support site for hardware asset management for Bill Anderson.
- Answered some questions about AMSTAR hardware for Bill Anderson.
- Started decommissioning activities of MSSG systems and RAIDS. Made archives of the last active versions of the Unix based software and associated documentation for Gene. Cleaned up the MSS web site of DCS and FTP related documentation.
Data Transfer Cluster
Other