Data Analysis Services Group - August 2011

News and Accomplishments

xxx

VAPOR Project

Project information is available at: http://www.vapor.ucar.edu

TG GIG PY6 Award:

Yannick continued testing of the PIOVDC library, discovering during the process that non block-aligned data were not supported. The code is now being generalized to handle this important case. Block-aligned data appear to function correctly now.

Funding for this award was exhausted in August. We're sad to say goodbye to our student assistant for the last two years, Kendall Southwick.

XD Vis Award:

John and Alan gave a tutorial on VAPOR at the TACC Summer Supercomputing Summer Institute. The 2-hour tutorial was attended by ~35 students from around the country, working in a variety of disciplines. We also spent time training TACC staff so that they could teach future VAPOR tutorials themselves.

KISTI Award:

The KISTI contract has finally been executed and work has officially begun on the first year's work. The period of performance runs through Nov. 31, 2011. Hence, there is much  to be done in a very short time period.

Karamjeet has adapted his Python scripts to identify all the grids in MOM data that can be visualized in VAPOR, and to convert these grids to a valid VDC using CDO.  His scripts now work with all the examples of POP and MOM data that we have seen, and correct some problems we have had with some of the data. These prototype scripts will be used to help guide the development of compiled-code POP and MOM4 data translators. A challenge with MOM (and POP) data is that unlike WRF data sets no documentation on MOM/POP output formats appears to exist; we are having to reverse engineer the data translators based on a sample of data sets.

An application was submitted to the CISL RSVP program to support a visitor from KISTI for this fall. The visitor, a MOM4 user, would provide direct guidance on KISTI's needs for MOM and POP data visualization.

Development:

We continue to prepare for the 2.1 release of VAPOR. A decision was made to, for the first time, distribute a beta release of the package before issuing a stable, supported version: a common practice for open source software. The beta will allow us to get important bug fixes into the hands of users who need them more quickly, and will also let us enlist the help of the user community for testing. Mac and Linux installers were created for the release candidates and all VAPOR staff were involved in regression testing. A new, more formal test plan was developed for these purposes.

A number of development items were completed:

• Alan completed the lat/lon support for vapor 2.1.
• Alan modified the flow and probe panels to make them more usable.
• Alan reorganized the VAPOR Python software into three modules, which will make the Python environment more extensible and understandable.
• Alan and Kendall generated a number of global maps that could conveniently be loaded by users of vaporgui with the click of a button.

Bug fixes:

• Data stored in the new VDC2 format were not machine-portable (endianess was not preserved).
• wrf2vdf was not able to distinguish between time steps of data with very small time samples
• A number of problems with WRF data sets computed on idealized grids (no lat-lon information) were resolved
• Workarounds had to be implemented for a bug introduced in the 4.1.3 version of netCDF
• Several new bugs were corrected in the Mac version of tiff2geotiff that appeared after the upgrade of the libtiff library
• Data sets with user times occurring before the EPOCH were causing problems for a number of applications on 32-bit machines.
• The various XML parsers used for reading metadata such as VAPOR session files, user preferences, VDC data, etc., were made forward compatible, no longer complaining when they come across elements that they do not recognize.

Outreach and Consulting:

John and Alan demonstrated VAPOR before a group of researchers participating in the Science and Cyberinfrastructure in Africa workshop held at NCAR.

Alma met with Alma Hodzic at ACD and helped her set up a visualization session of WRF-CHEM.  We expect to use this or similar data in a tutorial on WRF-CHEM in a month or two.

Alan met with Mel Shapiro and made several visualizations of unsteady flow in the ERICA storm, such as the animation at

http://vis.ucar.edu/~alan/shapiro/HighETHTop.mov .

Mel is very interested in using vapor’s unsteady flow capability to understand the storm.

Hsaio-Ming Hsu returned from Germany and reported that his collaborators in Hamburg are very pleased about the VAPOR particle traces that he provided them, because these provided a much improved understanding of the air flow that caused the serious pollution problem in eastern Europe a few years ago.

Andy Heymsfeld says that his article about aircraft “punching” holes in clouds, with VAPOR illlustration, was “published with a lot of press” in Science

John gave an invited talk entitled Visualizing High Resolution Climate Simulations to a group of statisticians at the Joint Statistics Meeting, held in Miami Beach.

Software Research Projects

Feature Tracking:

John and Alan revised their IEEE VisWeek manuscript on physically based feature tracking and submitted it to the journal Transactions on Visualization and Computer Graphics. The expanded version of the paper demonstrated the methods application to tracking multiple features simultaneously, better quantified the reliability of the method, and addressed numerous other reviewer comments.

Data Analysis & Visualization Lab Projects

File System Space Management Project

• Continued work on FMU development and documentation.
• Due to the unsatisfactory features of OpenOffice, started looking at using the DocBook 5 format and tools to capture FMU documentation so as to be able to flexibly produce a variety of output formats.  This is similar to how DocBook 4 was used for the MSS DCS software documentation.
• Also started looking at using XML Stylesheet Transforms to avoid writing FMU code to produce fancy accounting report output (this is synergistic with the tools used in during DocBook processing).

Accounting & Statistics Project

• Tested the capabilities of "tsdbfs" command. The most useful part of the command for now is the capability to locate the physical data blocks for a given file
• Evaluated the "tsgetusage" of GPFS in /usr/lpp/mmfs/samples/util to get the file system status. Test runs for /glade/home/ completed within a minute and for /glade/scratch/ in 1 minutes 4 seconds.
• Compared LDAP schema between NICS and UCAR to prepare for unified account management.
• Started looking at gathering some simple statistics for GLADE GridFTP usage to see how GridFTP is being used.

Security & Administration Projects

• Modified the KROLE management utilities run time configuration to work around encryption key life time mismatch with the life time of the ticket granting tickets.  Made further modifications to support setting KROLE principal's encryption keys separately from initializing the certificates.
• Found a memory leak in the Heimdal Kerberos library, reported it to the project maintainers and received a patch.
• Attended the first Unix Account and Group Automation working group meeting.

CISL Projects

Data Transfer Services Project

• Had another session for Si Liu of USS to demonstrate the GridFTP+HPSS transfer. Noticed the change in myproxy server IP change in Teragrid and updated the Shorewall config for DASG machines.
• Checked the options for recursive copies using globus-url-copy command when used for HPSS transfer. With its current syntax check, appending option will not work for automatic recursive copy to HPSS.
• Worked on HTAR package from Erich to make it work on mirage, strom, and datagate nodes. The package compiled fine and the only undocumented obstacle was the magic environment variable KRB5CCNAME.  We built a wrapper to pass the variable to point to the current krole-cache to automate the authentication.

System Support

Data Analysis & Visualization Clusters

• Replaced the network cards in the mirage3-5 systems so that they will use the 10G pass-through modules instead of the switch and will support a 9000 MTU size.
• Upon request for migration of 834 bluefire home users, created scripts and estimated the time to complete and checked the root_squash flag for additional steps. Generated the lists for number of files and block sums for each user in preparation for migration. Complete migration of these bluefire users to /glade/home.
• Created 1298 /glade/home/ directories for legitimate users from bluefire. In preparation for the rest of bluefire users (1692), generated the list for number of files and
  block sums for the users.  The total will be roughly twice the size of Aug 23 migration.
• Started testing the "fail2ban" package on the storm systems, which will temporarily block any connections after a number of unsuccessful ssh attempts from outside the UCAR networks.  The aim is to discourage the brute-force ssh login attempts that we've been seeing on our external hosts.

GLADE Storage Cluster

• Wrote a script to create a list of total GLADE usage by user, divided into projects, scratch, user, and home usage.
• Wrote a script to list the number of blocks and which LUNs they reside on for any given file on a GPFS filesystem.
• Restored files from /gpfs/proj2/fis/scd/home as requested by a user.
• Caught the SCSI error on oasis[4,7] at 6:00AM and performed the "mmnsddiscover" on servers and clients checks.  Mirage[1,3-4] needed remounts, but bluefire and lynx were fine.
• Reinitialized the DAV island KROLE certificates for all 2048 roles when the first batch ov KROLE principals' encryption keys expired 6 months after their creation in February for the MSS to HPSS switch over.

TeraGrid Cluster

• Found a workaround for a twister user experiencing very slow X11 performance. Suggested a workaround in "ssh" compression option to get consistent response in remote displays