Mike Waite, MMM
3/28/07
Performing 500x500x100 dipole simulations using Piotr's code (3d rotating stratified turbulence)
10's of timesteps saved
Analysis done on desktop
Using VAPOR currently
Fortran code for computing derived quantities from model output
Data has high dynamic range - problematic to volume render with 8 bits (interesting intervals at extreme positive and negative values)
Starting to run WRF simulations similar to Piotr's. Will compare simulation to lab experiments
Wants isosurfaces
Jonathan Graham, IMAGe
3/28/07
LES simulations
Compare LES to DNS
512^3 data
Currently using VAPOR
Ben Brown & Geoff Vasil, CU
3/28/07
VAPOR feeback:
• Problems creating desired TF. Suggest palette of pre-defined TFs that can be used as a starting point and then modified
• RGB space for TFs would be helpful
• Make it easier to import TFs
• Ben has code for importing an IDL TF into vapor
• Need histogram & opacity annotation in visualization window
6 to 8 ASH users at CU
Joe Klemp, Bill Skamarock, Yongshen Chen, MMM
4/5/07
I'm summarizing my meeting yesterday with Joe Klemp while it's fresh in my mind. Joe is with MMM, and he brought in two of his colleagues, Bill Skamarock and Yongshen Chen for the discussion.
These guys are involved in 3D weather simulation. They use WRF and other software. All their models are consistent with WRF, some are less general.
Their datasets are as large as 1500x1500x100, but usually smaller, more typically half that size.
The grids they use are often terrain-following, i.e. using distance from the ground as the z-coordinate. Sometimes the grid vertical coord is hydrostatic pressure.
They are interested in 3d visualization, and were willing to make some compromises in order to visualize their data. Some of the things they thought would be useful:
• using wavelets for accessing large data.
• Interactive volume rendering
• Flow vis, especially unsteady (e.g. looking at wind)
• Isosurfaces
• Contour planes (with iso-contours, too)
• IDL import/export (but they also have their own analysis libraries)
We discussed several grid types between where we are now and where they are:
1. Uniform cartesian grids. There is only one user there who uses these grids (George Bryan). They did think George could immediatedly use vapor, and that would be a good way to flesh out the requirements for doing weather on vapor.
2. Stretched cartesian grids (with 1-dimensional stretching functions for each axis). They thought this would be immediately useful to about 20-30 users.
3. Spherical grids (with 1-dim stretching functions). This would not be any more useful to them than stretched cartesian grids.
4. Full support of grids as general as theirs.
There was some discussion of using vapor to visualize their general grids, with the thought that they could resample the data for visualization, but continue to do their analysis on the original grid, possibly coming up with a way of importing/ exporting between their general grids and our VDF's.
Kieth Julien, CU
4/11/07
Angular geometry
Predefined color functions - automated color table
Other 4GLs - Matlab
Scripting and math needed
Need to suspend long running operations in VAPOR
Mostly Cartesian grids, may be non-uniform
Tri-linear not sufficient for ????
Mike Wiltberger, HAO
4/11/07
Magnetospheric Physics
Irregular grids
8 state variables for plasma
Grids:
• Stretched Cartesian
• Topologically Cartesian with irregular geometry
• AMR
Three main magnetosphere codes, each uses different grid above
Community MHD codes:
• FLASH (AMR)
• Zero and Athena - spherical with adaptation
Staggered meshes common
Peter Sullivan, Ned Patton, MMM
4/11/07
Ned and Peter are in MMM. They use their own modeling code, not WRF.
Their grids are almost the same: stretched and/or displaced in the z-direction, sometimes with terrain.
They noted that the vertical stretching and displacing can change over time.
Grid sizes are bigger in the vertical direction; typical problems are 600x600x400, with 200 timesteps.
They are currently using tecplot, but they are very interested in VAPOR, especially now that it's on the Mac.
Some of their additional requirements:
- generally they are interested in getting "publication quality" output. Not clear if our jpegs meet this need.
- isosurfaces (although they were thinking that dvr gives them most of what they need here)
- scripting of animation sequences
- annotation such as axis labeling
- they would prefer Fortran to IDL as far as doing analysis with vapor.
Peter and Ned do not have additional collaborators who would also use vapor, but they are pretty interested in using it, and would be happy to work more closely with us on the requirements.
July 10 2007
Met again, demonstrated latest WRF capabilities, asked for feedback.
Ned and Peter would like to be able to use the layered capability we are developing for WRF. So we should make a point of enabling layered data sets other than wrf to be loaded into vapor. But, this model is not as general as the most general model they are using. Their general model allows all 3 coordinates (x,y,z) to vary at the points on each layer, so that it could not easily be interpolated to a cartesian grid.
They reiterated their need for publication qualitiy, especially axis labelling and tic marks.
Would like to be able to capture a scene rotation animation.
Very interested in parallel conversion and compression of data from the simulation.
Piotr Smolarkiewicz ,MMM
4/13/07
Cloud physics primarily
Structured grids most common in meteorological community
• Regular Cartesian topology, irregular geometry
Typical grid size: 800x400x200
Horizontal cords are time dependent
Vertical cords more simple
Colleague Andrzej Wyszogrodzki uses vapor
• Interested in spherical cords
• Wants isosurfaces
Colleague Zbyszek Piotrowski (zbych) will arrive shortly for 4-month visit. Will do large scale convection simulations (Cartesian grid)
1/28/08 (met with Piotr and Zbigniew).
Zbignew will be here for 6 weeks, and is coming again in summer '08.
Zbig wants placement of vector arrows to be easier. Also would like to rotate them (not axis aligned).
Would like to smooth the probe image
Zbig is using parallel netcdf and is thinking it would be a convenient way of moving data from simulation to Vapor
Both Zbig and Piotr are interested in using IBFV
Josh Hacker, RAL
4/25/07
Currently working on nature run (with wrf?), 2800x2800x100, 10s to 100s of time steps, global domain, terrain following in future, height is time dependent
Matlab and NCL preferred tools for analysis
Vis5d not good for high quality images - no vector graphics
Current simulations are IO bound - need to reduce data
WRF user meeting in June
Grid staggering used in WRF - sometimes need to be preserved, sometimes can be regridded
Wants isosurfaces
George Bryan, MMM
5/4/07
Has own weather code
• Horizontal resolutions on order of 2k points
• University users (how many?)
• Cloud model + thunder storms
• Small scale compared to WRF
• Stretched grid in horizontal and vertical
• Terrain following grid (flat upper boundary, terrain following below)
•
Generally do mean statistics, only saving full domain periodically
Soon to do tornado simulations with collaborators Leigh Orf (U. Mich)
Important viz features:
• flow integration
• cross sections
• isosurface
• GIS data on ground & geographic boundary mapping (needed for weather prediction, not so much for weather research community)
• Derived field calculator
Currently uses GRADS, not NCL
Weather prediction community requires geopolitical boundaries, terrain, and GIS info
WRF workshop in June
George is doing resolution sensitivity study next week using simple Cartesian grid. Would like to use VAPOR
Richard Rotuno is doing terrain following simulations
George has collaborators with terrain data sets that Victor might use
Comparing ensemble runs is desirable
Leigh Orf, U. Mich
5/8/07
Yes, I am very interested. I am actively seeking vis tools that can manage TB of data. I recently discovered VisIt
(http://www.llnl.gov/visit/) and managed to do some interesting things with it. See for example:
The file lockney100-ThSfc-PPert-VolDBZ.mp4 was rendered with VisIt.
VisIt does a lot of things I like (parallelizes, does nice volume rendering, handles vectors and particles) but there are also some things I'm not crazy about (is a bit buggy, its software volume rendering is a memory hog, etc.).
I have not fully integrated my data into VisIt yet - for instance the model grid is stretched but I have not un-stretched it in the mp4 file. VisIt offers an interface for writing your own data plugin but I have yet to do that.
My data format is distributed HDF4 files. They are "distributed" because I run my models in parallel where each node dumps its own subset of the entire physical domain. There is metadata within each HDF4 file, which allows you to stitch things back together. I have written an API for this format that makes it easy to read data in as if it were one big HDF4 file (http://hdftools.sourceforge.net).
The biggest simulation I have done thus far was 2400x2400x200 points.
However as is usually the case in these cloud simulations, the interesting stuff comprises only about 20% of the entire model domain. So any vis tool useful to me should be able to read in subsets of data, or handle memory allocation in a reasonable way so as it doesn't require a supercomputer to render the data (just run the model!). I think your application may be much better at this than VisIt.
Anyway I will have some time to stop by and talk with your group when I am in town. Do you want to take a stab at getting my data into VAPOR before then? I can give you my HDF4 files + code to read them in easily, or single netcdf files (I wrote a converter) if that is easier initially. I don't think I'll have time to write a converter to your native file format before I head out to Boulder.
Cindy Bruyere, MMM
5/11/07
WRF user support and basic science
WRF supports nested grids, but in general only need to look at one grid level at a time
Grids are time-varying by default (follow pressure in vertical ??)
Supported WRF dav tools: NCL, vis5d, RIP4, GRADS
Vis5d interest has been growing, but vis5d does not scale
GRADS is tool of choice for data processing
Bill Quo heading up WRF climate "channel" run (channel is 45N to 45S)
Bill Skamarock leading global WRF run in 2 months
WRF uses a staggered grid: U, V, W are centers of cell faces (each on its own face). Scalars are cell centered
Netcdf is primary output format for WRF- all WRF files basically look the same, though users can select which variables to output and control how many vars/timesteps are written to a single file. No missing data values (grid follows terrain and is always above surface)
WRF has two very different dynamical cores (ARW, the common case, and NMM)
Cindy can provide initial data for Victor to get started with and provide guidance. Some interpolation tools already exist.
7/10/07 Met again with Cindy to get update on requirements for the next release. She generally agreed with the priorities we already established, but she had the following additional comments:
She thinks we should use non-staggered coords both vertically and horizontally, does not like vertical extrapolation
Would like to see extrapolation below ground of terrain values of variables as an option in volume rendering
Does not want a GUI
Would like option of calculating TK (temperature Kelvin), and P+PB
Thinks that there would be lots of problems with pasting NCL images into scene because of the various map projections that are used. Thought it would be better if we used the "Land Use" variable in the wrf data to color the terrain, at least for water vs land
Marie Farge, et al
6/8/07
Volume rendering quality needs to be better for low-res data
TF editor should facilitate symmetries around zero
Need more default color palette options to choose from besides current rainbow. Also need to be able to operate in RGB space as well as HSV
Color bar annotation needs improvement - too had to use, font is often illegible, etc.
Would like to animate isovalue space
Patrice Klien, IFREMER, Brest, Paris
6/8/07
Patrice is a colleague of Marie
Collaborates with Frank Bryan on POP model, but is primarily a weather modeler I believe.
Will be in the U.S. in Nov.
Computes on Earth Simulator
Uses staggered grids
Emmanuel Dormy (dormy@phys.ens.fr)
6/9/07
Geodynamo modeler. Collaborates with Gary Glantzmeyer (sp?)
Wants to be notified when spherical grid prototype is ready
Sasha Brun, Saclay, Paris
6/9/07
stellar simulations with ASH code (collaborates with Juri)
spherical grids
Wants to be able to extract subregions from the sphere for in-depth analysis
Yongshen Chen, Wei Wang, MMM
6/22/07
Concerned about need to transform data to VDC to use in VAPOR. Already using multiple tools and would prefer that they could all read the same data file formats. Thought support for other data format importers would be useful even if did not have multiresolution capabilities.
WRF model outputs 2D fields that provide topology info
Suggest talking to Chin-Hoe Moeng (MMM), a WRF turbulence modeler. Chin-Hoe is on sabatical until August 1
Would like Mac port of HPRGS.
Yongshen and Wei will experiment with VAPOR using Yongshen's hurricane data.
July 9, 2007
Met with Wei again (Yongsheng did not make it, but wrote back later)
Generally agreed with our feature prioritization, but wanted to get back with us on a few issues, including
- Unsure if vertical unstaggered coordinates were the right way to go
- Will name additional variables that should be available at the vdf creation time.
Wei said she will get Yongsheng's input too.
They thought arrows for wind direction are important.
Thought it would be nice to be able to calculate additional variables on the fly
Would like to have a library of useful transfer functions, keyboard shortcuts, click to reset transfer function.
Mark Miesch
Mark will be doing a Flash AMR simulation in the next 2-3 years - modeling CMEs
Yuhong has new spherical simulation
Stretched spherical grids (radially) are the norm for spherical
Josh Hacker
9/5/07
Josh is struggling with his "nature run" due to IO problems - IO performance is in adequate. Can't compute on Frost - too small. Computing at Watson and elsewhere (LLNL?). Not sure how to get data back. Interested in VDC API that would permit resolution control on simulation output.
Only working on WRF nature run in "spare time". Interested in 3d features of data, but no ready to do analysis
Stay in touch re petascale plans