CCSM/ESMF joint meeting

Date: Friday Sept. 15th, 10:30AM MST

Invited: Cecelia, Erik, and Mariana

Gerhard, Samson, and Peggy via speaker phone

(Peggy (818)354-1341)
(Samson (510) 429-9305)
(Gerhard (951)488-1837)

Where: Directors Conference Room

Action items:

  • Erik try out snapshot_12 and see if my issues are resolved.
  • Erik put indices in gridCreate rather than lat/lon in degrees - since we aren't sure the right way to do it.
  • Erik get back to everyone on my issues.
  • Erik go ahead with creating needed bundles in coupler component.
  • Erik get back to Gerhard on the build.
  • Brian E. get back to Gerhard on build.
  • Gerhard fix Darwin for snapshot_12.
  • Peggy look into Cray-X1 build issue.
  • Gerhard, Peggy, Samson – check into coordinates issue for creating a grid.
  • Erik check with Brian E. on grid as Gaussian edges aren't quite right in CAM.
    Content of meeting:
  • 0.) Review last meeting
    • Gerhard working on mpiuni directory issue (already resolved build issues)
    • Clock issue resolved.
    • ESMF clocks are in our code.
    • Upcoming ESMF releases are: ESMF_2_2_2r (version specific for CCSM)
  • 1.) Specific issues we have:
    • My lack of understanding on GridCreate.
      We thought for now – we'll enter index coordinates rather than degrees and do things
      the same way as Peggy has. Since, we aren't doing a re-grid this should be fine. The redist
      just needs decomposition information.
      I need to do a redist for atm to land arbitrary to arbitrary distriubtion.
      For simplicity, let's consider the T5 Spectral case and the 10x15 degree FV case
      T5:
      [G1@(0,0)
      Unknown macro: {16}

      +P;G2@(0,0)

      Unknown macro: {8}

      ] -> [G1@(0,0)

      +P;G2@(0,0)
      Unknown macro: {8}
      ]
      10x15 FV:
      [G1@(0,0)
      Unknown macro: {24}

      +P;G2@(0,0)

      Unknown macro: {19}

      ] -> [G1@(0,0)

      +P;G2@(0,0)
      Unknown macro: {19}
      ]
      Where G1 is longitude and G2 is latitude:
      The code has the coordinate grid point centers as:
      centerLon =
      (/ 0., 22.5, 45., 67.5, 90., 112.5, 135., 157.5, 180., 202.5, 225.,
      247.5, 270., 292.5, 315., 337.5 /)
      centerLat =
      (/ -73.7992136285632, -52.8129431899943, -31.704091745008, -10.5698823125761,
      10.5698823125761, 31.704091745008, 52.8129431899943, 73.7992136285632 /)
      The question is then how to convert from cell centers to vertices.
      Originally I thought you put 0. at bottom and 360 at top for Longitude and -90 and 90 for latitude and then take the averages of two cell centers to get the verticies. I've realized this isn't quite right especially for the FV case which has a pole-point.
      So for longitude it should be:
      (/-11.25, 11.25, 33.75, 56.25, 78.75, 101.25, 123.75, 146.25, 168.75, 191.25, 213.75, 236.25, 258.75, 281.25, 303.75, 326.25, 348.75/)
      As the first point covers from -11.25 to 11.25 degrees centered at 0.0 degrees. Correct?
      and latitude
      (/-90, -63.306079864501953, -42.258517265319824, -21.136987209320068, 0, 21.136987209320068, 42.258517265319824, 63.306079864501953, 90/)
      Now if I use these values – when I inspect the grid for grid centers should I not get the original centered values I used to calculate the vertices? What I have gotten is an extra Long point at the first index at -22.5 degrees. And the latitude points are close – but not quite the same.
      For the FV case:
      Longitude works the same. Here's the specific points:
      FVcenterLon =
      (/ 0., 15., 30., 45., 60., 75., 90., 105., 120., 135., 150., 165.,
      180., 195., 210., 225., 240., 255., 270., 285., 300., 315., 330., 345./)
      FVcenterLat =
      (/ -90., -80., -70., -60., -50., -40., -30., -20., -10., 0., 10., 20., 30.,
      40., 50., 60., 70., 80., 90. /)
      For latitude in order to have pole points the only thing that makes sense is to give it
      two vertices at -90 and 90 so ...
      (/-90, -90, -85, -75, -65, -55, -45, -35, -25, -15, -5, 5, 15, 25, 35, 45, 55, 65, 75, 85, 90, 90/)

All of the above made sense to folks. It appears to be correct.
– Grid issues with arbitrary vs. blocked not working for me.
When I create an arbitrary and a blocked grid for the same physical grid and I compare them some things don't make sense. The latitude on the blocked grid comes back as all zero and dimensioned wrong.
Erik will check this out on snapshot_12 and see what happens.
– mpiuni issue – suggestion – implement all MPI functions in mpiuni.
Gerhard thinks he has this fixed in snapshot_12 – Erik will test and let folks know.

  • 2.) Look at what our code is looking like and how we are using ESMF.
    • Pass internal objects (CCSMInit and SyncClockInfo) by adding additional states to import objects that have data as attributes. 
      CCSMInitEState
Number of attributes =  10
Attribute #:   1 = long_name = shr_inputinfo_mod object ESMF State
Attribute #:   2 = start_type = branch
Attribute #:   3 = case_name =
Attribute #:   4 = case_desc = 
Attribute #:   5 = archive_dir = mss:/CCSM_LONGPATHCASETESTING/csm/long_casename123456789/drv
Attribute #:   6 = mss_irt =        32
Attribute #:   7 = mss_wpass = passw
Attribute #:   8 = aqua_planet = F
Attribute #:   9 = atm_ideal_phys = F
Attribute #:  10 = atm_adiabatic = F

      Name of state to print attributes on: ClockInfoEState                                                                                                                 State
Number of attributes =  13
Attribute #:   1 = long_name = eshr_timemgr_clockInfo object ESMF State
Attribute #:   2 = orb_iyear_AD =      2006
Attribute #:   3 = orb_eccen =     0.16701E-01
Attribute #:   4 = orb_mvelp =      103.00    
Attribute #:   5 = orb_obliq =      23.439    
Attribute #:   6 = orb_obliqr =     0.40909    
Attribute #:   7 = orb_lambm0 =    -0.32454E-01
Attribute #:   8 = orb_mvelpp =      4.9393    
Attribute #:   9 = calendar = NO_LEAP
Attribute #:  10 = desc = My clock
Attribute #:  11 = perpetual_run = F
Attribute #:  12 = NoRestarts = F
Attribute #:  13 = perpetual_ymd =         0
    • CCSM Components expect these states to exist and will abort if they aren't there.
    • CCSM Components use specific lists of 2D fields on states for each type of component. CCSM Components require the specific names.
    • We have separate gridded components to merge fields and couplers to redist or copy.
    • CCSM Components return error code if time is not coordinated.
    • Stage-1 driver requires components to use assumed list of fields.
    • IF driver detects component does not need CCSMInit or SyncClockInfo will abort iff
      (perpetual_run=T, ocn-comp and aqua_planet set, atm-comp and atm_adiabatic or atm_ideal_phys set). Possibly should always abort if ClockInfo not used as orbital information won't be synchronized.
      Looks ok.
  • 3.) Some specific questions for ESMF
    • Fields or bundles? Do bundles for communication to get optimization.
    • When doing a redist – I assume I should do the redist on a bundle rather than loop over a set of fields – correct? Yes.
      But, can I create a temporary bundle to pass all fields, or should I put all data on states as bundles rather than fields? Should be ok to create a temporary bundle.
    • Also when we remap – we need two bundles (one for fluxes one for states as you do bilinear for one and conservative for the other). But, for redist – you want one bundle to the whole list. So how do we separate the two in our states so that a coupler can do the right thing in either case? Create the communication bundles int the couplers. Don't put bundles directly on states as the bundles created will depend on how the mapping is done (if it's a copy, redist or regrid).
    • Here's my thinking:
      Components have module data for fields that are put into import and export states. At initialization Couplers will get pointers to the fields as module data. The coupler will also make the bundles it needs to transfer to another state, by putting the pointers to the relevant fields into bundles that it owns. So the coupler can then either create two bundles for remapping the two different ways OR create one bundle to do a redist. Or don't create any bundles if it's simply copying data from one state to another.
      We think this should be fine – so we will go ahead with it.
    • What is Peggy's work looking like? What's performance looking like?
      Peggy's code is available from
      http://esmf.cvs.sourceforge.net/esmf/use_test_cases/ESMF_Arb2ArbGridRedist/ESMF_Arb2ArbGridRedistUseTestCase.F90?hideattic=0&view=markup&pathrev=ESMF_2_2_2_beta_snapshot_11
      Peggy said that performance is best with the default communication.
    • In looking at beyond Stage-1 – we'll need to use distributed arrays in ESMF_3 for mapping from Ocean to atmosphere correct? Can you have bundles of arrays to do the remapping?
      This isn't available now – but will be. By the time we get to Stage-2 and start needing it – it's likely to be available. We won't be to this point for several months out.
  • 4.) Our status
  • 5.) How is ESMF_2_2_2_r looking?
    • Release date?
      When build issues are resolved and it works for Brian E. and I.
      Gerhard found a build issue on Darwin. And Peggy found a problem with the latest Cray-X compiler (it sounds like a compiler bug – not a problem with ESMF itself).
    • What other specific issues are being worked on besides ours? – None – this is a CCSM specific release.
  • No labels