Test Recommendations
h3. Issues
- tests at http://www.cgd.ucar.edu/cseg/testing/cases/testcases.html are not completely up to date. some test cases exist that aren't documented. some test cases are documented that don't exist. do we want to review cases to
– remove doc/scripts for test cases we no longer need
– verify that some/all test cases actually work properly in scripts
- do we need a performance test?
- do we need a few new tests like TBR.02i, THY.02j
h3. General Recommendations
- 5 Levels of testing. Each level contains a list of specific tests, as documented below. Depending on the phase of development/testing, different levels of testing are appropriate. The individual/lead always has discretion to execute appropriate tests or substitute tests depending on availability of machines and changes to components.
- All testing should be done with the compare option if possible and appropriate.
- Post tag testing should be done from collections whenever possible. If not collections, then it should be a clean checkout only. All post tag tests should be done using the generate option to save the results. Results should be saved in /fs/cgd/csm/ccsm_baselines or the local equivalent at nersc/ornl/etc. Baseline generation should be done only by designated people.
- In all situations, additional tests can be run. For instance, if a specific test fails on one machine, it may be appropriate to run the same test on other machines, to run the same test at a different resolution, or to run the same test in a different mode to provide further guidance on problems.
- Pre-tag testing should NOT be reported on the tag page unless it's highly relevant (ie. non-standard tests were run). Generally, pre-tag testing by developers is done from sandboxes with components NOT entirely consistent with the tag. If pre-tag testing is reported, it should be clearly indicated and the ccsm base version plus different component tags should be noted.
- All post-tag, weekly, and monthly tests results should be reported on the tag web page for that tag. Always note the date.

h3. Testing Roles and Coverage
- Developers, Pre-Tag Testing from a sandbox, tests depend partly on scope of changes and type of component
– Level 0 - required
– Level 1 - run at least 3 tests (3 machines and appropriate compsets)
– Level 2 - all tests recommended
– Level 3 - run at least 2 tests (2 machines and appropriate compsets)
- Tagger, Pre-Tag Testing from sandbox
– Level 0 - as needed
– Level 1 - run at least 9 tests (3 machines x all compsets)
– Level 2 - all tests recommended
- Tester, Post-Tag Testing from collections
– Level 0 - as needed
– Level 1 - all tests required
– Level 2 - all tests required
– Level 3 - all tests recommended, at discretion of CSEG lead
– Level 4 - all tests recommended, at discretion of CSEG lead
- Weekly Testing from collections if no new tags generated that week
– Level 1 - all tests required
– Level 2 - all tests required
- Monthly Testing from collections
– Level 1 - all tests required
– Level 2 - all tests required
– Level 3 - all tests required
– Level 4 - all tests required
– Level 5 - all tests required
h3. Test Levels and Specific Test Cases
- Level 0
– run tests appropriate for model changes
- Level 1
– TER.01a.4x5_gx3v5.A.lightning
– TER.01a.4x5_gx3v5.B.lightning
– TER.01a.4x5_gx3v5.X.lightning
– TER.01a.T31_gx3v5.A.bluesky
– TER.01a.T31_gx3v5.B.bluesky32
– TER.01a.T31_gx3v5.X.bluesky
– TER.01a.1.9x2.5_gx1v3.A.bluevista
– TER.01a.1.9x2.5_gx1v3.B.bluevista
– TER.01a.1.9x2.5_gx1v3.X.bluevista
– TER.01a.1.9x2.5_gx1v3.A.phoenix
– TER.01a.1.9x2.5_gx1v3.B.phoenix
– TER.01a.1.9x2.5_gx1v3.X.phoenix
- Level 2
– TBR.02a.4x5_gx3v5.A.bluevista
– TBR.02a.4x5_gx3v5.B.bluevista
– THY.02a.4x5_gx3v5.A.bluevista
– THY.02a.4x5_gx3v5.B.bluevista
– TER.01i.4x5_gx3v5.B.bluevista
– TER.01j.4x5_gx3v5.F.bluevista
– TER.01k.4x5_gx3v5.F.bluevista
- Level 3
– TDR.01a.4x5_gx3v5.A.lightning
– TDR.01a.4x5_gx3v5.B.lightning
– TDR.01a.4x5_gx3v5.X.lightning
– TDR.01a.T31_gx3v5.A.bluesky
– TDR.01a.T31_gx3v5.B.bluesky32
– TDR.01a.T31_gx3v5.X.bluesky
– TDR.01a.1.9x2.5_gx1v3.B.bluevista
– TDR.01a.1.9x2.5_gx1v3.A.phoenix
– TDR.01a.1.9x2.5_gx1v3.B.phoenix
– TDR.01a.1.9x2.5_gx1v3.X.phoenix
- Level 4
– TBR.02a.T31_gx3v5.A.lightning
– TBR.02a.T31_gx3v5.B.lightning
– THY.02a.T31_gx3v5.A.lightning
– THY.02a.T31_gx3v5.B.lightning
– TER.01i.T31_gx3v5.B.lightning
– TER.01j.T31_gx3v5.F.lightning
– TER.01k.T31_gx3v5.F.lightning
– TER.01k.4x5_gx3v5.F.bluesky
– TER.01a.T42_gx3v5.D.bluesky
– TER.01a.1.9x2.5_gx1v3.I.bluevista
– TBR.02a.T85_gx1v3.A.bluevista
– TER.01b.T85_gx1v3.B.bluevista
– TBR.02a.1x1.25_gx1v3.A.bluevista
– TBR.02a.1x1.25_gx1v3.B.bluevista
– TBR.02a.1x1.25_gx1v3.A.phoenix
– TBR.02a.1x1.25_gx1v3.B.phoenix
– TBR.02a.1x1.25_gx1v3.X.phoenix
– THY.02a.T85_gx1v3.A.phoenix
– THY.02a.T85_gx1v3.B.phoenix
– TER.01i.1.9x2.5_gx1v3.B.phoenix
– TER.01j.1.9x2.5_gx1v3.F.phoenix
– TER.01k.1.9x2.5_gx1v3.F.phoenix
- Level 5
– TBR.02a.T42_gx3v5.A.lightning
– TBR.02a.T42_gx3v5.B.lightning
– TBR.02a.T42_gx3v5.X.lightning
– TER.01i.4x5_gx3v5.B.lightning
– TER.01j.4x5_gx3v5.F.lightning
– TBR.02a.T42_gx1v3.A.bluesky
– TBR.02a.T42_gx1v3.B.bluesky32
– TBR.02a.T42_gx1v3.X.bluesky
– TER.01i.4x5_gx3v5.B.bluesky32
– TER.01j.4x5_gx3v5.F.bluesky32
– TDR.01a.1x1.25_gx1v3.A.bluevista
– TDR.01a.1x1.25_gx1v3.B.bluevista
– TER.01b.T85_gx1v3.B.phoenix
– TDR.01a.1x1.25_gx1v3.A.phoenix
– TDR.01a.1x1.25_gx1v3.B.phoenix
– TBR.02a.4x5_gx3v5.A.seaborg
– TBR.02a.4x5_gx3v5.B.seaborg
– TBR.02a.4x5_gx3v5.X.seaborg
– TBR.02a.1.9x2.5_gx1v3.A.bassi
– TBR.02a.1.9x2.5_gx1v3.B.bassi
– TBR.02a.1.9x2.5_gx1v3.X.bassi
h3. Tag Submission Template
The following is a recommended tag submission template. A filled-in template (or the equivalent information) should be provided as components are tagged to help document the ccsm tag and summarize developer testing. The sandbox parameter should contain the baseline ccsm version of the sandbox plus any updated component tags or indications of non-standard component versions.
Tag:
Date:
Summary of changes:
Sandbox:
Test results:

  • No labels