One requirement for the HOMME tests is that changes to the trunk should preserve bit for bitness of the regression tests. In order to support this requirement on multiple platforms the testing system requires the creation of baseline test results to which new results produced by modified code will be compared. In order to successfully run the tests, first the baseline results must be created.
Creating the Baseline Results
From a clean svn checkout configure and build (make) HOMME and run "make baseline". This will run the regression runs to create Netcdf output files.
Pointing the Development Build to the Baseline Results
Once the baseline results have been produced you are set to do development. Note that the testing system needs to be aware of the location of the baseline results. This is done by passing the following option to the cmake configure.
-DHOMME_BASELINE_DIR=/path/to/baseline/results
Here, "/path/to/baseline/build" represents the path to the HOMME build where "make baseline" has been performed. If this option is not specified then the current build directory will be used. Note that the tests will fail if the test results are not located in the appropriate directory.
Running the Tests
Now that the baseline results have been created, and the development build has been configured and built properly (by running make in the build directory), the tests can be run. This is performed with a simple "make check". Here, the regression runs are ran again producing new Netcdf files corresponding to the development changes. Then the Netcdf files from the development build are compared with the Netcdf files produced from the baseline build. Each test passes if the Netcdf files are identical.
Note that the tests will fail if the baseline results do not exist in the directory specified above.
Example Workflow
To set up HOMME on a machine on which you'd like to do development please do the following steps.
Check out a version of the HOMME trunk - call it homme-baseline. Create a build directory in a different location (say /scratch/userName/builds/homme-baseline) and configure and build the code. Next, create the baseline results with "make baseline".
Check out another copy of the HOMME trunk - call it homme-trunk. This is the source directory in which you will do development. Create a build directory in a different location (this time call it say /scratch/userName/builds/homme-trunk). When configuring this build be sure to pass to cmake the following option.
-DHOMME_BASELINE_DIR=/scratch/userName/build/homme-baseline
After you have made your development changes and are ready to commit the code you should now run the tests. Since you have already pointed the build/tests to the baseline results with the above option all you need to do is run "make check". The regression runs will be performed and the output Netcdf files will be diffed. The individual tests will pass if these Netcdf files match the baseline results.
Once the tests have passed, and bit for bitness has been shown, you are free to commit the code.