16.5.4. Answer Files

When solving ODEs/DAEs and nonlinear algebraic equations numerically, there is not always a known analytical solution for the problem to compare with. Additionally, there may be many paths to a correct solution and some may be less efficient than others. This presents an issue for testing SUNDIALS. How do we ensure SUNDIALS is working correctly if we cannot write down what “correct” is? To solve this problem SUNDIALS has tests (which often also serve as example codes) that have been verified by the author to be functioning correctly. These tests produce output which is stored in a file known as the “answer” file (aka “out” or “output” file). The output typically consist of the solution and/or some quantities derived from the solution as well as statistics about how the integration and/or solve (e.g., number of time steps). When changes are made to SUNDIALS, we ensure that these output files do not change unless it is expected/desired. Practically, this is ensured by our CI testing.

Due to difference in microarichtectures and the nature of floating point arithmetic, it is quite possible that the output generated on one machine may differ from the output generated on another. As such, we specify that the the answer files that are embedded in examples/ (the .out files) should match what is produced on the Jenkins CI machine. We also have a separate git repostiory which holds answers for other machines, such as the GitHub Actions virtual machines.

16.5.4.1. Updating and Adding Answer Files for Existing Machines

Sometimes it is necessary to update the answer files, or add a new file, so that the tests pass, if a change in output is expected/desired. Changing output files requires careful verification that the output is still “correct”. The procedure for updating answer files is as follows:

  • For the .out files embedded in examples, simply overwrite the file with the new output. If the tests still fail, then you will need to download the output file generated on the Jenkins CI machine and commit it. Extracting this file from the Jenkins machine may require help from David or Cody currently.

  • For the .out files in sundials-codes/answers:

    1. Navigate to test/answers in the sundials repository. If there is nothing in this directory, you need to run git submodule init && git submodule update. Alternatively clone the answers repository outside of the SUNDIALS source tree.

    2. Checkout the main branch: git checkout main.

    3. Create a new branch off of main. It is ideal if you use a branch name that matches the name of the branch that you are developing in sundials on: git checkout -b <branch name>.

    4. Update the relevant .out files in test/answers/linux-ubuntu20.04-x86_64/gcc-9.4.0/<double|single|extended> (this is the files used for the GitHub Actions CI). Like with the embedded .out files, you can try and use the output generated on your machine, but it may be different enough from what is produced on the GitHub actions to trigger a failure. You can download the output files generated on the GitHub machines by going to https://github.com/LLNL/sundials/actions finding your failing test, clicking it, then at the bottom downloading the “output_files” artifact (you can also find your failing test at the bottom of a PR). The downloaded zip file will be the SUNDIALS build directory. To update just the failed files you can use ``scripts/updateOutFiles.py ``e.g.,

      cd scripts
./updateOutFiles.py <source path> <destination path>

      When updating files in the answers repo the source path is the path to the build directory downloaded from GitHub and the destination path is <path to answers repo>/linux-ubuntu20.04-x86_64/gcc-9.4.0/<double|single|extended>. When updating with output from Jenkins the source path is the build directory for the Jenkins run and the destination path is the top level of the sundials repo (i.e., ../. if running from scripts). The script will automatically traverse the examples and test/unit_test directories to update output files for failed tests.

    5. If you need to use ssh authentication for pushing to GitHub, then you may need to update the remote for the submodule:

    git remote set-url origin git@github.com:sundials-codes/answers.git

    1. Create a pull-request in sundials-codes/answers with your updates.

16.5.4.2. Adding Answer Files for a New Machine

  • For the .out files embedded in examples, simply overwrite the file with the new output. If the tests running on Jenkins still fail, then you will need to download the output file generated on the Jenkins CI machine and commit it. Extracting this file from the Jenkins machine may require help from David or Cody currently.

  • For the .out files in sundials-codes/answers:

    1. To add new answers, start by copying the relevant answers in linux-ubuntu20.04-x86_64 to your new directory (following the naming scheme outlined above).

    2. Next Commit these files and open a PR to the “staging” branch.

    3. Once the PR is merged, you can generate your new answer files and overwrite the ones you copied from linux-ubuntu20.04-x86_64.

    4. Compare the diff and make sure it is reasonable, then commit.

    5. Finally, open a new PR targeting the “staging” branch.

    6. Eventually, “staging” will be merged into main.