16.5.5. Continuous Performance Testing (CPT)

In order to protect against performance regression of SUNDIALS at all scales, we leverage the GitLab CI infrastructure setup for CI testing to perform continuous performance testing.

The CPT suite consists of the SUNDIALS benchmark programs programs, which can scale up to full supercomputers, and the normal SUNDIALS examples program suite (i.e., the programs in the examples directory of the SUNDIALS repo).

The CI suite can run the regular SUNDIALS CI tests, or it can run the SUNDIALS CPT suite by setting the BENCHMARK variable to ON when running a pipeline from the GitLab CI UI. The benchmark problems are run with Caliper and a report for Spot and a human-readable runtime-report are generated. The runtime-report is printed to the stdout and can be viewed in the GitLab CI job output. The Spot output files are made available as job artifacts. We maintain a pipeline that runs the CPT suite weekly on a schedule, see https://lc.llnl.gov/gitlab/sundials/sundials/-/pipeline_schedules.

Performance over time is tracked with Caliper and SPOT web framework.

16.5.5.1. Locally Building and Running the CPT

The SUNDIALS example suite can be run with Caliper profiling enabled and Adiak enabled for the CPT suite by setting the CMake options

$ cmake \
> -DSUNDIALS_BUILD_WITH_PROFILING=ON \
> -DENABLE_CALIPER=ON \
> -DCaliper_DIR=/path/to/caliper \
> -DENABLE_ADIAK=ON \
> -Dadiak_DIR=/path/to/adiak/lib/cmake/adiak \
> -DSUNDIALS_TEST_DEVTESTS=ON \
> -DSUNDIALS_TEST_PROFILE=ON \

This command will result in --profile option being passed to the SUNDIALS test runner Python script, test/testRunner, which will in turn set the CALI_CONFIG environment variable before running every test so that when you run make test the examples will produce .cali output files documenting the performance.

Note: Caliper prints to the .out files by default. Ensure all Caliper configs requested have the output option defined to ensure output data is saved in a separate location from the test output. Otherwise, the .out files for each test will contain the output and the tests will fail.

Turning on the BUILD_BENCHMARKS option will build benchmarks. Running make benchmark will execute all the available benchmarks and produce .cali output files for each one. To change what parameters benchmarks are run with, edit the respective CMakeLists.txt. The BENCHMARK_VARS variable determines how many tests to run with different parameters. Arguments passed into the sundials_add_benchmark macro change how the benchmark is run.

To specify where .cali output files are placed, define the CMake option SUNDIALS_CALIPER_OUTPUT_DIR with the directory path. By default .cali output files are placed in the build directory (Acquiring SUNDIALS) under Benchmarking/output and Testing/output.

Refer to section Benchmarks for details on instructions on building and running the benchmarks/ programs locally. Refer to section <<examples>> for instructions on build and running examples locally.