16.5.6. SPOT Performance Visualization Tool
In order to protect against performance regression of SUNDIALS at all scales, we leverage the GitLab CI infrastructure setup for CI testing to perform continuous performance testing.
To track and visualize performance over time, we collect data with Caliper and Adiak. Steps to setting up continuous performance testing(CPT) can be found in Continuous Performance Testing.
Documentation to using the SPOT performance visualization tool can be found here.
16.5.6.1. Setting Up Your Own SPOT Visualizations
To display data from non-GitLab jobs or create a local collection of runs,
input into the SPOT search bar the absolute path to the directory containing
the .cali files and refresh the page. By default .cali` files will be
generated in the build directory under Benchmarking
and
Testing/output/Caliper/Example
. To specify where .cali output files are placed, define
the CMake option SUNDIALS_CALIPER_OUTPUT_DIR with the desired directory path.
To retain the same filters as a given SPOT visualization link in
Bookmarks to Notable SPOT Visualizations, swap out the sf
value in the URL with the
directory path containing the .cali files.
16.5.6.2. Bookmarks to Notable SPOT Visualizations
Notes:
SPOT will be put into maintenance mode in the coming months (as of July 2023). The team behind SPOT plans to bring the functionality to Thicket as SPOT’s successor.
Aggregate results for example runs on SPOT may not be accurate. The aggregation method involves combining .cali files in ways that do not necessarily equal the true sum of the results.
Examples
Benchmarks
Advection Reaction 3D
By configuration
Serial
CUDA
HIP
Diffusion 2D
Note: CUDA Diffusion 2D visualizations are not available as the benchmark errors out before completion.