The Stochastic Benchmark open source package answers how a parameter-setting strategy is expected to perform on unseen instances from that family. Per-instance repeat-count reliability asks whether each solver, parameter setting, and resource level has enough repeated stochastic runs to support its own success probability, repeat count, and time-to-solution estimates. Papers employing the package include: introduction paper, QAOA benchmarks 1, and QAOA benchmarks 2.
Resource archive / active
Stochastic Benchmarking Software
Repository for Stochastic Optimization Solvers Benchmark implementation of the Window Sticker framework. The benchmarking approach is described in Benchmarking the Operation of Quantum Heuristics and Ising Machines: Scoring Parameter Setting Strategies on Optimization Applications.