Theorem: Simultaneous candidate and paired-contrast bounds

Condition on $J$ candidate checkpoints independent of the held-out input sample. For $n$ independent held-out examples, possibly with repeated independent noise draws within each example, let ${\hat{Q}}_c$ be the average correctness. Then

Thus

is a simultaneous radius.

For $J_D$ predeclared paired contrasts, where each example's contrast lies in $[-1,1]$, a simultaneous radius is

Dependence between candidate evaluations on the same example is permitted. Independence is required between example blocks.

Proof. Each example's mean correctness over its noise draws lies in $[0,1]$, has the required expectation, and is independent of the other example blocks. Apply Hoeffding's inequality and a union bound Hoeffding. For contrasts the bounded interval has length two.