Lemma: Uniform bounded-record shot certificate

Suppose $K$ declared scores lie in $[0,1]$. For each score, use $N$ independent attempted runs from the same fixed actual model. If its actual mean differs from the target mean by at most $b$, then, simultaneously for all $K$ scores,

with probability at least $1-\alpha$, provided

If the allowance $b$ depends on calibration valid with probability at least $1-\beta$, the unconditional coverage is at least $1-\alpha -\beta$.

Proof. Hoeffding's inequality bounds each two-sided deviation from the actual mean by $2e^{-2N{h}^2}$ Hoeffding. Use a union bound and then the mean allowance. Add the calibration-failure probability.