Manuscript Psy03-A@1 · Psy03-A.tex · section · b30eb6e9

Held-out uncertainty and repeated noise

Matched-budget comparisons and statistical certificates › Held-out uncertainty and repeated noise · lines 1080–1092

Back to manuscript

Repeated noise on fixed inputs can reduce Monte Carlo uncertainty conditional on those inputs. It does not turn them into independent population examples.

The theorem permits selecting among the candidates after their simultaneous evaluation. It does not permit arbitrary additional test-driven retraining without expanding or changing the statistical analysis. Across-seed uncertainty requires independent training replicates and a seed-level analysis.

Nearby in this section