Manuscript Psy01-A@1 · Psy01-A.tex · section · 94477fa3

Held-out evaluation uncertainty

Concentration: what survivor counts do and do not imply › Held-out evaluation uncertainty · lines 1711–1724

Back to manuscript

For repeated stress draws on the same held-out inputs, a radius based on independent noise outcomes controls Monte Carlo error conditional on those inputs. It does not turn reused inputs into independent population examples.

Selecting a preferred density after viewing test results requires accounting for the selection, for example through a simultaneous bound or a separate validation set. Variation across density settings is not a confidence interval for variation across independent training seeds.

Nearby in this section