Manuscript Psy01-A@1 · Psy01-A.tex · section · 94477fa3
Held-out evaluation uncertainty
Concentration: what survivor counts do and do not imply › Held-out evaluation uncertainty · lines 1711–1724
For repeated stress draws on the same held-out inputs, a radius based on independent noise outcomes controls Monte Carlo error conditional on those inputs. It does not turn reused inputs into independent population examples.
Selecting a preferred density after viewing test results requires accounting for the selection, for example through a simultaneous bound or a separate validation set. Variation across density settings is not a confidence interval for variation across independent training seeds.