Manuscript Psy03-B@1 · Psy03-B-v1.tex · section · 8ca3caf9
Two distinct evaluation targets
Feedback-aware threshold and severity certification › Two distinct evaluation targets · lines 1179–1202
In simulate_kindling, the observed test accuracy
controls whether scars are added.
The same test inputs supply the activation diagnostic that
controls scar severity.
The test sample is therefore part of the controller.
Later checkpoints depend on that sample.
The ordinary independent-held-out hypothesis of Psy03-A
thm:evaluation is no longer automatically available
CheungPsy03A.
Likewise, its evaluation-isolation proposition does not apply:
the source deliberately feeds the readout into a future update.
This does not make every conditional statement impossible. There are two different targets:
population accuracy under new inputs and new noise;
finite-design accuracy averaged over new noise on the fixed, reused inputs.
They must not be interchanged.