Manuscript Psy03-B@1 · Psy03-B-v1.tex · section · 8ca3caf9

Two distinct evaluation targets

Feedback-aware threshold and severity certification › Two distinct evaluation targets · lines 1179–1202

Back to manuscript

In simulate_kindling, the observed test accuracy controls whether scars are added. The same test inputs supply the activation diagnostic that controls scar severity. The test sample is therefore part of the controller.

Later checkpoints depend on that sample. The ordinary independent-held-out hypothesis of Psy03-A thm:evaluation is no longer automatically available CheungPsy03A. Likewise, its evaluation-isolation proposition does not apply: the source deliberately feeds the readout into a future update.

This does not make every conditional statement impossible. There are two different targets:

  1. population accuracy under new inputs and new noise;

  2. finite-design accuracy averaged over new noise on the fixed, reused inputs.

They must not be interchanged.