Proposition: Deletion can improve, worsen, or preserve a checkpoint

In the stated architecture class, permanent deletion can improve, worsen, or leave unchanged the accuracy of a current checkpoint.

Proof. Take $q=1$, $p=0$, zero challenge noise, and one constant final ReLU feature equal to one. Use a test law whose label is always zero. Unused units embed these examples in the declared widths.

For improvement, use head logits

where the $0.5$ contribution is an active head weight. The network fails before that weight is scarred and succeeds after it is removed.

For deterioration, use

with the first logit supplied by one active head weight. Removing that weight changes the first logit to zero and causes failure.

For no effect, use constant output biases and disconnect every hidden unit from the output. Scarring a hidden connection leaves all predictions unchanged, including under hidden noise.