Proposition: Same clean function, different stress tolerance

There are networks with identical masks and identical clean logits for every input, but different stressed accuracies at every fixed $\sigma >0$.

Proof. Disconnect the first two hidden layers from a selected third-layer unit by setting its incoming weights to zero. Give that unit bias $t>0$, so its clean activation is $t$ and its stressed activation is $t+\sigma G$. Set its weight to output class $0$ equal to $1/t$. Let the other class logits be $0,-1,-1$, using output biases.

For every input, the clean logits are

For a test distribution with label $0$, stressed correctness is equivalent to

and hence

All masks and all nonzero-position counts are unchanged as $t$ varies. Upstream noise cannot influence this output. The clean function is unchanged, but the stressed accuracy is strictly increasing in $t$.