Theorem: Zero weight dose does not control implementation

There exist pairs in the stated architecture with identical weights, biases, and masks, hence $D_W=0$, but different clean logits and accuracies. This holds for a damping change alone, and also for an activation change alone.

Proof. For damping, use one constant positive feature. Give its first hidden unit bias one and zero incoming weights. Relay that unit through one unit in each subsequent hidden layer using weight one and bias zero. Use ReLU and send the final feature to output class $0$. Its clean logit is $a^3$. Set the class-$1$ output bias to $c$, where

and the other logits to $-1$. For a test law with label $0$, changing $a_{-}$ to $a_{+}$ changes accuracy from zero to one. No parameter or mask changes.

For activation, isolate a third-layer unit with zero incoming weights and bias two, set $a=1$, and send it to class $0$. Let class $1$ have bias $3/2$, and the other logits be $-1$. ReLU gives $z_0=2>3/2$, whereas $\tanh (2)<1<3/2$. The prediction again changes without a parameter change. Unused coordinates embed both examples in the specified widths.