Proposition: Clean rescaling does not preserve the budget

For a ReLU network and positive diagonal $D_1,D_2,D_3$, set

At fixed damping $a$, clean logits are unchanged. With isotropic hidden noise in the new coordinates, the transformed computation corresponds to hidden noises $\sigma {D_{\ell }}^{-1}{G}_{\ell }$ in the old coordinates. Magnitude rankings and weight-displacement norms need not be preserved.

There are pairs of distinct clean functions whose weight-displacement dose can be made arbitrarily small by applying the same such rescaling to both parameterizations.

Proof. Positive homogeneity gives $\mathrm{ReLU}(Dv)=D\mathrm{ReLU}(v)$, and $D$ commutes with scalar damping. Induction proves the clean identity. For noise, factor $G_{\ell }=D_{\ell }({D_{\ell }}^{-1}{G}_{\ell })$ at each layer.

For the last assertion, use a constant final ReLU feature one and head weight one in the baseline, two in the intervention. The clean logit difference is one. Rescale that hidden feature by $t>0$ in both networks. The head weights become $1/t$ and $2/t$, so their weight-displacement dose is $1/(t{N}_W)$. It tends to zero while both implemented clean functions remain unchanged.