Manuscript Psy01-A@1 · Psy01-A.tex · theorem · 94477fa3

Theorem: Parameter-to-logit stability

Stability, hard-mask discontinuity, and schedule dependence › Logit perturbation · lines 1153–1188 · proof included or linked

Psy01-A@1#thm:parameterstability, Psy01-A@1#eq:parameterstability

For common stress intensity $σ$ ,

\norm z_{θ} - z_{θ^{'}}_{2} \leq ℓ = 1 \sum^{4} (k = ℓ + 1 \prod^{4} a_{k}) (e_{ℓ} \norm h_{ℓ - 1}^{'}_{2} + β_{ℓ}) .

The expectation of the right side is finite for the stated Gaussian task distribution and fixed finite parameters.

Proof. For hidden layers, common additive noise cancels when the two states are subtracted. Hence

\norm h_{ℓ} - h_{ℓ}^{'}_{2} \leq a_{ℓ} \norm h_{ℓ - 1} - h_{ℓ - 1}^{'}_{2} + e_{ℓ} \norm h_{ℓ - 1}^{'}_{2} + β_{ℓ} .

The same inequality, without ReLU, holds at the head. Starting from equal inputs and iterating proves (eq:parameterstability).

The Gaussian input has finite first moment. Recursively,

\norm h_{ℓ}^{'}_{2} \leq \norm B_{ℓ}^{'}_{op} \norm h_{ℓ - 1}^{'}_{2} + \norm b_{ℓ}^{'}_{2} + σ \norm G_{ℓ}_{2},

so all hidden first moments are finite.

Displayed formulas in this passage

Psy01-A@1#eq:parameterstability · line 1156
$\norm z_{θ} - z_{θ^{'}}_{2} \leq ℓ = 1 \sum^{4} (k = ℓ + 1 \prod^{4} a_{k}) (e_{ℓ} \norm h_{ℓ - 1}^{'}_{2} + β_{ℓ}) .$
unlabeled · line 1171
$\norm h_{ℓ} - h_{ℓ}^{'}_{2} \leq a_{ℓ} \norm h_{ℓ - 1} - h_{ℓ - 1}^{'}_{2} + e_{ℓ} \norm h_{ℓ - 1}^{'}_{2} + β_{ℓ} .$
unlabeled · line 1182
$\norm h_{ℓ}^{'}_{2} \leq \norm B_{ℓ}^{'}_{op} \norm h_{ℓ - 1}^{'}_{2} + \norm b_{ℓ}^{'}_{2} + σ \norm G_{ℓ}_{2},$