Manuscript Psy03-A@1 · Psy03-A.tex · theorem · b30eb6e9

Theorem: Full implemented-logit comparison

Stress, margins, and complete perturbation budgets › Coupled parameter comparisons · lines 736–773 · proof included or linked

Psy03-A@1#thm:param, Psy03-A@1#eq:fullparam

Back to manuscript

The coupled outputs obey

\norm z - z^{'}_{2} \leq (ℓ = 1 \prod^{4} A_{ℓ}) \abs ξ - ξ^{'} \norm G_{x}_{2} + ℓ = 1 \sum^{4} (k = ℓ + 1 \prod^{4} A_{k}) r_{ℓ} .

All displayed first moments are finite for fixed finite parameters and the stated Gaussian input law.

If $a, φ, σ, ξ$ are common, the noise and modulation terms vanish, leaving only weight and bias perturbations.

Proof. Subtract the hidden updates. Insert $a φ (v_{ℓ}^{'})$ and $a φ^{'} (v_{ℓ}^{'})$ . The Lipschitz property gives

\norm h_{ℓ} - h_{ℓ}^{'}_{2} \leq A_{ℓ} \norm h_{ℓ - 1} - h_{ℓ - 1}^{'}_{2} + r_{ℓ} .

The head satisfies the analogous recurrence. Iterate.

The activations satisfy $\norm φ (v)_{2} \leq \norm v_{2}$ . The difference between the two activations is at most $2 \norm v_{2}$ . Finite Gaussian input and noise moments therefore imply finite hidden and residual first moments by recursion.

Displayed formulas in this passage

Psy03-A@1#eq:fullparam · line 739
$\norm z - z^{'}_{2} \leq (ℓ = 1 \prod^{4} A_{ℓ}) \abs ξ - ξ^{'} \norm G_{x}_{2} + ℓ = 1 \sum^{4} (k = ℓ + 1 \prod^{4} A_{k}) r_{ℓ} .$
unlabeled · line 760
$\norm h_{ℓ} - h_{ℓ}^{'}_{2} \leq A_{ℓ} \norm h_{ℓ - 1} - h_{ℓ - 1}^{'}_{2} + r_{ℓ} .$

Theorem: Full implemented-logit comparison

Displayed formulas in this passage

Nearby in this section