Manuscript Psy01-A@1 · Psy01-A.tex · section · 94477fa3

A local affine Gaussian response

Stress certificates and a conditional trade-off › A local affine Gaussian response · lines 698–748

Psy01-A@1#eq:frozengaussian, Psy01-A@1#eq:gateerror

For a fixed input $u$ , let

a_{ℓ}^{0} = B_{ℓ} h_{ℓ - 1}^{0} + b_{ℓ}, D_{ℓ} = diag (1 {a_{ℓ, i}^{0} > 0}),

where $h^{0}$ is the clean forward pass. Freeze these gates and define

h_{ℓ} = D_{ℓ} (B_{ℓ} h_{ℓ - 1} + b_{ℓ}) + σ G_{ℓ} .

Let

R_{ℓ, ℓ} = I_{n_{ℓ}}, R_{ℓ, r} = D_{ℓ} B_{ℓ} R_{ℓ - 1, r} (r < ℓ) .

Then

z = z_{θ} (u; 0) + σ V G, V = [B_{4} R_{3, 1} B_{4} R_{3, 2} B_{4} R_{3, 3}],

where $G \in R^{640}$ is standard Gaussian.

For $ℓ = 2, 3$ , define

v_{ℓ, i}^{2} = r < ℓ \sum \norm (B_{ℓ} R_{ℓ - 1, r})_{i, :}_{2}^{2} .

Assume the deterministic preactivations at these gates are nonzero. Define

ε_{gate} (u, σ) = min {1, ℓ = 2 \sum^{3} i = 1 \sum^{n_{ℓ}} Φ (- \frac{\abs a _{ℓ, i} ^{0}}{σ v _{ℓ, i}})},

where a summand is zero if $v_{ℓ, i} = 0$ .

Displayed formulas in this passage

unlabeled · line 701
$a_{ℓ}^{0} = B_{ℓ} h_{ℓ - 1}^{0} + b_{ℓ}, D_{ℓ} = diag (1 {a_{ℓ, i}^{0} > 0}),$
unlabeled · line 707
$h_{ℓ} = D_{ℓ} (B_{ℓ} h_{ℓ - 1} + b_{ℓ}) + σ G_{ℓ} .$
unlabeled · line 713
$R_{ℓ, ℓ} = I_{n_{ℓ}}, R_{ℓ, r} = D_{ℓ} B_{ℓ} R_{ℓ - 1, r} (r < ℓ) .$
Psy01-A@1#eq:frozengaussian · line 719
$z = z_{θ} (u; 0) + σ V G, V = [B_{4} R_{3, 1} B_{4} R_{3, 2} B_{4} R_{3, 3}],$
unlabeled · line 728
$v_{ℓ, i}^{2} = r < ℓ \sum \norm (B_{ℓ} R_{ℓ - 1, r})_{i, :}_{2}^{2} .$
Psy01-A@1#eq:gateerror · line 735
$ε_{gate} (u, σ) = min {1, ℓ = 2 \sum^{3} i = 1 \sum^{n_{ℓ}} Φ (- \frac{\abs a _{ℓ, i} ^{0}}{σ v _{ℓ, i}})},$