Manuscript Psy01-A@1 · Psy01-A.tex · proposition · 94477fa3

Proposition: A projection–gradient commutator bound

Stability, hard-mask discontinuity, and schedule dependence › Nested pruning and a quantitative learning-induced defect · lines 1481–1506 · proof included or linked

Psy01-A@1#prop:gradientcommutator, Psy01-A@1#eq:gradientcommutator

Back to manuscript

Let $P$ be a fixed parameter-coordinate projection and let

G_{η} (x) = x - η \nabla L (x) .

If $\nabla L$ is $L$ -Lipschitz on the relevant segment, then

\norm P G_{η} (x) - G_{η} (P x)_{2} \leq η L \norm (I - P) x_{2} + η \norm (I - P) \nabla L (x)_{2} .

Proof. Direct calculation gives

P G_{η} (x) - G_{η} (P x) = η {\nabla L (P x) - P \nabla L (x)} .

Add and subtract $\nabla L (x)$ , then use the Lipschitz and triangle inequalities.

Displayed formulas in this passage

unlabeled · line 1484
$G_{η} (x) = x - η \nabla L (x) .$
Psy01-A@1#eq:gradientcommutator · line 1489
$\norm P G_{η} (x) - G_{η} (P x)_{2} \leq η L \norm (I - P) x_{2} + η \norm (I - P) \nabla L (x)_{2} .$
unlabeled · line 1500
$P G_{η} (x) - G_{η} (P x) = η {\nabla L (P x) - P \nabla L (x)} .$

Proposition: A projection–gradient commutator bound

Displayed formulas in this passage

Nearby in this section