Theorem: Ordinary-ReLU invariant cube

Under $0\le a\le 1$ and $0\le p\le 1$, the ordinary ReLU computation satisfies

for every $t$, every finite input sequence, and every finite noise realization. Consequently this holds almost surely at any finite Gaussian input-noise intensity.

Proof. The assertion holds at $h_0=0$. If it holds at step $t-1$, Lemma lem:convex places the cell output in $[-1,1{]}^H$. Multiplication by $a$, subtraction of $o_p\ge 0$, and ReLU produce coordinates between zero and one. Induct.