Theorem: External-tanh bound and biased-ReLU envelope

For either path with external tanh,

For biased ReLU, set $R_0=0$. A pathwise envelope is

Then $\text{norm}{h_t}_{\infty }\le R_t$.

If all hidden-noise coordinates up to time $T$ have magnitude at most $r$, a convenient looser bound is

For fixed finite parameters and $T$, all hidden first moments are finite.

If $a\sigma >0$, biased ReLU does not have a deterministic unit-cube bound: conditional on a finite preceding state, each noisy preactivation has an unbounded upper Gaussian tail.

Proof. The tanh assertion follows from its range. For ReLU, use Lemma lem:convex, the nonnegative offset, and $\mathrm{ReLU}(v)\le \text{abs}v$ coordinatewise. Use $\max \{1,R\}\le 1+R$ and iterate for (eq:Rloose). Gaussian norm first moments are finite, so the finite recurrence has finite first moments. A nonzero Gaussian term before ReLU has arbitrarily large positive values with positive probability.