Proposition: Bounded signal, unbounded stressed activation

For a tanh layer,

If $a\sigma >0$, each coordinate of $a[\tanh (v)+\sigma G]$, conditional on $v$, has unbounded Gaussian support. Thus the complete stressed hidden state is not bounded.

Proof. Use $\text{abs}\tanh x\le 1$. Adding a nondegenerate Gaussian after tanh gives support equal to the whole real line in each coordinate.