Weights, masks, and ambient gradients

In the supplied implementation, masked parameters are set to zero and masks are reapplied after optimizer steps. The forward pass uses ordinary dense linear modules. Thus its derivatives at inactive zero positions are derivatives of the ambient dense extension of the sparse network.

This distinction matters. If a replacement implementation instead computes $W_{\ell }\circ M_{\ell }$ inside autograd while differentiating with respect to $W_{\ell }$, inactive-position derivatives are multiplied by zero. That implementation cannot use those derivatives as the source's regrowth scores without another scoring interface.

The method is unstructured weight pruning, despite the source manager's “structured pruning” description. Gradient-ranked changes of sparse topology have established precedents EvciRigL; the supplied algorithm is not identified with RigL.