Your first example is commonly referred to as network pruning, and is typically used to compress a model (the nodes are still there, but a sparse weights networ

Your first example is commonly referred to as network pruning, and is typically used to compress a model (the nodes are still there, but a sparse weights network can be stored in compressed form). It's also possible to remove nodes themselves, rather than individual weights. This is typically done on a filter level (for convnets), so that entire filters are removed. The second example (cloning the nodes) is typically performed to improve network robustness (by avoiding important nodes a single point of failure). To do either one during training you need dynamic graphs, so either PyTorch, or TF eager mode. Here's one filter pruning implementation: https://github.com/jacobgil/pytorch-pruning https://github.com/jacobgil/pytorch-pruning