3 ms·
I'm not an expert, but I wonder if the wide network could then be trimmed down for special purposes? E.g., find the parts of the network needed for a certain ta
by foota 3y ago
I'm not an expert, but I wonder if the wide network could then be trimmed down for special purposes? E.g., find the parts of the network needed for a certain task by looking at activations on inputs and dump the rest.
- littlestymaar 3y agoThat was my thought as well, it would be nice if someone more knowledgeable came and explained us why it's stupid/doesn't work
- neodypsis 3y agoWeight pruning is already a thing. See, for example: https://www.tensorflow.org/model_optimization/guide/pruning https://www.tensorflow.org/model_optimization/guide/pruning
- hedgehog 3y agoBesides pruning (which is what you describe) you might find the "lottery ticket hypothesis" interesting. The idea is essentially that in a randomly initialized large network there is a subset of weights that are already closeish for a given task and training helps tune that and suppress everything else, and that knowing this you can actually get better results by pruning before training.