2 ms·

We need to see something like "dropout" gain popularity again where you take a trained network and keep removing hidden neurons and re-training a few iterations

We need to see something like "dropout" gain popularity again where you take a trained network and keep removing hidden neurons and re-training a few iterations until you can't obtain the same accuracy anymore, at which point you've reached what should be the smallest that network can be while still encoding the same knowledge. With models of these sizes though, few have the compute or the time to do stuff like that. There's a way of doing the whole thing iteratively though while you train that shouldn't add too much extra complexity.. The fact that the big players aren't doing this doesn't surprise me --- the more parameters, the less they have to worry about small-time competititors. I wouldn't be surprised if you could make an order of magnitude reduction in parameters and get the same results, and they're all just terrified we'll figure that out As an undergrad I did some research on mapping "computational complexity" concepts onto neural networks to answer questions like what classes of problems require what minimum hidden layer sizes or network topologies using strategies similar to this. I was actually astounded how simple some of the resulting networks were that could nonetheless solve complex tasks like OCR with very few weights.

snerbles 4y ago
"Pruning" is the main term you are looking for, with a variety of methods to do so. https://blog.dataiku.com/making-neural-networks-smaller-for-better-deployment-solving-the-size-problem-of-cnns-using-network-pruning-with-keras https://blog.dataiku.com/making-neural-networks-smaller-for-... https://arxiv.org/abs/2301.00774 https://arxiv.org/abs/2301.00774 https://tivadardanka.com/blog/how-to-compress-a-neural-network https://tivadardanka.com/blog/how-to-compress-a-neural-netwo... There's also quantization. It's currently where a lot of the current grassroots research is happening, if you've seen the llama.cpp repo posted here on HN, it uses the original 16-bit float weights downsized to 4-bit integers with a comparable reduction in resource usage with relatively little performance loss. The big players are also using quantization in hardware, most notably with Google's TPUs.
stu2b50 4y ago
That's not what dropout is, dropout is just a method of regularization (you remove a random, and small, subset of the neurons per training iteration, in order to reduce overfitting), and pretty much all LLM transformer blocks have dropout.

snerbles 4y ago

"Pruning" is the main term you are looking for, with a variety of methods to do so. https://blog.dataiku.com/making-neural-networks-smaller-for-better-deployment-solving-the-size-problem-of-cnns-using-network-pruning-with-keras https://blog.dataiku.com/making-neural-networks-smaller-for-... https://arxiv.org/abs/2301.00774 https://arxiv.org/abs/2301.00774 https://tivadardanka.com/blog/how-to-compress-a-neural-network https://tivadardanka.com/blog/how-to-compress-a-neural-netwo... There's also quantization. It's currently where a lot of the current grassroots research is happening, if you've seen the llama.cpp repo posted here on HN, it uses the original 16-bit float weights downsized to 4-bit integers with a comparable reduction in resource usage with relatively little performance loss. The big players are also using quantization in hardware, most notably with Google's TPUs.

stu2b50 4y ago

That's not what dropout is, dropout is just a method of regularization (you remove a random, and small, subset of the neurons per training iteration, in order to reduce overfitting), and pretty much all LLM transformer blocks have dropout.