3 ms·
A slightly related question: There's been a few articles in the recent years about ways to reduce the size of NNs. Does anyone know how efficiently packed are t
by viraptor 4y ago
A slightly related question: There's been a few articles in the recent years about ways to reduce the size of NNs. Does anyone know how efficiently packed are the new models like GPT? Essentially, what percentage of weights could be thrown out post-training without much impact on the quality?
- mxkopy 4y agoSparse matrices (i.e. filled mostly with zeroes) are apt to represent discrete structures, since you can interpret them as binary/n-hot vectors as opposed to points in space. I'd imagine there's some potential to do sparse matrix optimization on GPT since natural language has a tree-like structure, but it's a well known technique and I'm sure the big players are already doing it
- p1esk 4y agowhat percentage of weights could be thrown out post-training At least 60% according to this paper: https://arxiv.org/abs/2301.00774 https://arxiv.org/abs/2301.00774
- nwoli 4y agoThe silly theoretical lower bound is finding a seed for a random number generator that unrolls weights in the order they’re used. So down to 32 bits minimum :) Silly answer but with enough compute you could find it and enable enormous models like that
- detrites 4y agoIt's an attractive idea but I believe that probability says "no". Assuming a perfect RNG, the odds are stacked that you can assume any length(index), into the correct set of data, will be longer than that data you're seeking a location of. More background here: https://math.stackexchange.com/questions/432017/can-random-number-generators-be-used-in-compression https://math.stackexchange.com/questions/432017/can-random-n... I'm sure there may be isolated exceptions where this wouldn't happen and it'd be fascinating to learn about what they'd be. To my knowledge none have been found yet. If you know of any, do share.
- nwoli 4y agoI guess one counter argument is lossy compression. Another argument is to think about how rng generators are constructed in some cases (algebraic curves) and imagine instead of finding a seed, solving for an algebraic curve with the desired properties. More data necessary (the curve data) and tons of compute still (solving multivariate polynomials) but less so than finding a seed. Then there’s also ideas like creating the weights out of another network (basically moving around in that sub space) which is a much more feasible idea and something Hardmaru has a cool article about
- p1esk 4y agoSuch “random number generator” would be equivalent to the training algorithm.
- viraptor 4y agoThat would reduce the storage requirement, but not energy/CPU. After decompressing you'd need to still process the original network. I was interested in dropping the weights for processing on smaller devices.