4 ms·
The silly theoretical lower bound is finding a seed for a random number generator that unrolls weights in the order they’re used. So down to 32 bits minimum :)
by nwoli 4y ago
The silly theoretical lower bound is finding a seed for a random number generator that unrolls weights in the order they’re used. So down to 32 bits minimum :) Silly answer but with enough compute you could find it and enable enormous models like that
- detrites 4y agoIt's an attractive idea but I believe that probability says "no". Assuming a perfect RNG, the odds are stacked that you can assume any length(index), into the correct set of data, will be longer than that data you're seeking a location of. More background here: https://math.stackexchange.com/questions/432017/can-random-number-generators-be-used-in-compression https://math.stackexchange.com/questions/432017/can-random-n... I'm sure there may be isolated exceptions where this wouldn't happen and it'd be fascinating to learn about what they'd be. To my knowledge none have been found yet. If you know of any, do share.
- nwoli 4y agoI guess one counter argument is lossy compression. Another argument is to think about how rng generators are constructed in some cases (algebraic curves) and imagine instead of finding a seed, solving for an algebraic curve with the desired properties. More data necessary (the curve data) and tons of compute still (solving multivariate polynomials) but less so than finding a seed. Then there’s also ideas like creating the weights out of another network (basically moving around in that sub space) which is a much more feasible idea and something Hardmaru has a cool article about
- p1esk 4y agoSuch “random number generator” would be equivalent to the training algorithm.
- viraptor 4y agoThat would reduce the storage requirement, but not energy/CPU. After decompressing you'd need to still process the original network. I was interested in dropping the weights for processing on smaller devices.