4 ms·
It’s explained in detail here: https://kolinko.github.io/effort/equations.html https://kolinko.github.io/effort/equations.html Long story short - I kind of so
by kolinko 3y ago
It’s explained in detail here:
https://kolinko.github.io/effort/equations.html https://kolinko.github.io/effort/equations.html
Long story short - I kind of sort them and pick only the top % that would give a highest result.
One part is choosing them though - I think this was done before in some papers. But the second part was an implementation of multiplication that is efficient on both gpus and cpus when choosing weights almost at will.
All explained on the site, but I just got feedback that it may not be easy enough to read, so I’ll push it through gpt for grmamar fixes soon :) It’s also a bit complicated as an algorithm.
- throwaway2562 3y agoHmmm. Very interesting. I wonder if you could speed up your clever approximation still further with this approach https://arxiv.org/pdf/2205.09120.pdf https://arxiv.org/pdf/2205.09120.pdf
- kolinko 3y agoNah, sadly this most likely will not work with Q1/Q1.5 implementations - initially I was playing with monte carlo approximations (before I arrived at bucketMul), and the convergence was very slow for binary/ternary networks. Or, in simpler terms - if you have just ones and zeroes, and minus ones, you can remove zeroes from calculations, but that's it. No good method to figure out which ones are more important than the other ones. Also, there are no bits left to store positional information when bucketing. There are some paths that could be explored in this fashion, but it would require a redesign of the algorithm from the ground up.
- throwaway2562 3y agoLate response: thank you. Saved me some time thinking further on this.