3 ms·
It originally started as a fork to Recmo’s cria pure numpy llama impl :) https://github.com/recmo/cria https://github.com/recmo/cria Took a whole night to com
by kolinko 3y ago
It originally started as a fork to Recmo’s cria pure numpy llama impl :)
https://github.com/recmo/cria https://github.com/recmo/cria
Took a whole night to compute a few
tokens, but I used it to do the first tests.
Also, my friend pasted the paper to claude and it produced a working basic impl instantly :D
But in all seriousness - I think MLX implementation would be doable, or a wrapper to the Metal gou functionality
- kwikiel 3y agoWill share python implementation soon as a kind of executable pseudo code which then can be ported to any platform. This project is kind of like ultimate nerdsnipe as math is quite simple, you don’t need PhD to understand it and actually implementing things would teach you linear algebra faster vs just mindlessly doing exercises sets.
- kolinko 3y agoHaha yes :) Publish it, Kacper! The project is a nerdsnipe for math geeks, because there are multiple small things that beg to be proven / described by math there. For example - what's the tradeoff between the number of bits we loose when embedding position vs the bits of information that we gain by knowing which bucket a weight belongs to? In other words - is it possible that when storing weights in the bucketed form we can actually end up having a higher precision than using a regular form? For Q8 we get just 4 bits to store the weight (and 1 bit for sign, and 3 bits for location), but these 4 bits need to express numbers from a smaller range than before.