5 ms·

do you have a simple python impl? :)

kolinko 2y ago
It originally started as a fork to Recmo’s cria pure numpy llama impl :) https://github.com/recmo/cria https://github.com/recmo/cria Took a whole night to compute a few tokens, but I used it to do the first tests. Also, my friend pasted the paper to claude and it produced a working basic impl instantly :D But in all seriousness - I think MLX implementation would be doable, or a wrapper to the Metal gou functionality
- kwikiel 2y ago
  Will share python implementation soon as a kind of executable pseudo code which then can be ported to any platform. This project is kind of like ultimate nerdsnipe as math is quite simple, you don’t need PhD to understand it and actually implementing things would teach you linear algebra faster vs just mindlessly doing exercises sets.
  - kolinko 2y ago
    Haha yes :) Publish it, Kacper! The project is a nerdsnipe for math geeks, because there are multiple small things that beg to be proven / described by math there. For example - what's the tradeoff between the number of bits we loose when embedding position vs the bits of information that we gain by knowing which bucket a weight belongs to? In other words - is it possible that when storing weights in the bucketed form we can actually end up having a higher precision than using a regular form? For Q8 we get just 4 bits to store the weight (and 1 bit for sign, and 3 bits for location), but these 4 bits need to express numbers from a smaller range than before.
brrrrrm 2y ago
not sure why I got downvoted for asking this. seems like a reasonable thing in the ML space anyway here's a numerical simulation written in PyTorch for those who want to consider this algo on their projects before fully integrating: https://gist.github.com/bwasti/78d7058aad7f42dc893d906c877b710c https://gist.github.com/bwasti/78d7058aad7f42dc893d906c877b7... happy hacking
- kolinko 2y ago
  If you got downvoted, it was briefly. That was a good question. PyTorch/numpy is awesome for initial testing.

It originally started as a fork to Recmo’s cria pure numpy llama impl :) https://github.com/recmo/cria https://github.com/recmo/cria Took a whole night to compute a few tokens, but I used it to do the first tests. Also, my friend pasted the paper to claude and it produced a working basic impl instantly :D But in all seriousness - I think MLX implementation would be doable, or a wrapper to the Metal gou functionality

kwikiel 2y ago
Will share python implementation soon as a kind of executable pseudo code which then can be ported to any platform. This project is kind of like ultimate nerdsnipe as math is quite simple, you don’t need PhD to understand it and actually implementing things would teach you linear algebra faster vs just mindlessly doing exercises sets.
- kolinko 2y ago
  Haha yes :) Publish it, Kacper! The project is a nerdsnipe for math geeks, because there are multiple small things that beg to be proven / described by math there. For example - what's the tradeoff between the number of bits we loose when embedding position vs the bits of information that we gain by knowing which bucket a weight belongs to? In other words - is it possible that when storing weights in the bucketed form we can actually end up having a higher precision than using a regular form? For Q8 we get just 4 bits to store the weight (and 1 bit for sign, and 3 bits for location), but these 4 bits need to express numbers from a smaller range than before.

brrrrrm 2y ago

not sure why I got downvoted for asking this. seems like a reasonable thing in the ML space anyway here's a numerical simulation written in PyTorch for those who want to consider this algo on their projects before fully integrating: https://gist.github.com/bwasti/78d7058aad7f42dc893d906c877b710c https://gist.github.com/bwasti/78d7058aad7f42dc893d906c877b7... happy hacking

kolinko 2y ago
If you got downvoted, it was briefly. That was a good question. PyTorch/numpy is awesome for initial testing.