4 ms·
Oh thanks bro, nope it uses the simple llama 2 rope with tetha changed to 50k to match llama 3's. I'll check your python PR, have a deeper look at the meta llam
by AMICABoard 2y ago
Oh thanks bro, nope it uses the simple llama 2 rope with tetha changed to 50k to match llama 3's. I'll check your python PR, have a deeper look at the meta llama 3 & 3.1 implementation and hack together something soonish. Awesome!
- danielhanchen 2y agoNp! Oh ye my PR is just a random conversion from Python to C - haven't done C in ages though so probs won't even compile!
- AMICABoard 2y agoTo be honest your pr and these notes are super helpful cos, otherwise I'd have been too lazy to read up the original implementation, but I can't merge it. Will make the fix soonish and credit it to you. I'll tell you a secret, nobody is good at C except for a few wizards like woz, Linus, jart, Knuth, Bellard and the like. I am also a total looser at C. I just love C. And also the base of this is mostly Karpathy's work and other awesome folks who are way better at everything than me.
- danielhanchen 2y agoOh thanks! :) Oh ye Karpathy's repo is pretty sick! Well still great work on it! If you need any help on it, feel free to ask!
- AMICABoard 2y agoWill do! Thanks :)
- danielhanchen 2y ago:)