3 ms·
I've seen his nano GPT implemented using JAX, now we have C/CUDA. I'd love to see if nano GPT could be doable in Mojo. I took a stab at a Mojo conversion of his
by yinser 3y ago
I've seen his nano GPT implemented using JAX, now we have C/CUDA. I'd love to see if nano GPT could be doable in Mojo. I took a stab at a Mojo conversion of his Wavenet project (Andrej's zero to hero course) and I gotta say... python has so many nice features lol. Stating the obvious I know but what you see done in 6 lines of python takes so much more work in other languages.
- cb321 3y agoFor a prior generation of karpathy-splaining this is this Nim port: https://github.com/Vindaar/llama2nim https://github.com/Vindaar/llama2nim - maybe of interest if you are interested in Mojo.
- yinser 3y agoThank you!
- pavelstoev 3y agoHow in Mojo do you support GPU data parallelism and all the benefits it brings ?
- KeplerBoy 3y agoYou don't. Mojo doesn't support GPUs at the moment, which says a lot about a language which claims to be AI first.
- pjmlp 3y agoThey only made Mojo available outside the preview circle about a couple of months ago, and it is yet to run on Windows laptops of researchers. I love the attitude of considering 0.x languages production ready for all imaginable kinds of workloads.
- yinser 3y agoIf you want CUDA up front go write PyTorch. No one is stopping you. Modular’s goal was to leverage MLIR first and bring GPUs in later. They’re barely a year old company.
- auraham 3y agoWhere is the GPT implementation in JAX? I only found this [1] in PyTorch and NumPy. [1] https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT