7 ms·
Reproducing GPT-2 in llm.c
- indigodaddy 2y agoLooks like this is re: training, but wonder how inference would be on some garbage older machine with no GPU on this model?
- ryankrage77 2y agoLast time I tried GPT-2 on CPU (which I think was shortly before chatGPT was launched), I was getting about 0.2 tokens/sec. CPU utilization was low though, so running inference in parralel gave better results. I was using 2 x E5-2660's.
- int_19h 2y agoDDR5 helps a lot. You can actually run stuff like LLaMA at >1 tok/s on the CPU with high-end gaming hardware these days.
- doubloon 2y agoI have a 24 core Intel cpu and llama3.cpp runs llama3 surprisingly fast in surprisingly little RAM. Yes it becomes a space heater but theres light at the end of the cuda free tunnel
- benterix 2y agoI just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.
- ravetcofx 2y agoAccessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4
- CamperBob2 2y agoSomeone will come along and say "Why don't you just mirror Anna's Archive?" in 3...2...1...
- sebzim4500 2y agoI think between Anna's Archive, fineweb and as many github repos as you can scrape you can get a pretty decent dataset. I doubt Anna's Archive would produce a good model on its own though.
- exe34 2y agoi suppose you wouldn't be able to use it for external services, but internally, I'm sure you can find some books that fell off the back of a truck...
- HeatrayEnjoyer 2y agoNo reason you can't go external. GPT was trained using ebook torrent sites
- artninja1988 2y agoOpenAI has enough money to hire lawyers to defend it until the end of time though
- meiraleal 2y agoI'm okay with paying for datasets
- CamperBob2 2y agoDepends on how the courts rule. If the copyright maximalists prevail, only the wealthiest entities will be able to afford to license a useful data set. Paradoxically enough, this is the outcome that most "Hacker News" denizens seem to be rooting for.
- vineyardmike 2y agoWe won’t ever get there or need to because GPT-4 wasn’t trained on one GPU it was trained on thousands. The (most likely) biggest meaningful difference between -2 and -4 is the number of parameters and the training data/duration. I don’t think you’d really learn much more.
- elicksaur 2y agoIt’s not about learning. It’s about owning. Exactly the reason OpenAI stopped being open. Having GPT-4-quality LLMs created by anyone with a gaming PC would be pretty radical.
- vineyardmike 2y agoAnd you won’t get there. Those models are far too large for a 2024 GPU. Llama-3 70b is arguably close to GPT-4 but is still too large for gaming GPUs (and probably for many years of GPU updates)
- elicksaur 2y ago“You won’t get there” is a pretty vast statement for all of the future. Two fairly reasonable predictions: 1) the compute needed to get GPT4 performance will decrease. 2) the compute on consumer GPUs will increase. At some point they cross, and you will be able to run a GPT4-quality LLM on a consumer GPU. At some point after that, you’ll be able to run a GPT4-quality LLM on a 2024 consumer GPU if you can find one. Important to emphasize, I’m not saying “GPT-4”. Llama-3 was trained on 24k GPU clusters. “Able to do the exact same processing at 1/24k the compute” is different from “Able to get equivalent performance at 1/24k compute”. Even then, given a long enough time scale, the former is possible.
- vineyardmike 2y ago> 1) the compute needed to get GPT4 performance will decrease. 2) the compute on consumer GPUs will increase. I’m assuming we’re just talking inference here… Sure compute abilities for consumers will increase but the original comment had a fixed GPU - the 4090. I can already eke out LLama3:8b on my MacBook Air, and Apple will sell you a laptop capable of running the full sized LLama. There is a direct correlation between parameters and “knowledge” for an LM. There’s some open questions as to density (LLaMa3 specifically challenged previous assumptions) but it seems implausible to fit an equivalent model as GPT4 into 24gb vram. Just like compression, you can’t shrink forever. GPT-4 and GPT-2 are pretty similar architecturally (I assume). So if abilities don’t matter, we can already run GPT-2 so we’re basically there for 4.
- Invictus0 2y agoI'm not saying this to be rude, but I think you have a deep misunderstanding of how AI training works. You cannot just skip the matrix multiplications necessary to train the model, or get current hardware to do it faster.
- xdavidliu 2y agowas the first sentence really necessary? The second sentence seems fine by itself.
- nickpsecurity 2y agoThere's work on replacing multiplication. Here's four examples: https://openaccess.thecvf.com/content_CVPR_2020/papers/Chen_AdderNet_Do_We_Really_Need_Multiplications_in_Deep_Learning_CVPR_2020_paper.pdf https://openaccess.thecvf.com/content_CVPR_2020/papers/Chen_... https://arxiv.org/abs/2012.03458 https://arxiv.org/abs/2012.03458 https://openaccess.thecvf.com/content/CVPR2021W/MAI/papers/Elhoushi_DeepShift_Towards_Multiplication-Less_Neural_Networks_CVPRW_2021_paper.pdf https://openaccess.thecvf.com/content/CVPR2021W/MAI/papers/E... https://arxiv.org/pdf/2106.10860 https://arxiv.org/pdf/2106.10860
- benterix 2y agoNo offence taken! As far as my (shallow!) understanding goes, the main challenge is the need for many GPUs with huge amounts of memory, and it still takes ages to train the model. So regarding the use of consumer GPUs, some work has been done already, and I've seen some setups where people combine of these and are successful. As for the the other aspects, maybe at some point we distill what is really needed to a smaller but excellent dataset that would give similar results in the final models.
- auspiv 2y agoConsidering it takes 8x A100 GPUs (80GB VRAM) to train GPT-2, I think it'll take far more than a single 4090.
- bufo 2y agoThe RTX 4090 has about the same BF16 Tensor Core TOPs than the A100, assuming 50% MFU (like the A100 40 GB PCIe) it would take 8x longer on 1 RTX 4090 vs 8x A100 80GB SXM, so 12 hours. Datasheet here for the TOPs https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid... 50% MFU should be achievable on the 4090.
- anthonix1 2y agoNah, I reproduced on 4x 7900 XTX machine in 8.75 hours, so a single 7900 XTX (costs less than $1k) could do it in under 24 hours. Was hitting 55.4% MFU.
- benterix 2y agoDo you have a link to a description of your setup? I assume there are some quirks on the way and getting this done requires some substantial work. A submission on that would be very interesting, at least to me.
- anthonix1 2y agoFWIW, I'm seeing ~318,000 toks/sec throughput on a 4x AMD 7900 XTX machine (less than $4k worth of GPU), using the same settings as in the post (0.5M batch size etc).
- pama 2y agoDid you reproduce the evaluation as well?
- anthonix1 2y agoIt converges similarly on smaller datasets. About to kick off a training from scratch run on the same fineweb-10B, which at 324k toks/sec should take about 8.6 hours. And with my kWh cost, that is about $2.50 cost to train. Will report back tomorrow when the training has finished..
- anthonix1 2y agoSo... successfully reproduced in ~8.75 hours, taking about 18 kWh / $2.70 The first run actually failed at step 3000 or so, and I realized I had a bug in my attention / matmul kernels, but after fixing that and restarting it worked great [1] https://github.com/anthonix/llm.c https://github.com/anthonix/llm.c
- pama 2y agoWhat was the final loss? Is this hardware available for rent somewhere?
- anthonix1 2y agoFinal loss from that fineweb-10B run (since then I'm up to ~100k toks/sec/GPU): step 18865/18865 | train loss 3.280550 | norm 0.4362 | lr 0.00e+00 | 1669.06 ms | 55.4% A100 fp16 MFU | 314058 tok/s Writing state to log124M/state_00018865_00003.bin val loss 3.296179 You can buy these GPUs on Amazon for under $1k. I heard the MI300X may be available in Azure now or at least very soon.
- 2y ago
- sabareesh 2y agoWell here is a comment on 4090 https://github.com/karpathy/llm.c/discussions/481#discussioncomment-9585564 https://github.com/karpathy/llm.c/discussions/481#discussion...
- huac 2y ago25% MFU :( maybe because of the P2P nerf?
- anthonix1 2y agoMaybe get a 7900 XTX. 122 TFLOPS of BF16/FP16 for less than $1k and I'm getting 55.4% MFU
- sabareesh 2y agoThese are not apples to apple comparison, as this is running across GPU and much bigger model
- sabareesh 2y agoThis much bigger model (500M), P2P is enabled via Mailbox. It is expected because of memory to compute ratio
- huac 2y agocan you elaborate?
- doubloon 2y agoDude im hoping we get rid of Nvidia completely. I can run llama.cpp inference on a 7B model on my 24 core cpu intel machine using just CPU and it only uses about 4gb of ram and is not that slow. If we could have massive parallel arm core or even riscv machines without the Cuda issues with proprietary driver hell it would be much more open source. And much less wonkage for the normie user
- karpathy 2y agoHi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!
- 1024core 2y agoThank you, from an appreciative reader!
- ngiyabonga 2y agoHi Andrej! First, thank you for your teaching, it has helped me a lot, didn't think I'd ever have the chance to say thank you, but here you are and I hope this gets to you! Question - what's a relevant (05-2024) baseline to compare the performance of c code to? Back when you made nanoGPT you were seeing "the file train.py reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of training". So twice the memory on the c node, but unsure of data size /epochs, any other details I may be missing. I.e. what's the net uplift of running c vs "legacy" torch code? Thanks again for everything.
- karpathy 2y agoThe baseline is definitely PyTorch (or JAX), and indeed something like nanoGPT. I just never got nanoGPT "past the finish line" of really crossing the t's and dotting the i's and reproducing the models with as much care as I did now and here in llm.c, and getting to the point where it's a single launch command that just does the thing. I think I'll try to develop the `train_gpt2.py` inside llm.c to be that, so that we have the two implementations exactly side by side, and it's all nice and comparable. The C/CUDA code is currently a little bit faster than PyTorch (last time I measured ~2 weeks ago it was about 6% faster), and I think we can push this further. This is done by manually hard-coding a bunch of fusions/optimizations that are non-trivial for torch.compile to find (e.g. our FusedClassifier). But PyTorch has some pending work/PRs that will also speed up their side a lot. Ultimately my interest in llm.c is to have a nice, clean, minimal, super dependency-light repo in direct C/CUDA implementation, which I find aesthetically pleasing. And on top of that, educational, i.e. using all of the above as an endpoint of an intro LLM course.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- notg963 2y agoDo you have plans to create videos for the llm.c?
- anoy8888 2y agoCan it be done in rust ?
- celltalk 2y agoTime for llm videos!
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- natsucks 2y agoIn your opinion is it important for ML engineers to know C?
- brcmthrowaway 2y ago0% chance
- esafak 2y agoYou'd have to be deep into ML Infrastructure to use C, probably via CUDA. No-one who develops or uses ML models touches C or even C++. tinygrad and llama.cpp are exceptions.
- adeptima 2y agoSpend one year to study multiple languages - bash, C, C++, Go, Python ... and even Mojo or Rust. 10-20 hours a week. Being able to read top programming languages is the best investment I ever made. You will become fearless and can see the matrix ;)
- mode80 2y agoI did this and wrote about my experience: https://mode80.github.io/7-langs-in-12-months.html https://mode80.github.io/7-langs-in-12-months.html I don't regret it. But if ML is your main goal, Python is where you will end up because it's where the libraries are.
- natsucks 2y agointeresting! did you consider Go?
- deleted 2y ago[deleted]
- aliljet 2y agoIs there a reason you're not trying to port this into an even more stack agnostic world without CUDA?
- zimabluerain 2y agothe code works well on H100: https://x.com/Yuchenj_UW/status/1795554739633221804 https://x.com/Yuchenj_UW/status/1795554739633221804
- QuincyYang 2y ago[dead]
- ls612 2y agoIs this the sort of thing that a person with curiosity and a 4090 could do? It says he used 8xA100s in the cloud to do this but is it just a matter of the 4090 going 8x slower or will memory constraints kill the whole endeavour?
- smaddox 2y ago4090 should have enough VRAM for 124M param training. Even at float32 precision, with AdamW optimizer, parameters should only be ~2GB (124M params x 4 bytes per param x ~4 for optimizer weight overhead). So there should be plenty of remaining space for activations.
- adeptima 2y agoAndrej Karpathy karpathy is a magician! But being the coolest kid on the block with pure C/CUDA implementation is not enough https://github.com/karpathy/llm.c https://github.com/karpathy/llm.c Studying a baby Llama 2 model source code in pure Mojo is the next level https://github.com/tairov/llama2.mojo https://github.com/tairov/llama2.mojo https://github.com/tairov/llama2.mojo/blob/master/llama2.mojo https://github.com/tairov/llama2.mojo/blob/master/llama2.moj... Mojo Lang - Tomorrow's High Performance Python? (with Chris Lattner) https://www.youtube.com/watch?v=JRcXUuQYR90 https://www.youtube.com/watch?v=JRcXUuQYR90 Andrej Karpathy and Chris Lattner collab is on my wishlist ;)
- metalloid 2y agoThis awesome! We need a series on how to build the llm.c from the scratch. Any volunteer? :-)
- akkishore 2y agoHi Andrej, Huge fan of all the work you do. Wanted to understand something fundamental and whom better to ask than you: Whats so special about the transformer architecture that its able to predict the next token so beautifully understanding all the intricate previous token relationships? I understand Attention but what so special about this architecture that no other architectures are able to "attend" appropriately to previous tokens? Being a CS guy, its really hard for me to fathom that we have not yet created another architecture which can perform similarly.
- smaddox 2y agoTransformers have quadratic computational complexity in sequence length, i.e. O(N^2) where N is the sequence length. RNNs, Linformer, Mamba, etc. have linear or quasi-linear computational complexity in sequence length, which often bottlenecks information movement across tokens. In theory, if you grew the RNN's state quadratically vs sequence length, you could likely achieve comparable performance to transformers, but it would likely be less efficient than transformers.
- unknown2342 2y agoThank you for your work Andrej! <3