5 ms·
Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c
- alecco 2y agoAlso https://x.com/karpathy/status/1811467135279104217#m https://x.com/karpathy/status/1811467135279104217#m
- iforiq 2y agoHow much did gpt2 training cost when it came out in 2019?
- ozr 2y agoAbout $50,000.
- withinboredom 2y agoThey probably spent more on the training data, to be honest. They had to get it the hard way.
- jamestimmins 2y agoAnyone have an idea if this is feasible to do on a Macbook with a built-in GPU?
- arthurcolle 2y agoWill take more hours
- michaelmior 2y agoProbably not with the same amount of training time, but I'd imagine a recent MBP GPU could handle GPT-2 training. The biggest challenge is that the training would need to be reimplemented for Metal instead of CUDA.
- jamestimmins 2y agoAh so I couldn't just run this on my laptop for ~48 hours? That's too bad.
- danielmarkbruce 2y agoHe does it on 8 H100's in 24 hours, ie 192 H100 hours. It's going to be thousands of laptop hours.
- deleted 2y ago[deleted]
- mmoskal 2y agoH100 SXM is 2000 TFLOPS at FP16. Multiply by 8. M3 Max is 28 TFLOPS at FP16. Based on FLOPS alone, it would be more like a year or two.
- Davidzheng 2y agoCan you estimate how long it would take to replicate alphago zero today on one set of 8xH100.
- karpathy 2y ago(H100 SXM is 1000 TFLOPS, *2 is from "with sparsity", which is not used here.)
- mmoskal 2y agoRight... and there are probably some communication overheads over NVLink that would not be present on single laptop. So a few months maybe :)
- latchkey 2y agoMI300x is 1300 TFLOPs at FP16 (without sparsity). Looking forward to seeing the results. https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html https://www.amd.com/en/products/accelerators/instinct/mi300/...
- tomalaci 2y agoWith how much NVidia is developing AI-workload accelerating hardware, I expect this will cost maybe few dozen dollars and train in few hours within next few years. What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models. Pretty sure it will spawn a new golden age of AI-powered desktop applications. For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI).
- up2isomorphism 2y ago> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). To me this is a downside compared to the NPC generated by humans, since that’s the only reason I would like to read them.
- MisterBastahrd 2y agoYou don't even necessarily need to have them coming up with valid speech. Simply giving out random quests and rewards would keep people running on a loot treadmill for most open world multiplayer games.
- LeanderK 2y agoI think at first even background NPCs that don't give quests and rewards would be nice. Sort of give everyone a character and let them just babble. It breaks the immersion to hear the same phrases repeatedly. You can still handcraft every important quest and interaction to achieve high fidelity, but I would like random NPCs you bump into to not just repeat things all the time.
- swatcoder 2y agoYou're more likely to see studios (who want lots of more varied content) use generative AI in the studio, where they might generate and review it before release. Letting generators run free on the client sets up different kinds of immersion-breaking, where NPC's hallucinate misleading details about the story/world, can be tricked into reciting off-topic absurdities or age-rating violations, etc. AAA studios can't afford the embarrassment of that and smaller designers with pride of craft won't see their signature come through the art in it. Surely, some designers will figure out ways to make it work great for some specific idea, but it's not the best way to use the technology in most cases.
- deleted 2y ago[deleted]
- alecco 2y agoIt will be interesting to see this with today's FlashAttention 3 for H100.
- rurban 2y agoWould be free for us because we have those H100`s, but currently it's way too hot now. They will reach 70°C, even watercooled.
- deleted 2y ago[deleted]