8 ms·
All You Need Is 4x 4090 GPUs to Train Your Own Model
- rldjbpin 2y agonice writeup, but i feel that for most people, the software side of training models should be more interesting and accessible. for one, "full" gpu utilization, one or many, remains an open topic in training workflows. spending efforts towards that, while renting from cloud, is a more accessible and fruitful to me than to finetune for marginal improvements. this course was a nice source of inspiration - https://efficientml.ai/ https://efficientml.ai/ - and i highly recommend looking into this to see what to do next with whatever hardware you have to work with.
- m463 2y ago"This needs 30 AMP circuit..." lol
- ratmeadow 2y agoIs anyone else concerned with the power usage of recent AI? Computational efficiency doesn't seem to be a strong point... And for what benefit? IMO the usefulness payoff is too low
- nitred 2y agoCan someone definitively say for sure that I can just use two independent PSUs? One for GPUs and one for GPUs and motherboard and SATA? No additional hardware?
- sabareesh 2y agoHey HN I am sharing my experience on how i pretrained my own LLM by building a ML rig at home
- senectus1 2y agothis is a decent birds eye view thanks, could you expand on this to show how long it took to produce... what model you produced? What did you produce? what did you train for.. the posts seems to suggest its for diffusion purposes?
- sabareesh 2y agoHere is some LLM models not diffusion though. https://huggingface.co/sabareesh88/fw14k https://huggingface.co/sabareesh88/fw14k this post has additional details https://sabareesh.com/posts/llm-intro/ https://sabareesh.com/posts/llm-intro/
- _just7_ 2y agoI would be much more intrested in a piece on what you can train with this kind of rig, rather than the rig itself
- minimaxir 2y agoThe bottleneck for most model training sizes is VRAM, and since each 4090 has 24 GB VRAM, that's 96 GB VRAM total. The article mentions that it can train LLMs from scratch up to 1 billion hyperparameters, which tracks. Nowadays that's not a lot: a single H100 that you can now rent has 80 GB VRAM, and doesn't have the technical overhead of handling work across GPUs.
- sabareesh 2y agoDefinitely agree but part of the reason why i built this to learn about all the overhead and gotchas
- Bancakes 2y agoI doubt the RAM is added up. I think that’s only a feature reserved for their NVLinked HPC series cards. In fact, without nvlink, I don’t see how you’d connect them together to compute a single task in a performant and efficient way.
- whimsicalism 2y agoyeah essentially this
- minimaxir 2y agoIt depends on how the parallelism is implemented, e.g. distributed data parallel (DDP) to synchronize gradients: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html https://pytorch.org/tutorials/intermediate/ddp_tutorial.html It's a rabbit hole I stay away from for pragmatic reasons.
- tmostak 2y agoYou should be able to train/full-fine-tune (i.e. full weight updates, not LoRA) a much larger model with 96GB of VRAM. I generally have been able to do a full fine-tune (which is equivalent to training a model from scratch) of 34B parameter models at full bf16 using 8XA100 servers (640GB of VRAM) if I enable gradient checkpointing, meaning a 96GB VRAM box should be able to handle models of up to 5B parameters. Of course if you use LoRA, you should be able to go much larger than this, depending on your rank.
- master_crab 2y agoAll you need is 4x 4090 GPUs to Train Your Own Model -- and $12000 to buy them
- sabareesh 2y agoI was contemplating between building rig vs using the cloud but for some reason I want to get hands on. So you can always rent them for a fraction of cost
- mcphage 2y agoWell, one or the other, at any rate :-)
- bfung 2y agoAlso (at least in Southern California) electricity prices and how long the rig is on. Not as bad as the initial build cost, but run costs will add up over time.
- echelon 2y agoYou can get a used A100 for that cost and have better software support for training. 4090s are too small for training and you'll have to write your own suboptimal batching. Unless you value the learning, it'd be better to rent GPUs in the cloud for training.
- sabareesh 2y ago
- bb88 2y agoAll you need is a 4x 4090 GPUs and a dedicated 30 amp circuit.
- fzzzy 2y agoYou sure? In my experiments with multi gpu inference, I couldn't get anywhere close to max theoretical power draw.
- sabareesh 2y agoWell during training all GPUs were consuming max ~ 450W
- fzzzy 2y agoThanks, good to know. Perhaps it is different for diffusion; with llms, layers are generally split across gpus, meaning inference has to happen on one gpu before the values can be passed between the layer split.
- bb88 2y agoYes! His power supplies are 2x1500 Watt. That puts it at 3KW max which is more than a 20A circuit can provide (2400W). The standard outlet is typically rated at 15 amps or 1800W. And the 15A breaker is on one circuit. You can get 20A circuits but they need to be wired for it, and replacing the breaker won't cut it. Assuming his GPU is ~450W (his number) and power supplies are 80% efficient, well that means he's pulling close to ~2400 watts which is super close to the limit of a 20A circuit. 4 * 450 / 0.80 efficiency = 2250W. That doesn't include the power consumed by the CPU or mother board or other things on that circuit. But a 170W CPU would easily push this over 2400W provided by a 20A circuit.
- keyle 2y agoThis article was written or rewritten via your model right? The last paragraphs fell totally like AI. Anyway I'd like a follow up on the curating, cleaning and training part which is far more interesting than how to select hardware which we've been doing for over 25 years.
- sabareesh 2y agoYes it is , thanks for the feedback. I will soon add it to github
- red2awn 2y ago> Architecture Advantages: Enhanced ray tracing, Shader Execution Reordering, and DLSS 3 technology for improved efficiency. This jumps right out as written by AI, these features have nothing to do with training LLMs.
- jsheard 2y agoIt's probably better to hold out for the 5090 at this point, it's coming very soon as is expected to have 32GB of VRAM.
- abc-1 2y agoFun for a wealthy hobbyist, but if you want to do real work, you’re better off renting from Runpod. Good blog though.
- sabareesh 2y agoOne of the motivation is to do several distillation, experimentation, research. But as you mentioned there are better ways to do this
- JacksonDam 2y agoInteresting that DLSS 3 is mentioned as an advantage?
- Retr0id 2y agoBecause the article was clearly co-authored by AI
- sabareesh 2y agoIt is co-authored by AI but I left it because it made some indirect sense. I clarified on the parent comment
- sabareesh 2y agoI clarified bit more on the article regarding this. But basically "Well this may not directly provide benefit but because this is a consumer grade card these features enabled having support for more advanced features such as bfloat16 and event float8 training support also the sheer number of cuda cores."
- gzer0 2y agoThis is a great build, thanks for sharing your learnings. The best build I have seen so far had 6x4090's. Video: https://www.youtube.com/watch?v=C548PLVwjHA https://www.youtube.com/watch?v=C548PLVwjHA Specifications - GPU Accelerator - 6 x 24GB NVIDIA GeForce RTX 4090 - Processor - Intel Xeon W7-3465X, 28C/56T, 2.5GHz - 4.8GHz - Memory - 256GB (8x32GB) DDR5 ECC 4800MHz - System Drive - 2TB Samsung 980 PRO NVMe PCIe 4.0 M.2 SSD - Storage Drive - 4TB Samsung 870 EVO SSD - Operating System - Ubuntu 20.04 An interesting choice to go with 256GB of DDR5 ECC; if spending so much on the 6x4090's, might as well try to hit 1 TB of RAM as well. The cost of this... not even sure. Astronomical.
- manmal 2y agoOn Reddit there's reports of 8x4090, or even 8xH100. I don't know where people get this kind of money for this, and why they don't rent infra instead.
- KuriousCat 2y agoWhy do you think RoI is better when infra is rented?
- taskforcegemini 2y agoI mean, at some point someone has to buy them to be able to offer services on them to others. Renting comes with certain limitations owners don't have. And some people have too much money to not invest in fun.
- FridgeSeal 2y agoProbably because they are after a lot of fast, local storage, and _that_ is where rented ML infra providers will sting you. Edit: could also just be more-money-than-sense. Never discount stupidity.
- dogma1138 2y agoHardware can be resold, and bought 2nd hand also. The 4090 will likely maintain 50% of its current value due to its memory capacity over the next 12-18 months. CapEx vs OpEx is a thing even if you are not a business…
- NKosmatos 2y agoWouldn’t a cluster of M4 minis cost less and provide more VRAM? There are posts about people getting decent performance for a lot less than 12k USD.
- whimsicalism 2y agono, these chips are optimized for inference not training & frankly cuda is still table stakes. HN loves it some Apple
- lostmsu 2y agoThey are not optimized for inference vs RTX GPUs.
- sabareesh 2y agoYou get more vram but not enough cores
- angoragoats 2y agoIf you are willing and able to put together the type of system described in the OP (a workstation-class PC, with multiple discrete GPUs and often multiple power supplies), a Mac never makes sense. There are hardware options available at essentially every price point that beat (in some cases drastically) the performance and memory capacity of a Mac. And I say this at the risk of being called pedantic, but a cluster of Mac minis would have zero VRAM.
- lostmsu 2y agoIf you want to wait for over a year to get your model trained (vs 7 days).
- anonytrary 2y agoThanks for sharing. Have you prodded the model with various inputs and written an article that show various output examples? I'd love to get an idea of what sort of "end product" 4x4090s is capable of producing.
- sabareesh 2y agoYou might find more information here helpful https://sabareesh.com/posts/llm-intro/ https://sabareesh.com/posts/llm-intro/ But i am still in process of evaluating post training process with RL. RLHF is almost a mirage that shows what is possible but not the full capability of what model can do
- halyconWays 2y agoWhy not 3090s? Same VRAM and cheaper. With both setups you'd be limited to 1B. By contrast, you can run 4-bit quants of Llama 70B on two {3,4}090s, and it's still pretty lobotomized by modern standards. You can also train your own model even without GPUs. Just depends on parameter size.
- sabareesh 2y agoIt is previous architecture and it doesnt support newer version of Flash Attention , fp8 training etc
- halyconWays 2y agoThat's fair. I did run into that issue when trying to speed up Hunyuan
- jszymborski 2y agoIt is, however, like 3x cheaper.
- patagonianboy 2y agoYeah, it's powerful, but can it run crysis?
- KeplerBoy 2y agoLet's talk riser cables. I keep encountering issues with riser connectors claiming to support PCIe 4.0, which seem to have sub-par performance. They work fine with the GPUs and NICs I tested them with, but attaching a nvme drive causes all kinds of issues and prevents the machine from booting. I guess nvme isn't as tolerant of elevated bit-error-rates. That just doesn't inspire a lot of confidence in those risers, so now I'm contemplating mcio risers.
- Neywiny 2y agoNVMe sits over PCIe. I'd be more inclined to believe they're playing games with their voltage levels to lower power consumption on mobile/embedded (not based on anything but I wouldn't be surprised). Or, if you're then going to an m.2 adapter, something with that.
- sabareesh 2y agoI ran several nccl test had no issue with bandwidth. https://github.com/NVIDIA/nccl-tests?tab=readme-ov-file https://github.com/NVIDIA/nccl-tests?tab=readme-ov-file
- jmward01 2y agoYou can get 4060 ti 16GB cards for ~$450 or 4070 ti 16gb for ~850 instead of the $2.5k for a 4090. I wonder how well 4 of those cards would perform. The 4060 TDP is 165w instead of 450w for the 4090. The 4070 looks like the best tradeoff though for cost/power/etc though. You could probably set up an 8 card 4070 ti 16gb system for less than the 4 card 4090 system
- sabareesh 2y agoI was eyeing 4060 before going with 4090. But it boils down to cuda cores and memory bandwidth
- jmward01 2y agoThe 4090 computer per watt is the best (on paper) between the 4060 ti, 4070 ti and 4090. Best bang for $$ though looks like the 4070ti 16GB. I've been eying that one for a new dual card training rig.
- wruza 2y agoI’ve heard that people buy multiple 24GB P40’s for a bucket of dirt. But that was for inference, not sure about training. g Tesla p40 llm reddit
- magicalhippo 2y agoThe 4060 Ti is hampered by having a narrow memory bus, there's various benchmarks out there, here[1][2] are some examples, and here's[3] one which tests dual 4060 Ti's. [1]: https://www.pugetsystems.com/labs/articles/llm-inference-consumer-gpu-performance/ https://www.pugetsystems.com/labs/articles/llm-inference-con... (8GB model tested but it has same bus width and overall bandwidth as 16GB model) [2]: https://www.reddit.com/r/LocalLLaMA/comments/1b5uwr4/some_graphs_comparing_the_rtx_4060_ti_16gb_and/ https://www.reddit.com/r/LocalLLaMA/comments/1b5uwr4/some_gr... [3]: https://www.reddit.com/r/LocalLLaMA/comments/178gkr0/performance_report_inference_with_two_rtx_4060_ti/ https://www.reddit.com/r/LocalLLaMA/comments/178gkr0/perform...
- 486sx33 2y agoI’d love to hear the dev story of H100 , it seemed to come out of left field !
- Bancakes 2y agoAnyone care to publish AMD training/inference benchmarks using ROCm? They’re hard to find.
- sabareesh 2y agoAt this point it is still not worth considering AMD but may me this will change soon. I would look into semianalysis report
- paxys 2y agoWhere exactly do you plug in this beast?
- AnarchismIsCool 2y agoCouldn't you do better with 2x AGX Orin 64gb?
- xena 2y agoI'd love to read something you wrote, not something you had an AI model write for you.
- magicalhippo 2y agoOn a tangent, if I wished to fine-tune one of those medium sized models like Gemma2 9B or Llama 3.2 Vision 11B, what kind of hardware would I need and how would I go about it? I see a lot of guides but most focus on getting the toolchain up and running, and not much talk about what kind of dataset do I need to do a good fine tuning. Any pointers appreciated.
- MuffinFlavored 2y agoWhat would you expect from fine tuning? What would the input training material be, and what would the expected differences in output be?
- magicalhippo 2y agoIn several cases I've been wanting better prompt adherence. Llama 3.2 Vision is very strictly trained to output a summary at the end which I find difficult to get it stop doing for example. Another one is that when given a math problem and asked to generate some code that computes the result, most models outputs code fine but insists on doing calculations themselves even if the prompt explicitly say they shouldn't. As expected, sometimes these intermediate calculations are incorrect and hence I don't want the LLM to do that when the produced code would handle it perfectly. If the input prompt contains "four times five" I want the model to generate "4 * 5" rather than "20", consistently. I've been curious to see if I could tune them to adhere better to the kind of prompts I would be giving. For LLama 3.2 Vision I've also been curios if I can get it to focus on different details when asked to describe certain images. In many cases it is great but sometimes misses some key aspects. As for the input training material, that's what I'm trying to figure out what I need. I feel a lot of the guides are like that "how to draw an owl" meme[1], leaving out some crucial aspects of the whole process. Obviously I need input prompts and expected answers, but how many, how much variation on each example, and do I need to include data it was already trained on to avoid overfitting or something like that? None of the guides I've found so far touch on these aspects. [1]: https://knowyourmeme.com/memes/how-to-draw-an-owl https://knowyourmeme.com/memes/how-to-draw-an-owl
- pilooch 2y ago