4 ms·
But this is not a problem, if you have money to buy 1TB memory, you can easily get a motherboard that supports it for around $800-$1000. It will be cheaper than
by atemerev 2y ago
But this is not a problem, if you have money to buy 1TB memory, you can easily get a motherboard that supports it for around $800-$1000. It will be cheaper than attempting to build a cluster with 4x256GB RAM.
GPU inference is another thing, as high-VRAM GPUs are artificially priced way high so they could only be bought by corporations. However, if you attempt to build a cluster with say 10 4090s to obtain some 240GB VRAM, you won’t have enough electricity to run it at home.
I am currently building a 4x4090 rig, but that’s probably the maximum I could have at home giving my budget / available power restrictions. And that’s only 96G VRAM, slightly more than a single A100.
- angoragoats 2y ago> However, if you attempt to build a cluster with say 10 4090s to obtain some 240GB VRAM, you won’t have enough electricity to run it at home. The average home (in the United States at least) has more than enough power to run 10 GPUs, or even more, with plenty left over for other appliances. Back of the napkin math is that ten high-end GPUs, at 350W each, need approximately 30-35A of power, and these days the bare minimum service in a home is 100A (and many homes have more than this; mine has 200A, for example). This could be accomplished on two standard 15A circuits, or perhaps more comfortably two 20A circuits. You can also throttle your cards down to conserve power; I know that 3090s set to max power of 225W only lose ~5% of performance when running LLM workloads, while saving ~125W of power at full load.
- atemerev 2y agoWell, I live in an apartment in Switzerland, we have different standards.
- angoragoats 2y agoYeah, makes sense. I was just trying to point out that "you won't have enough elecricity to run it at home" is not true for many single-family homeowners. I'm not familiar with the typical Swiss apartment, so I'm sorry if this is a stupid question, but do you not think that your apartment could sustain a ~15A load at 230V? That's how much you'd need for ten RTX 4090s at full power.
- atemerev 2y agoIt is not just the video cards. CPUs, mainboards, cooling systems, losses/inefficiencies… E.g. 4x4090 Comino Grando workstation is rated for 2811W, which is OK. 8x will probably trip my circuit breaker already, unless I plug it into higher-power circuit for kitchen appliances.
- angoragoats 2y agoYep, in the US you’d need to use at least two separate circuits as well. I’m just saying that there probably is sufficient electricity for a crazy 10+ GPU setup, in most single family houses at least.
- BizarroLand 2y agoIf you're going that far out you might as well just tap into your breaker box and add a 40a 110v circuit. As long as you can run your cluster near the breaker box it would not be a huge expense to do that, and if like most homes your breaker is in the garage then that would make venting the excess heat out fairly simple.
- angoragoats 2y agoI don’t condone doing this under any circumstances without first consulting an electrician. Lots of things could go wrong; for example, 40A @ 120V is not a very common configuration, and most standard household receptacles are rated for only 20A at most. Consult an electrician, folks.
- deleted 2y ago[deleted]
- BizarroLand 2y agoThanks, I did mean to imply that you would be paying an electrician to do the work to code, but I did also overlook that someone who wants to run a 405b Llama ai at home might be more of a diy'er than the average person.