5 ms·

anyone got a guide on setting up and running the business-class stuff (70B models over multiple A100, etc)? i'd be willing to spend the money but only if i coul

anyone got a guide on setting up and running the business-class stuff (70B models over multiple A100, etc)? i'd be willing to spend the money but only if i could get a good guide on how to set everything up, what hardware goes with what motherboard/ram/cpu, etc.

foundry27 2y ago
Aye, there’s the kicker. The correct configuration of hardware resources to run and multiplex large models is just as much of a trade secret as model weights themselves when it comes to non-hobbyist usage, and I wouldn’t be surprised if optimal setups are in many ways deliberately obfuscated or hidden to keep a competitive advantage Edit: outside the HPC community specifically, I mean
- deleted 2y ago
  [deleted]
- codybontecou 2y ago
  The economic barrier to entry probably has a lot to do with it. I'd happily dig into this problem and share my findings but it's simply too expensive for a hobbyist that isn't specialized in it.
dkkergoog 2y ago
[dead]
talldayo 2y ago
I don't think you're going to find much of a guide out there because there isn't really a need for one. You just need a Linux client with the Nvidia drivers installed and some form of CUDA runtime present. You could make that happen in a mini PC, a jailbroken Nintendo Switch, a gaming laptop or a 3600W 4U rackmount. The "happy path" is complicated because you truly have so many functional options. You don't want an A100 unless you've already got datacenter provisioning at your house and an empty 1U rack. I genuinely cannot stress this enough - these are datacenter cards for a reason. The best bang-for-your buck will be consumer-grade cards like the 3060 and 3090, as well as the bigger devkits like the Jetson Orin.
nickpsecurity 2y ago
People shared them regularly on r/LocalLlama: https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ Search for terms like hardware build, running large models, multiple GPU’s, etc. Many people there have multiple, consumer GPU’s. There’s probably something about running multiple A100’s. HuggingFace might have tutorials, too. Warning: If it’s A100’s, most people say to just rent them from the cloud as needed cuz they’re highly costly upfront. If they’re normally idle, then it’s not as cost effective to own them. Some were using services like vast.ai to get rentals cheaper.
- bubaumba 2y ago
  it may be even cheaper to use big models through API. Claude, GPT, whatever. Rental is efficient only for big butches, while API is priced per call/size and is cheap for small models.
Aurornis 2y ago
> anyone got a guide on setting up and running the business-class stuff (70B models over multiple A100, etc)? If you need a guide, the A100 class cards are certainly not what you want. Running data center level clusters in the home is no walk in the park. You wouldn’t even be able to power a decent sized cluster without getting an electrician involved for proper power. Networking is expensive. Even second-hand it all adds up fast. Don’t underestimate how much it would cost. Getting some 3090s in a case with a big gaming PSU is the way to go for home use. If you really want A100 level, rent it from a cloud provider. Trying to build your own cluster would be like lighting money on fire for a rapidly depreciating asset.
- jwrallie 2y ago
  Question: what can you do with the A100s that you cannot to with multiple 3090s? Or rephrasing for somebody that never tried to experiment with models: which one do you need to have something like ChatGPT-4o (text only) at home?
- rcdwealth 2y ago
  [dead]

foundry27 2y ago

Aye, there’s the kicker. The correct configuration of hardware resources to run and multiplex large models is just as much of a trade secret as model weights themselves when it comes to non-hobbyist usage, and I wouldn’t be surprised if optimal setups are in many ways deliberately obfuscated or hidden to keep a competitive advantage Edit: outside the HPC community specifically, I mean

deleted 2y ago
[deleted]
codybontecou 2y ago
The economic barrier to entry probably has a lot to do with it. I'd happily dig into this problem and share my findings but it's simply too expensive for a hobbyist that isn't specialized in it.

dkkergoog 2y ago

[dead]

talldayo 2y ago

I don't think you're going to find much of a guide out there because there isn't really a need for one. You just need a Linux client with the Nvidia drivers installed and some form of CUDA runtime present. You could make that happen in a mini PC, a jailbroken Nintendo Switch, a gaming laptop or a 3600W 4U rackmount. The "happy path" is complicated because you truly have so many functional options. You don't want an A100 unless you've already got datacenter provisioning at your house and an empty 1U rack. I genuinely cannot stress this enough - these are datacenter cards for a reason. The best bang-for-your buck will be consumer-grade cards like the 3060 and 3090, as well as the bigger devkits like the Jetson Orin.

nickpsecurity 2y ago

People shared them regularly on r/LocalLlama: https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/ Search for terms like hardware build, running large models, multiple GPU’s, etc. Many people there have multiple, consumer GPU’s. There’s probably something about running multiple A100’s. HuggingFace might have tutorials, too. Warning: If it’s A100’s, most people say to just rent them from the cloud as needed cuz they’re highly costly upfront. If they’re normally idle, then it’s not as cost effective to own them. Some were using services like vast.ai to get rentals cheaper.

bubaumba 2y ago
it may be even cheaper to use big models through API. Claude, GPT, whatever. Rental is efficient only for big butches, while API is priced per call/size and is cheap for small models.

Aurornis 2y ago

> anyone got a guide on setting up and running the business-class stuff (70B models over multiple A100, etc)? If you need a guide, the A100 class cards are certainly not what you want. Running data center level clusters in the home is no walk in the park. You wouldn’t even be able to power a decent sized cluster without getting an electrician involved for proper power. Networking is expensive. Even second-hand it all adds up fast. Don’t underestimate how much it would cost. Getting some 3090s in a case with a big gaming PSU is the way to go for home use. If you really want A100 level, rent it from a cloud provider. Trying to build your own cluster would be like lighting money on fire for a rapidly depreciating asset.

jwrallie 2y ago
Question: what can you do with the A100s that you cannot to with multiple 3090s? Or rephrasing for somebody that never tried to experiment with models: which one do you need to have something like ChatGPT-4o (text only) at home?
rcdwealth 2y ago
[dead]