4 ms·

Any clue about how well ML frameworks do with multiple cards? I'm completely bottlenecked on RAM, but for what I do, even a 3060 would be adequate performance.

by blagie 4y ago

Filligree 4y ago
For training? The note cards the better, up to a couple hundred. Custom code may be required, but your bottleneck will be the GPU count. For inference? You can’t generally shard an AI across multiple cards, but you can run multiple instances. Depends on what your trying to do. In no case will it be a simple linear speedup, and you certainly can’t add the memory together.
- blagie 4y ago
  I do much more inference than training, but a bit of both. Speed is generally not a serious issue: - I'm not trying to scale to millions of users on inference; it's a dev machine - Training is a bit of fine-tuning; I'm not building big models from scratch. - Worst-case speed has gone into "annoying" territory, but never into prohibitive. Worst case, I can grab a coffee while things run. - It's not a lot of coffee, since I'm trying to visualize some interesting things about some data. The majority of that is data cleaning, visualization, data engineering, UI/UX, meetings, and a million other things. My GPU is sitting idle >>95% of the time. I am constrained by RAM. In the era of GPT2 -> GPT3, Stable Diffusion, etc., RAM becomes a major bottleneck. There are cloud workflows, but they aren't a great fit, both due to data security / regulatory issues, to how we plan to do dev ops / deployment, as well as just to time for integration. Dropping $8k on 4x 24GB GPUs would be a no-brainer if it let me work with bigger models. I'm sure people can describe Better Ways to Do Things, but changing workflows, system architecture, etc. to those workflows would be obviously out-of-scope for what I'm trying to do. At that point, I'm better off working with smaller models. They're good enough.

Any clue about how well ML frameworks do with multiple cards? I'm completely bottlenecked on RAM, but for what I do, even a 3060 would be adequate performance.

by blagie 4y ago

Any clue about how well ML frameworks do with multiple cards? I'm completely bottlenecked on RAM, but for what I do, even a 3060 would be adequate performance.

Filligree 4y ago

For training? The note cards the better, up to a couple hundred. Custom code may be required, but your bottleneck will be the GPU count. For inference? You can’t generally shard an AI across multiple cards, but you can run multiple instances. Depends on what your trying to do. In no case will it be a simple linear speedup, and you certainly can’t add the memory together.

blagie 4y ago
I do much more inference than training, but a bit of both. Speed is generally not a serious issue: - I'm not trying to scale to millions of users on inference; it's a dev machine - Training is a bit of fine-tuning; I'm not building big models from scratch. - Worst-case speed has gone into "annoying" territory, but never into prohibitive. Worst case, I can grab a coffee while things run. - It's not a lot of coffee, since I'm trying to visualize some interesting things about some data. The majority of that is data cleaning, visualization, data engineering, UI/UX, meetings, and a million other things. My GPU is sitting idle >>95% of the time. I am constrained by RAM. In the era of GPT2 -> GPT3, Stable Diffusion, etc., RAM becomes a major bottleneck. There are cloud workflows, but they aren't a great fit, both due to data security / regulatory issues, to how we plan to do dev ops / deployment, as well as just to time for integration. Dropping $8k on 4x 24GB GPUs would be a no-brainer if it let me work with bigger models. I'm sure people can describe Better Ways to Do Things, but changing workflows, system architecture, etc. to those workflows would be obviously out-of-scope for what I'm trying to do. At that point, I'm better off working with smaller models. They're good enough.