4 ms·

I have some CUDA code that I need to run as fast as possible, so if I was going to blow $1500, my use case would imply that I should go with two 3080s. However

I have some CUDA code that I need to run as fast as possible, so if I was going to blow $1500, my use case would imply that I should go with two 3080s. However, I also play around with deep learning stuff, expect to do so more in the future, but don't currently follow it so closely. Would someone care to ponder on what difference they think a 24GB gpu vs a 10GB gpu will have as a tool for deep learning dev over the next 3 years? For what it's worth, I'm a computer vision guy, but I did have a play with DeepSpeech earlier this year.

godelski 6y ago
Being in the ML space myself I can tell you that memory is pretty important. With my current workload (vision tasks) it is my largest constraint. That said, rumor has it that they will announce a 20Gb 3080 later.
- abledon 6y ago
  and its a 20gb with the ability to do FP16 right? So theoritcally lots of the models can actually be 40gb?
  - atty 6y ago
    Well, they can be 40 GB fp32 models technically, but translating a model trained in fp32 to fp16 is not trivial (trust me, we’re working on this right now for a model). But remember that training the model requires a lot more memory than just the model parameters, because you need to store the gradients as well.
hwillis 6y ago
I'm fairly lay but imo GPT-3 demonstrates pretty soundly that huge models are no magic bullet- it's got >2x as many parameters as the human brain has neurons and it can't do long division. Dogs and other animals get by just fine having less than 1% as many neurons as humans. Even a billion parameters is a huge model, and a factor of 2.4x increase is not going to make a tremendous difference in your performance. In particular the data-heavy nature of vision stuff means that you'll be bottlenecked by training more than memory, AFAIK (again, lay).
- i-am-curious 6y ago
  This is a misleading comparison. You are comparing a massive model with huge models. What you should be comparing are big models vs medium models that a single consumer GPU will fit. And - you don't need to take my word for it, there's tons of papers - the bigger models definitely perform better.
- ipsum2 6y ago
  Model parameters and biological neurons do not map 1:1.
- benlivengood 6y ago
  The human brain has about 100 trillion synapses which is a closer analog to ML model parameters.
- junipertea 6y ago
  It can't do long division because it literally can't read the number inputs, due to the way text is encoded (BPE). That it manages to learn any sort of arithmetic despite that is pretty impressive to me.
dplavery92 6y ago
It's not impossible to distribute training across multiple GPUs, but it's certainly not straightforward. And if you want activations for an entire 4K image in the same model, having a lot of memory on one card is your friend.
6gvONxR4sf7o 6y ago
Just use AWS or GCP. Here's the calculation: Two 3080s = $1500 + number of hours times electricity cost (for me, this would be about $0.14/hour) A K80 from AWS (close enough to the same RAM) = $0.45/hour (spot pricing) or $0.9 (on demand) I've been able to use spot instances for basically all of my AWS hobby work, so I'll use that. The crossover point where it's worth buying is where 1500 + 0.14h = 0.45h, about 5000 hours or 30 weeks of training. For hobby work, do you expect to have it training for 20% of the next 3 years? That's all without considering the fact that you might want to upgrade to a newer card, AWS's prices will likely fall, and you might want a card with more RAM sometimes, but not others, or even a multi-GPU setup sometimes. I used to spend money on fancy cards and machines and justify it with my hobby learning. Now I just let somebody else do the heavy lifting and pay them rent. It's spoiled me. You go from spending a few nights trying to get a thing performant to throwing a burrito worth of money at amazon to just parallelize the dumb thing on a massive machine and having it done by bed time. I also used to justify buying the fancy card with my gaming hobby, but then when I actually did have a long running training job, it'd be super frustrating because now I can't use the desktop for anything else for a few days until training finishes.
- 0xfaded 6y ago
  I have to admit you're right. My case is a little different, where my jobs are about 1h long and it would take me longer to copy all the data to AWS than running it locally.
  - 6gvONxR4sf7o 6y ago
    If you transition your workflow to AWS, and do preprocessing and storage and training in AWS, it avoids the issue. As a benefit, storage in s3 is dirt cheap compared something like dropbox, and archival is easy and cheap enough to just forget about. I feel weird like I'm evangelizing amazon, but it's a workflow i'm very happy with and trying to streamline to take even more advantage of.
- winchester6788 6y ago
  > The crossover point where it's worth buying is where 1500 + 0.14h = 0.45h, about 5000 hours or 30 weeks of training. 1500 + 0.14h should be equal to 0.45h * 4. A 3080 will at the least be 2x faster for general training than one k80.
devilmoon 6y ago
What the other commenter said is absolutely true, but what's even more important is that 3080 doesn't seem to support SLI so you're "stuck" with one 3090 with that budget (which seems to be the only card supporting SLI this gen).
- Const-me 6y ago
  Not an expert in ML, but I don’t think CUDA uses SLI at all. SLI is specific to rendering. Depending on the workload, it’s sometimes makes perfect sense to split GPGPU jobs into multiple GPUs. An extreme example of that approach is crypto-currency miners who sometimes use a dozen of GPUs in a single computer. The only limitation, the working set used by each GPU needs to fit in VRAM of that GPU, otherwise GPUs gonna bottleneck on I/O as opposed to compute, will be very slow. For ML, this means the setup of two 3080 GPUs will be limited to 10GB model sizes.

godelski 6y ago

Being in the ML space myself I can tell you that memory is pretty important. With my current workload (vision tasks) it is my largest constraint. That said, rumor has it that they will announce a 20Gb 3080 later.

abledon 6y ago
and its a 20gb with the ability to do FP16 right? So theoritcally lots of the models can actually be 40gb?
- atty 6y ago
  Well, they can be 40 GB fp32 models technically, but translating a model trained in fp32 to fp16 is not trivial (trust me, we’re working on this right now for a model). But remember that training the model requires a lot more memory than just the model parameters, because you need to store the gradients as well.

hwillis 6y ago

I'm fairly lay but imo GPT-3 demonstrates pretty soundly that huge models are no magic bullet- it's got >2x as many parameters as the human brain has neurons and it can't do long division. Dogs and other animals get by just fine having less than 1% as many neurons as humans. Even a billion parameters is a huge model, and a factor of 2.4x increase is not going to make a tremendous difference in your performance. In particular the data-heavy nature of vision stuff means that you'll be bottlenecked by training more than memory, AFAIK (again, lay).

i-am-curious 6y ago
This is a misleading comparison. You are comparing a massive model with huge models. What you should be comparing are big models vs medium models that a single consumer GPU will fit. And - you don't need to take my word for it, there's tons of papers - the bigger models definitely perform better.
ipsum2 6y ago
Model parameters and biological neurons do not map 1:1.
benlivengood 6y ago
The human brain has about 100 trillion synapses which is a closer analog to ML model parameters.
junipertea 6y ago
It can't do long division because it literally can't read the number inputs, due to the way text is encoded (BPE). That it manages to learn any sort of arithmetic despite that is pretty impressive to me.

dplavery92 6y ago

It's not impossible to distribute training across multiple GPUs, but it's certainly not straightforward. And if you want activations for an entire 4K image in the same model, having a lot of memory on one card is your friend.

6gvONxR4sf7o 6y ago

Just use AWS or GCP. Here's the calculation: Two 3080s = $1500 + number of hours times electricity cost (for me, this would be about $0.14/hour) A K80 from AWS (close enough to the same RAM) = $0.45/hour (spot pricing) or $0.9 (on demand) I've been able to use spot instances for basically all of my AWS hobby work, so I'll use that. The crossover point where it's worth buying is where 1500 + 0.14h = 0.45h, about 5000 hours or 30 weeks of training. For hobby work, do you expect to have it training for 20% of the next 3 years? That's all without considering the fact that you might want to upgrade to a newer card, AWS's prices will likely fall, and you might want a card with more RAM sometimes, but not others, or even a multi-GPU setup sometimes. I used to spend money on fancy cards and machines and justify it with my hobby learning. Now I just let somebody else do the heavy lifting and pay them rent. It's spoiled me. You go from spending a few nights trying to get a thing performant to throwing a burrito worth of money at amazon to just parallelize the dumb thing on a massive machine and having it done by bed time. I also used to justify buying the fancy card with my gaming hobby, but then when I actually did have a long running training job, it'd be super frustrating because now I can't use the desktop for anything else for a few days until training finishes.

0xfaded 6y ago
I have to admit you're right. My case is a little different, where my jobs are about 1h long and it would take me longer to copy all the data to AWS than running it locally.
- 6gvONxR4sf7o 6y ago
  If you transition your workflow to AWS, and do preprocessing and storage and training in AWS, it avoids the issue. As a benefit, storage in s3 is dirt cheap compared something like dropbox, and archival is easy and cheap enough to just forget about. I feel weird like I'm evangelizing amazon, but it's a workflow i'm very happy with and trying to streamline to take even more advantage of.
winchester6788 6y ago
> The crossover point where it's worth buying is where 1500 + 0.14h = 0.45h, about 5000 hours or 30 weeks of training. 1500 + 0.14h should be equal to 0.45h * 4. A 3080 will at the least be 2x faster for general training than one k80.

devilmoon 6y ago

What the other commenter said is absolutely true, but what's even more important is that 3080 doesn't seem to support SLI so you're "stuck" with one 3090 with that budget (which seems to be the only card supporting SLI this gen).

Const-me 6y ago
Not an expert in ML, but I don’t think CUDA uses SLI at all. SLI is specific to rendering. Depending on the workload, it’s sometimes makes perfect sense to split GPGPU jobs into multiple GPUs. An extreme example of that approach is crypto-currency miners who sometimes use a dozen of GPUs in a single computer. The only limitation, the working set used by each GPU needs to fit in VRAM of that GPU, otherwise GPUs gonna bottleneck on I/O as opposed to compute, will be very slow. For ML, this means the setup of two 3080 GPUs will be limited to 10GB model sizes.