4 ms·

An interesting question, shows how insanely overpriced GPUs still are, specially in the cloud environment

pqn 4y ago
Disclaimer: I work at Exafunction I empathize a bit with the cloud providers as they have to upgrade their data centers every few years with new GPU instances and it's hard for them to anticipate demand. But if you can easily use every trick in the book (CPU version of the model, autoscaling to zero, model compilation, keeping inference in your own VPC, using spot instances, etc.) then it's usually still worth it.
- lowdose 4y ago
  Not to mention AWS has had a GPU cloud offering monopoly because Google Cloud and Microsoft Azure were publicly available until 2019.
- fomine3 4y ago
  GCP still provides NVIDIA K80. I wonder is it still worth to hold.
  - varunkmohan 4y ago
    I think you'd probably always want to go with T4's since they are the same price unless there's just no availability for them.
NavinF 4y ago
*only in the cloud environment Throw some 3090s in a rack and you’ll break even in 3 months
- wmf 4y ago
  Because that's "illegal" so cloud providers can't do it.
  - jgauth 4y ago
    Can you describe what you mean by that?
    - wmf 4y ago
      The GeForce driver EULA doesn't allow it to be used in servers or something like that, so clouds all have to use the more expensive professional cards.
mistrial9 4y ago
the HPC crowd are not able to add GPUs, that I know of.. deepLearning group of algorithms do kick butt for lots of kinds of problems+data .. though I will advocate that dl is NOT the only game in town, despite what you often read here
- Frost1x 4y ago
  In what context? HPC and certain code bases have been effectively leveraging heterogenous CPU GPU workloads for a variety of applications for quite awhile. I know of some doing so in at least 2009 and know plenty of prior art was already there by that point, it's just a specific time I happen to remember.
  - mistrial9 4y ago
    ok - the academic study in front of me dated 2020 says "no" but it is non-US researchers, public science. I have no reason to believe one way or the other, but I literally read this today. reading again - it seems this paper calls HPC with GPUs a slightly different name "GPGPU" and lists the research activity separately.. so I didn't see it as HPC; basically what I wrote is not accurate. got it

pqn 4y ago

Disclaimer: I work at Exafunction I empathize a bit with the cloud providers as they have to upgrade their data centers every few years with new GPU instances and it's hard for them to anticipate demand. But if you can easily use every trick in the book (CPU version of the model, autoscaling to zero, model compilation, keeping inference in your own VPC, using spot instances, etc.) then it's usually still worth it.

lowdose 4y ago
Not to mention AWS has had a GPU cloud offering monopoly because Google Cloud and Microsoft Azure were publicly available until 2019.
fomine3 4y ago
GCP still provides NVIDIA K80. I wonder is it still worth to hold.
- varunkmohan 4y ago
  I think you'd probably always want to go with T4's since they are the same price unless there's just no availability for them.

NavinF 4y ago

*only in the cloud environment Throw some 3090s in a rack and you’ll break even in 3 months

wmf 4y ago
Because that's "illegal" so cloud providers can't do it.
- jgauth 4y ago
  Can you describe what you mean by that?
  - wmf 4y ago
    The GeForce driver EULA doesn't allow it to be used in servers or something like that, so clouds all have to use the more expensive professional cards.

mistrial9 4y ago

the HPC crowd are not able to add GPUs, that I know of.. deepLearning group of algorithms do kick butt for lots of kinds of problems+data .. though I will advocate that dl is NOT the only game in town, despite what you often read here

Frost1x 4y ago
In what context? HPC and certain code bases have been effectively leveraging heterogenous CPU GPU workloads for a variety of applications for quite awhile. I know of some doing so in at least 2009 and know plenty of prior art was already there by that point, it's just a specific time I happen to remember.
- mistrial9 4y ago
  ok - the academic study in front of me dated 2020 says "no" but it is non-US researchers, public science. I have no reason to believe one way or the other, but I literally read this today. reading again - it seems this paper calls HPC with GPUs a slightly different name "GPGPU" and lists the research activity separately.. so I didn't see it as HPC; basically what I wrote is not accurate. got it