6 ms·

We did a big analysis of this a few years back. We ended up using a big spot-instance cluster of CPU machines for our inference cluster. Much more consistently

We did a big analysis of this a few years back. We ended up using a big spot-instance cluster of CPU machines for our inference cluster. Much more consistently available than spot GPU, at greater scale, and at better price per inference (at least at the time). Scaled well to many billion inferences. Of course, compare cost per inference on your models to make sure logic applies. Article on how it worked: https://www.freecodecamp.org/news/ml-armada-running-tens-of-billions-of-ml-predictions-on-a-budget-f9505c820203/ https://www.freecodecamp.org/news/ml-armada-running-tens-of-... Training was always GPUs (for speed), non-spot-instance (for reliability), and cloud based (for infinite parallelism). Training work tended to be chunky, never made sense to build servers in house that would be idle some of the time, and queued at other times.

varunkmohan 4y ago
Disclaimer: I'm the Cofounder / CEO at Exafunction That's a great point. We'll be addressing this in an upcoming post as well. We've served workloads that run entirely on spot GPUs where it makes sense since a small number of spot GPUs can make up for a large amount of spot CPU capacity. The best of all worlds is if you can manage both spot and on-demand instances (with a preference towards spot instances). Also, for latency sensitive workloads, running on spot instances or CPUs sometimes is not an option. I could definitely see cases where it makes sense to run on spot CPUs though.
- fortysixdegrees 4y ago
  Disclaimer != Disclosure Probably one of HNs most common mistakes in comments
  - adgjlsfhk1 4y ago
    perhaps, but I think disclaimer in this context it's just an abbreviation since the disclosure carries with it the implicit disclaimer of "so the things I'm saying are subconsciously influenced by the fact that they potentially could make me money"
  - fragmede 4y ago
    The "so I'm biased and take my advice under advisement" is implied, so disclaimer works.
machinekob 4y ago
What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training? Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090. And if you are solo dev its even easier choice as you can reuse your rig for other stuff when you dont train anything (for example gaming :D). Only possibility is if you get free 100k from AWS and then 100k from GCP you can live with that for a year or even two if u stack both providers but it is special case and im not sure how easy it is to get 100k right now.
- varunkmohan 4y ago
  You have a good point. I think for small enough workloads self managing instances on-prem is more cost-effective. There is a simplicity gain in being able to scale up and scale down instances in the cloud but may not make sense if you can self-manage without too much work.
- beecafe 4y ago
  You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.
  - mushufasa 4y ago
    > You are years behind if you think you're training a model worth anything on consumer grade GPUs Ah yes, my code can't be useful to people unless it takes a long time to compile...
    - skimo8 4y ago
      To be fair, I think ML workloads are quite a bit different than the days of compiling over lunch breaks. What the above post was probably trying to get at is that the ML specific hardware is far more efficient these days than consumer GPUs.
ramoz 4y ago
We did a similar analysis for GCP. Preemptibles/spot were the way to go with inference. CPU performance was also faster for our scaled inference workloads. Times change though, we’re about to conduct the same analysis over again, with latest models better architected for accelerators.

varunkmohan 4y ago

Disclaimer: I'm the Cofounder / CEO at Exafunction That's a great point. We'll be addressing this in an upcoming post as well. We've served workloads that run entirely on spot GPUs where it makes sense since a small number of spot GPUs can make up for a large amount of spot CPU capacity. The best of all worlds is if you can manage both spot and on-demand instances (with a preference towards spot instances). Also, for latency sensitive workloads, running on spot instances or CPUs sometimes is not an option. I could definitely see cases where it makes sense to run on spot CPUs though.

fortysixdegrees 4y ago
Disclaimer != Disclosure Probably one of HNs most common mistakes in comments
- adgjlsfhk1 4y ago
  perhaps, but I think disclaimer in this context it's just an abbreviation since the disclosure carries with it the implicit disclaimer of "so the things I'm saying are subconsciously influenced by the fact that they potentially could make me money"
- fragmede 4y ago
  The "so I'm biased and take my advice under advisement" is implied, so disclaimer works.

machinekob 4y ago

What cloud is even remotely worth it over buying 20x rtx 3090 or even some quadro for training? Maybe if u have very small team and small problems but if you have CV/Video tasks and team more than 3 maybe even 2 people in house servers are always better choice as you'll get your money back in 2-3 months of training over cloud solution and maybe even more if you wait for rtx 4090. And if you are solo dev its even easier choice as you can reuse your rig for other stuff when you dont train anything (for example gaming :D). Only possibility is if you get free 100k from AWS and then 100k from GCP you can live with that for a year or even two if u stack both providers but it is special case and im not sure how easy it is to get 100k right now.

varunkmohan 4y ago
You have a good point. I think for small enough workloads self managing instances on-prem is more cost-effective. There is a simplicity gain in being able to scale up and scale down instances in the cloud but may not make sense if you can self-manage without too much work.
beecafe 4y ago
You are years behind if you think you're training a model worth anything on consumer grade GPUs. Table stakes these days is 8x A100 pods, and lots of them. Luckily you can just get DGX pods so you don't have to build racks but for many orgs just renting the pods is much cheaper.
- mushufasa 4y ago
  > You are years behind if you think you're training a model worth anything on consumer grade GPUs Ah yes, my code can't be useful to people unless it takes a long time to compile...
  - skimo8 4y ago
    To be fair, I think ML workloads are quite a bit different than the days of compiling over lunch breaks. What the above post was probably trying to get at is that the ML specific hardware is far more efficient these days than consumer GPUs.

ramoz 4y ago

We did a similar analysis for GCP. Preemptibles/spot were the way to go with inference. CPU performance was also faster for our scaled inference workloads. Times change though, we’re about to conduct the same analysis over again, with latest models better architected for accelerators.