3 ms·

I’m confused don’t a100s cost 10,000 to buy? Why would you pay 166k per year to rent?

sidnb13 3y ago
I would assume the datacenter and infra needed would also contribute a sizeable chunk to the costs when you consider upkeep to run it 24/7
- deleted 3y ago
  [deleted]
latchkey 3y ago
For the same reason people use AWS. Spending the capex/opex to run a cluster of compute isn't easy or cheap. It isn't just the cost of the GPU, but the cost of everything else around it that isn't just monetary.
- etothepii 3y ago
  This could be an interesting comparison. My experience with AWS is that it was super easy and cheap to start on. By the time we could use whole servers we were using so much AWS orchestration that it's going to be put off until we are at least $1M ARR, and probably til we are at $5M. Make adoption easy, give a free base tier but charge more could be a very effective model to get start ups stuck on you. It even probably makes adoption by small teams in big companies possible that can then grow ...
dekhn 3y ago
How much does an A100 consume in power a year (in dollar costs)? How much does it cost to hire and retain datacenter techs? How long does it take to expand your fleet after a user says "we're gonna need more A100s?" How many discounts can you get as a premier customer? Answer these questions, and the equation shifts a bunch!
- shrubble 3y ago
  Not really. A full rack with 16 amps usable power and some bandwidth is $400/month in Kansas City, MO. That is enough to power 5x A100s 24x7, so 10k plus $80 per month each, amortized, of course many more A100s would drop the price. Once installed in the rack ($250 1 time cost) you shouldn't need to touch it. So 10k plus $1250 per A100, per year including power. You can put 2 or 3 A100s per cheapo Celeron based CPU with motherboards. Of course if doing very bursty work then it may well make sense to rent...
  - akomtu 3y ago
    And how many A100s do you need to do something meaningful with LLMs?
    - shrubble 3y ago
      The funding has to come from somewhere, right? You either pay up front and save money over time, or pay as you go and pay more...
  - dekhn 3y ago
    Did you also include the network required to make the A100s talk to each other? Both the datacenter network (so the CPUs can load data) and the fabric (so the A100s can talk?) You also left out the data tech costs- probably at least $50K/individual-year in KC (although I guess I'd just work for free ribs). If you're putting A100s into celeron motherboards... I don't know what to say. You're not saving money by putting a ferrari engine in a prius.
    - shrubble 3y ago
      You are correct that I left out the fabric which needs to be added in. Again a one time expense, though there are different ways to hook it up, it seems, and if the model fits on one system you may not need it. You hire people who are employed by the DC, by the hour for DC work. How much work is there to do once it is screwed into the rack?
- 3y ago
amluto 3y ago
Those are 8x A100 systems.
joefourier 3y ago
AWS is extremely overpriced for nearly every service. I don’t know why anyone else outside of startups with VC money to burn or bigcos that need the “no one ever got fired for buying IBM” guarantee would use them. You’re better off with Lambdalabs or others which charge only $1.1/h per A100. Also that is a 8xA100 system as others have noted, but it is the 40GB one which can be found on eBay for as low as $3k if you go with the SXM4 one (although the price of supporting components may vary) or $5k for the PCI-e version.
- wg0 3y ago
  There are only two services that are dirt cheap and way too reliable, useful.That's S3 and SQS. Rest can get very expensive very soon. You can build a lot of stuff on top of these two.
  - ommpto 3y ago
    Even for S3 while the storage is dirt cheap they still have exorbitant bandwidth pricing.
    - wg0 3y ago
      Oh yes. You're right.
  - charcircuit 3y ago
    S3 is not dirt cheap. Bandwidth is ludicrously expensive.
- charlesischuck 3y ago
  You pay for the system not the gpu with AWS. It's absolutely worth the money when you look at the whole picture. Also lambda labs never has availability. I actually can schedule a distributed cluster on AWS.
  - AndroTux 3y ago
    > It's absolutely worth the money when you look at the whole picture. That highly depends on many things. If you run a business with a relatively steady load that doesn't need to scale quickly multiple times per day, AWS is definitely not for you. Take Let's Encrypt[1] as an example. Just because cloud is the hype doesn't mean it's always worth it. Edit: Or a personal experience: I had a customer that insisted on building their website on AWS. They weren't expecting high traffic loads and didn't need high availability, so I suggested to just use a VPS for $50 a month. They wanted to go the AWS route. Now their website is super scalable with all the cool buzzwords and it costs them $400 a month to run. Great! And in addition, the whole setup is way more complex to maintain since it's built on AWS instead of just a simple website with a database and some cache. [1] https://news.ycombinator.com/item?id=37536103 https://news.ycombinator.com/item?id=37536103
nharada 3y ago
Sometimes I need 512 GPUs for 3 days.
charlesischuck 3y ago
A top end gpu now to make you competitive cost 20-50k per gpu. To train a top model you need hundreds of them in a very advanced datacenter. You can't just plug gpus into standard systems and train, everything is custom. The technical talent required for these systems is rare to say the least. The technical talent to make a model is also rare. I trained a few foundation models with images, and I would NEVER buy any of them. These guys are on a wildly different scale than basically everyone.

I would assume the datacenter and infra needed would also contribute a sizeable chunk to the costs when you consider upkeep to run it 24/7

deleted 3y ago
[deleted]

latchkey 3y ago

For the same reason people use AWS. Spending the capex/opex to run a cluster of compute isn't easy or cheap. It isn't just the cost of the GPU, but the cost of everything else around it that isn't just monetary.

etothepii 3y ago
This could be an interesting comparison. My experience with AWS is that it was super easy and cheap to start on. By the time we could use whole servers we were using so much AWS orchestration that it's going to be put off until we are at least $1M ARR, and probably til we are at $5M. Make adoption easy, give a free base tier but charge more could be a very effective model to get start ups stuck on you. It even probably makes adoption by small teams in big companies possible that can then grow ...

dekhn 3y ago

How much does an A100 consume in power a year (in dollar costs)? How much does it cost to hire and retain datacenter techs? How long does it take to expand your fleet after a user says "we're gonna need more A100s?" How many discounts can you get as a premier customer? Answer these questions, and the equation shifts a bunch!

shrubble 3y ago
Not really. A full rack with 16 amps usable power and some bandwidth is $400/month in Kansas City, MO. That is enough to power 5x A100s 24x7, so 10k plus $80 per month each, amortized, of course many more A100s would drop the price. Once installed in the rack ($250 1 time cost) you shouldn't need to touch it. So 10k plus $1250 per A100, per year including power. You can put 2 or 3 A100s per cheapo Celeron based CPU with motherboards. Of course if doing very bursty work then it may well make sense to rent...
- akomtu 3y ago
  And how many A100s do you need to do something meaningful with LLMs?
  - shrubble 3y ago
    The funding has to come from somewhere, right? You either pay up front and save money over time, or pay as you go and pay more...
- dekhn 3y ago
  Did you also include the network required to make the A100s talk to each other? Both the datacenter network (so the CPUs can load data) and the fabric (so the A100s can talk?) You also left out the data tech costs- probably at least $50K/individual-year in KC (although I guess I'd just work for free ribs). If you're putting A100s into celeron motherboards... I don't know what to say. You're not saving money by putting a ferrari engine in a prius.
  - shrubble 3y ago
    You are correct that I left out the fabric which needs to be added in. Again a one time expense, though there are different ways to hook it up, it seems, and if the model fits on one system you may not need it. You hire people who are employed by the DC, by the hour for DC work. How much work is there to do once it is screwed into the rack?
3y ago

amluto 3y ago

Those are 8x A100 systems.

joefourier 3y ago

AWS is extremely overpriced for nearly every service. I don’t know why anyone else outside of startups with VC money to burn or bigcos that need the “no one ever got fired for buying IBM” guarantee would use them. You’re better off with Lambdalabs or others which charge only $1.1/h per A100. Also that is a 8xA100 system as others have noted, but it is the 40GB one which can be found on eBay for as low as $3k if you go with the SXM4 one (although the price of supporting components may vary) or $5k for the PCI-e version.

wg0 3y ago
There are only two services that are dirt cheap and way too reliable, useful.That's S3 and SQS. Rest can get very expensive very soon. You can build a lot of stuff on top of these two.
- ommpto 3y ago
  Even for S3 while the storage is dirt cheap they still have exorbitant bandwidth pricing.
  - wg0 3y ago
    Oh yes. You're right.
- charcircuit 3y ago
  S3 is not dirt cheap. Bandwidth is ludicrously expensive.
charlesischuck 3y ago
You pay for the system not the gpu with AWS. It's absolutely worth the money when you look at the whole picture. Also lambda labs never has availability. I actually can schedule a distributed cluster on AWS.
- AndroTux 3y ago
  > It's absolutely worth the money when you look at the whole picture. That highly depends on many things. If you run a business with a relatively steady load that doesn't need to scale quickly multiple times per day, AWS is definitely not for you. Take Let's Encrypt[1] as an example. Just because cloud is the hype doesn't mean it's always worth it. Edit: Or a personal experience: I had a customer that insisted on building their website on AWS. They weren't expecting high traffic loads and didn't need high availability, so I suggested to just use a VPS for $50 a month. They wanted to go the AWS route. Now their website is super scalable with all the cool buzzwords and it costs them $400 a month to run. Great! And in addition, the whole setup is way more complex to maintain since it's built on AWS instead of just a simple website with a database and some cache. [1] https://news.ycombinator.com/item?id=37536103 https://news.ycombinator.com/item?id=37536103

nharada 3y ago

Sometimes I need 512 GPUs for 3 days.

charlesischuck 3y ago

A top end gpu now to make you competitive cost 20-50k per gpu. To train a top model you need hundreds of them in a very advanced datacenter. You can't just plug gpus into standard systems and train, everything is custom. The technical talent required for these systems is rare to say the least. The technical talent to make a model is also rare. I trained a few foundation models with images, and I would NEVER buy any of them. These guys are on a wildly different scale than basically everyone.