4 ms·

How much does an A100 consume in power a year (in dollar costs)? How much does it cost to hire and retain datacenter techs? How long does it take to expand your

by dekhn 3y ago

How much does an A100 consume in power a year (in dollar costs)? How much does it cost to hire and retain datacenter techs? How long does it take to expand your fleet after a user says "we're gonna need more A100s?" How many discounts can you get as a premier customer? Answer these questions, and the equation shifts a bunch!

shrubble 3y ago
Not really. A full rack with 16 amps usable power and some bandwidth is $400/month in Kansas City, MO. That is enough to power 5x A100s 24x7, so 10k plus $80 per month each, amortized, of course many more A100s would drop the price. Once installed in the rack ($250 1 time cost) you shouldn't need to touch it. So 10k plus $1250 per A100, per year including power. You can put 2 or 3 A100s per cheapo Celeron based CPU with motherboards. Of course if doing very bursty work then it may well make sense to rent...
- akomtu 3y ago
  And how many A100s do you need to do something meaningful with LLMs?
  - shrubble 3y ago
    The funding has to come from somewhere, right? You either pay up front and save money over time, or pay as you go and pay more...
- dekhn 3y ago
  Did you also include the network required to make the A100s talk to each other? Both the datacenter network (so the CPUs can load data) and the fabric (so the A100s can talk?) You also left out the data tech costs- probably at least $50K/individual-year in KC (although I guess I'd just work for free ribs). If you're putting A100s into celeron motherboards... I don't know what to say. You're not saving money by putting a ferrari engine in a prius.
  - shrubble 3y ago
    You are correct that I left out the fabric which needs to be added in. Again a one time expense, though there are different ways to hook it up, it seems, and if the model fits on one system you may not need it. You hire people who are employed by the DC, by the hour for DC work. How much work is there to do once it is screwed into the rack?
latchkey 3y ago
$50m GPU capex (which is A LOT) is about 2-3MW of power, it isn't that much. The problem though is that getting 2-3MW of power in the US is increasingly difficult and you're going to pay a lot more for it since the cheap stuff is already taken. Even more distressing is that if you're going to build new data center space, you can't get the rest of the stuff in the supply chain... backup gennies, transformers, cooling towers, etc...

by dekhn 3y ago

shrubble 3y ago

Not really. A full rack with 16 amps usable power and some bandwidth is $400/month in Kansas City, MO. That is enough to power 5x A100s 24x7, so 10k plus $80 per month each, amortized, of course many more A100s would drop the price. Once installed in the rack ($250 1 time cost) you shouldn't need to touch it. So 10k plus $1250 per A100, per year including power. You can put 2 or 3 A100s per cheapo Celeron based CPU with motherboards. Of course if doing very bursty work then it may well make sense to rent...

akomtu 3y ago
And how many A100s do you need to do something meaningful with LLMs?
- shrubble 3y ago
  The funding has to come from somewhere, right? You either pay up front and save money over time, or pay as you go and pay more...
dekhn 3y ago
Did you also include the network required to make the A100s talk to each other? Both the datacenter network (so the CPUs can load data) and the fabric (so the A100s can talk?) You also left out the data tech costs- probably at least $50K/individual-year in KC (although I guess I'd just work for free ribs). If you're putting A100s into celeron motherboards... I don't know what to say. You're not saving money by putting a ferrari engine in a prius.
- shrubble 3y ago
  You are correct that I left out the fabric which needs to be added in. Again a one time expense, though there are different ways to hook it up, it seems, and if the model fits on one system you may not need it. You hire people who are employed by the DC, by the hour for DC work. How much work is there to do once it is screwed into the rack?

latchkey 3y ago

$50m GPU capex (which is A LOT) is about 2-3MW of power, it isn't that much. The problem though is that getting 2-3MW of power in the US is increasingly difficult and you're going to pay a lot more for it since the cheap stuff is already taken. Even more distressing is that if you're going to build new data center space, you can't get the rest of the stuff in the supply chain... backup gennies, transformers, cooling towers, etc...