4 ms·
I can't find how much it will cost or how much power it will use. I mean it will be a lot and maybe only Google and Microsoft and Facebook can afford it but I s
by ftxbro 3y ago
I can't find how much it will cost or how much power it will use. I mean it will be a lot and maybe only Google and Microsoft and Facebook can afford it but I still want to know.
- virtuallynathan 3y agoI would guess about $100m and ~1.5-2MW
- Bedon292 3y agoThe DGX A100 was $200k at launch. I found a DGX H100 in the mid $300k area. And those are 8 GPU systems. So you need 32 of those, and each one will definitely cost more plus networking. Super low estimate would be $500k each for $16M total. But considering its moving from 98GB to 480GB RAM per GPU. Might be more like $1.5M per 8, round it to say $50M. And at 1/8th the power per GB, you have 700 Watts / 96GB / 8 * 480GB come to around 450 Watts per. And 115kw for the 256.
- ftxbro 3y agoWhat does this mean for the AI race? For example what if a newish company (newer than Google/Facebook/Microsoft/etc.) like Anthropic, Scale, Perplexity, or Stability is able to scrape together $5B USD funding and spend their hardware budget on these things. Say that can buy $1B of them and spend the rest on hackers and operating expenses (idk if that's realistic). So maybe they could purchase and operate like 20 of them. Say that they spend six months doing experimental things and then the next six months training their Tsar Model. If they follow the Chinchilla scaling laws and normal architectures, how good will these models be?
- bushbaba 3y agoFor AI race means it opens the door to a competitor. Could be AMD, Google, or Amazon. All which have offerings in this space. However while the hardware isn’t cheap, it’s still likely not a blocker. Costs do inhibitor more experimental research.
- mirekrusin 3y agoStartup which will rent this compute when needed will likely have more advantage on AI front. Selling shovels is good business, but doesn't compete directly in AI area.
- BonoboIO 3y agoI have no expertise in GPU System used for AI Learning, but would It be possible to buy a bunch of consumer cards and get the same performance? Or is this not possible because consumer cards go to 40 ish GB RAM and Models would not fit or „swapping“ like crazy and be slow.
- p1esk 3y agowould It be possible No
- 01100011 3y agoNot the same. Not all problems can be efficiently divided among NUMA nodes with low bandwidth interconnects.
- Tepix 3y agoConsumer cards only have PCIe 4.0, at most 24GB VRAM and the only recent model with NVLink, the RTX 3090, can only be connected to exactly one other card. It doesn't scale beyond that. So you are limited to PCIe 4.0 x16 speeds.
- SirFlamenco 3y agoTechnically the A6000 has 48 Gb of VRAM and works in 2-way NVLink.
- Bedon292 3y agoThe NVLink interconnect on all the GPUs is a huge part of it, and cannot come even remotely close to that bandwidth with consumer goods. Then the density of RAM to compute and power is huge. A single 4090 is 450 watts, for 24GB where this is 20x the memory for the same watts. 2.3Mw or so. If you say $0.14 / kwh, thats something like $325 / hour in power costs to run. Not counting additional cooling you are definitely going to need. And I am sure there is inefficiency this doesn't cover but 240v 10,000+ Amps for that?
- nvy 3y agoWith that much memory I could probably run Crysis 4 and an Electron app side by side!
- bigmattystyles 3y agoI wonder if you can even run this on a regular 20A circuit - I'm thinking no - 20 * 120 = 2400 Watts - I assume that will not be enough...
- deleted 3y ago[deleted]
- zamadatix 3y agoFor a single Grace+Hopper node? I'd bet it fits in that budget, the Grace Hopper datasheet says the combo has a CPU + GPU + memory TDP of 450W - 1000W programmable, and that leaves more than half of the room for the rest of the node's power budget. For the DGX GH200? It's 18,432 CPU cores with 256 GPUs across 16 full racks of servers :p.
- jeffnappi 3y agoHere's an example of an 8x H100 machine - look at the tech specs: https://lambdalabs.com/deep-learning/servers/hyperplane https://lambdalabs.com/deep-learning/servers/hyperplane 6x 3000W PSUs in a 3x3 redundant config. So 9000 watts total. So at least 240v x 50A. x2 for redundancy.
- mattlondon 3y agoMy hob in my kitchen is 7.3kw? Normal 32a * 240 circuit allows up to 7.68kw, and the 6mm^2 cable is rated to something like 45amps This seems fairly common e.g. https://www.currys.co.uk/products/aeg-ikb64401fb-59-cm-electric-induction-hob-black-10188242.html https://www.currys.co.uk/products/aeg-ikb64401fb-59-cm-elect... I am sure data centers have larger circuit breakers and chunkier cables than my kitchen appliances!
- throwaway2037 3y agoWoah, I looked at the specs: Front left: 2.3 kW / 3.7 kW Cripes. You can boil water extremely fast on that IH setup! I'm living with 1.5 kW, and it is painful...
- 3y ago
- einarfd 3y agoIt’s in interest of Nvidia, to try to make sure they do not end up in a situation where they have a small group of very big customer that buy a large slice of their production. For Nvidia a market of the same size, with many small to medium customer is a lot better as those customers will have a lot less power to force Nvidia to do something that isn't in it's interest or it does not want. I expect to see moves from Nvidia to help smaller players, open source or semi open models to not be crushed by the big players. Not because they are nice, but because it is in their best interest.