7 ms·
An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all
by airhangerf15 1y ago
An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper"
I don't think people realize the size of these compute units.
When the AI bubble pops is when you're likely to be able to realistically run good local models. I imagine some of these $100k servers going for $3k on eBay in 10 years, and a lot of electricians being asked to install new 240v connectors in makeshift server rooms or garages.
- neko_ranger 1y agoFour H100 in a 2U rack didn't sound impressive, but that is accurate: >A typical 1U or 2U server can accommodate 2-4 H100 PCIe GPUs, depending on the chassis design. >In a 42U rack with 20x 2U servers (allowing space for switches and PDU), you could fit approximately 40-80 H100 PCIe GPUs.
- jzymbaluk 1y agoAnd the big hyperscaler cloud providers are building city-block sized data centers stuffed to the gills with these racks as far as the eye can see
- michaelt 1y agoWhy stop at 80 H100s for a mere 6.4 terabytes of GPU memory? Supermicro will sell you a full rack loaded with servers [1] providing 13.4 TB of GPU memory. And with 132kW of power output, you can heat an olympic-sized swimming pool by 1°C every day with that rack alone. That's almost as much power consumption as 10 mid-sized cars cruising at 50 mph. [1] https://www.supermicro.com/en/products/system/gpu/48u/srs-gb200-nvl72 https://www.supermicro.com/en/products/system/gpu/48u/srs-gb...
- handfuloflight 1y agoWhat about https://www.cerebras.ai/system https://www.cerebras.ai/system?
- procaryote 1y ago> as much power consumption as 10 mid-sized cars cruising at 50 mph Imperial units are so weird
- invaliduser 1y agoEven is the AI bubble does not pops, your prediction about those servers being available on ebay in 10 years will likely be true, because some datacenters will simply upgrade their hardware and resell their old ones to third parties.
- belter 1y agoExcept their insane electricity demands will still be the same, meaning nobody will buy them. You have plenty of SPARC servers on Ebay.
- cicloid 1y agoThere is also a community of users known for not making sane financial decisions and keeping older technologies working in their basements.
- dijit 1y agoBut we are few, and fewer still who will go for high power consumption devices with esoteric cooling requirements that generate a lot of noise.
- mattmanser 1y agoSomeone's take on AI was that we're collectively investing billions in data centers that will be utterly worthless in 10 years. Unlike the investments in railways or telephone cables or roads or any other sort of architecture, this investment has a very short lifespan. Their point was that whatever your take on AI, the present investment in data centres is a ridiculous waste and will always end up as a huge net loss compared to most other investments our societies could spend it on. Maybe we'll invent AGI and he'll be proven wrong as they'll pay back themselves many times over, but I suspect they'll ultimately be proved right and it'll all end up as land fill.
- mensetmanusman 1y agoUtterly? Moores law per power requirement is dead, lower power units can run electric heating for small towns!
- deleted 1y ago[deleted]
- wakamana 1y ago[dead]
- ActorNightly 1y agoTo piggyback on this, at enterprise level in modern age, the question is really not about "how are we going to serve all these users", it comes down to the fact that investors believe that eventually they will see a return on investment, and then pay whatever is needed to get the infra. Even if you didn't have optimizations involved in terms of job scheduling, they would just build as many warehouses as necessary filled with as many racks as necessary to serve the required user base.
- semi-extrinsic 1y agoWhat do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.
- quickthrowman 1y agoYou’ll need (2) 240V 20A 2P breakers, one for the server and one for the 1-ton mini-split to remove the heat ;)
- Dylan16807 1y agoMatching AC would only need 1/4 the power, right? If you don't already have a method to remove heat.
- quickthrowman 1y agoCooling BTUs already take the coefficient of performance of the vapor-compression cycle into account. 4w of heat removed for each 1w of input power is around the max COP for an air cooled condenser, but adding an evaporative cooling tower can raise that up to ~7. I just looked at a spec sheet for a 230V single-phase 12k BTU mini-split and the minimum circuit ampacity was 3A for the air handler and 12A for the condenser, add those together for 15A, divide by .8 is 18.75A, next size up is 20A. Minimum circuit ampacity is a formula that is (roughly) the sum of the full load amps of the motor(s) inside the piece of equipment times 1.25 to determine the conductor size required to power the equipment. So the condensing unit likely draws ~9.5-10A max and the air handler around ~2.4A, and both will have variable speed motors that would probably only need about half of that to remove 12k BTU of heat, so ~5-6A or thereabouts should do it, which is around 1/3rd of the 16A server, or a COP of 3.
- Dylan16807 1y ago
- torginus 1y agoI wonder if it's feasible to hook up NAND flash with a high bandwidth link necessary for inference. Each of these NAND chips hundreds of dies of flash stacked inside, and they are hooked up to the same data line, so just 1 of them can talk at the same time, and they still achieve >1GB/s bandwidth. If you could hook them up in parallel, you could have 100s of GBs of bandwidth per chip.
- potatolicious 1y agoNAND is very, very slow relative to RAM, so you'd pay a huge performance penalty there. But maybe more importantly my impression is that memory contents mutate pretty heavily during inference (you're not just storing the fixed weights), so I'd be pretty concerned about NAND wear. Mutating a single bit on a NAND chip a million times over just results in a large pile of dead NAND chips.
- torginus 1y agoNo it's not slow - a single NAND chip in SSDs offers >1GB of bandwidth - inside the chip there are 100+ wafers actually holding the data, but in SSDs only one of them is active when reading/writing. You could probably make special NAND chips where all of them can be active at the same time, which means you could get 100GB+ bandwidth out of a single chip. This would be useless for data storage scenarios, but very useful when you have huge amounts of static data you need to read quickly.
- slickytail 1y agoThe memory bandwidth on an H100 is 3TB/s, for reference. This number is the limiting factor in the size of modern LLMs. 100GB/s isn't even in the realm of viability.
- torginus 1y agoThat bandwidth is for the whole GPU, which has 6 chips. But anyways, what I'm proposing isn't for the high-end and training, but for making inference cheap. And I was somehat conservative with the numbers, a modern budget SSD with a single NAND can do more than 5GB/s read speed.
- torginus 1y agoMy personal sneaking suspicion is that publicly offered models are using way less compute than thought. In modern mixture of experts models, you can do top-k sampling, where only some experts are evaluated, meaning even SOTA models aren't using much more compute than a 70-80b non-MoE model.
- scarface_74 1y agoThis isn’t like how Google was able to buy up dark fiber cheaply and use it. From what I understand, this hardware has a high failure rate over the long term especially because of the heat they generate.
- eitally 1y agoWhat I wonder is what this means for Coreweave, Lambda and the rest, who are essentially just renting out fleets of racks like this. Does it ultimately result in acquisition by a larger player? Severe loss of demand? Can they even sell enough to cover the capex costs?
- dboreham 1y agoThey'll be in landfill in 10 years.
- brikym 1y agoAs a non-American the 240V thing made me laugh.
- thecommakozzi 1y ago[dead]
- shusaku 1y ago> When the AI bubble pops is when you're likely to be able to realistically run good local models. After years of “AI is a bubble, and will pop when everyone realizes they’re useless plagiarism parrots” it’s nice to move to the “AI is a bubble, and will pop when it becomes completely open and democratized” phase
- cootsnuck 1y agoIt's not even been 3 years. Give it time. The entire boom and bust of the dot come bubble took 7 years.
- tootie 1y agoYeah I think the crux of the issue is that chatgpt is serving a huge number of users including paid users and is still operating at a massive operating loss. They are spending truckloads of money on GPUs and selling access at a loss.
- RagnarD 1y agoAn RTX 6000 Pro (NVIDIA Blackwell GPU) has 96GB of VRAM and can be had for around $7700 currently (at least, the lowest price I've found.) It plugs into standard PC motherboard PCIe slots. The Max Q edition has slightly less performance but a max TDP of only 300W.