7 ms·
An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without an
by gpm 5mo ago
An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).
- Waterluvian 5mo agoWhat’s the lifespan/refurbishability of the capex elements like the “GPU” modules or even the DRAM soldered into them?
- jmalicki 5mo agoFor lifespan, AWS is still running a ton of T4 GPUs from 2018, that power a lot of computer vision models. A ton of these will have a long life, not all ML is about frontier LLMs.
- epolanski 5mo agoHow can it be economically viable to still run them? You can get 100x the output with the same energy use.
- deleted 5mo ago[deleted]
- jmalicki 5mo agoGood question! Maybe the capabilities of newer GPUs allow AWS to charge higher margins for them? I don't actually know.
- Marsymars 5mo agoPresumably people using AWS are paying more than they cost to run, and AWS has finite bandwidth to upgrade things due to personel, etc.
- HDBaseT 5mo agoThere has not been a "100x" in efficiency in the past 6-8 years.
- dragonwriter 5mo agoWhile the 100× is, I think, rather hyperbolic, there is a real and large efficincy difference, but its economically viable to run them because the supply of newer GPUs is insufficient to meet the demand for compute, so they can charge enough to cover costs for the old ones and a premium (relative to operating costs) for the newer ones. It would be economically unviable to run the older ones if the supply of newer ones were unconstrained, but that’s not the world we live in.
- Dylan16807 5mo agoGoing by the stats on wikipedia, T4 and B300 both do about one teraflop of half-precision math per watt? Where are the efficiency gains? Edit: It looks like they replaced INT8 and INT4 with FP8 and FP4, with the same speedups of 2x and 4x relative to FP16. That's an improvement but not that big of an improvement.
- Ekaros 5mo agoAs long as you have customers that are willing to pay more than it cost you are fine. And with AWS seemingly there is plenty of those. So question isn't is this most efficient way but will someone pay at price that is above what new hardware could attain.
- eldenring 5mo ago2-3x is completely dwarfed by the remaining improvements in training which is still in its infancy relatively
- BearOso 5mo agoUnless there's a new paradigm, scaling up is all they can do to improve performance. They've shrunk down all the way to 1-bit models and all the low-hanging fruit is gone. There's no way for them to get much smaller, so they have to get bigger and faster to meet expectations.
- eldenring 5mo agothis is just not true at all, there are massive leaps from algorithms, data, etc. every year. scale is one axis of many and you need to get them all correct.
- BearOso 5mo agoWhat novel data hasn't already been used in training? What new algorithms are there? Can you post some links so we can read about them?
- intelkishan 5mo agoThis hasn’t been true for the past 2 years
- oblio 5mo agoIs this based on an assumption that Opus 4.7 & co are equivalent or smaller to Opus 4.5 & co? I highly doubt the advanced models (Opus, Pro, etc) aren't biggen than the standard ones (Sonnet, Flash, etc) and fairly sure newer models are bigger than older ones.
- gpm 5mo agoProbably, but at some point we're very likely to run out of significant training improvements and it's not clear that we'll see that point coming from a long way out. Likewise it's probably dwarfed by improvements in how we make dram - continuing the roughly exponential (maybe a bit less recently) scaling of chips - but not necessarily. The 2x from returning to previous costs is interesting because it's practically guaranteed, and it's on top of everything else. We're just currently "overpaying" (relative to the stable market price) for the manufacture of dram because of a sudden increase in demand.
- sandworm101 5mo agoSupply will not meet demand. What incentive do the handful of dram manufacturers have to end the party? This is what happens when legal monopolies finally win control. Dont't worry. The patents will expire in a few decades. Our grandkids will see DDR5 get cheap again. The system functions as intended.
- fitblipper 5mo agoI have fairly simplistic view of the economics involved here. Could you explain why the ability to sell more chips wouldn't be sufficient enough incentive to increase supply?
- brookst 5mo agoNot the person you’re replying to, but RAM has historically been a boom-or-bust business, and companies that invest to meet demand during a boom cycle usually have that new capacity come online just in time for the bust. If it was just variable costs and new capacity was available today they’d do it. But there are substantial fixed costs and delays to increasing capacity, and that uncertainty makes it risky.
- bigbadfeline 5mo agoThat's such a nonsensical argument, it holds for every other business too and in this case it's just a lame excuse for monopolization. If you are that chicken and can't stomach competition you should not be in business anyway. The current RAM manufactures were convicted of conspiracy to manipulate prices back in the 2000s or thereabout, doing so is their modus operandi, but this time the government is participating in the racket.
- rcxdude 5mo agoChip manufacturing has unusually long spin-up times, high capital costs and relatively thin margins for anything but the latest and greatest processes, compared to most industries.
- cubefox 5mo agoFor some reason I still haven't heard any predictions on when new fabs will come online to meet the current demand. This shouldn't be too hard to find out, since the building time of fabs is very predictable process. The difficult question is more whether foreseeable memory demand will remain at the current level, grow even further, or shrink again.
- wmf 5mo agoNo new DRAM fabs are being built. That's why you don't see any predictions.
- cubefox 5mo agoSeems unlikely. Increased demand usually causes increased investment in increasing supply.
- ifwinterco 5mo agoNot necessarily in a notoriously cyclical industry where everyone has already been burned by doing exactly that multiple times
- cubefox 5mo agoWhat would they be doing with their enormous profits instead?
- ifwinterco 5mo agoReturn the money to shareholders instead of incinerating it
- wmf 5mo agoFor one thing they're paying enormous employee bonuses.
- 5mo ago
- andrepd 5mo agoI wonder if we will see an adoption of alternative floating point formats. IEEE floats are notoriously terrible at lower widths (<= 16 bits). Floating point formats such as posits do much better at 16 or 8 bits. If you could train at 16 bits per value instead of 32, and suffer a much smaller inaccuracy penalty than you would from IEEE32 to IEEE16...
- jfim 5mo agoThat's already the case with say bf16
- refulgentis 5mo agoThis has been around for quite some time, to the point I had to read this a couple times to understand what you meant. Mighta predated LLMs even.
- Dylan16807 5mo agoNotoriously terrible? Posits do a little better if your numbers are biased enough toward 1, but not much better. A 16 bit posit in a near-ideal situation matches an 18 bit IEEE float, and in a pretty wide range of situations loses to either fp16 or bf16. Training anything at 8 bits is going to be tough, and it's hard to say if the flexible exponent is worth the precision tradeoffs.
- andrepd 5mo ago> A 16 bit posit in a near-ideal situation matches an 18 bit IEEE float Unsure what you mean by this... A posit16 has up to 11 bits of precision. There's no such thing as an 18 bit IEEE float. > and in a pretty wide range of situations loses to either fp16 or bf16 Many papers have compared neural networks at 16 bits or 8 bits, and posits beat the hell out of floats and it's not even close. Which is very much expected. As they're particularly suited to this task. But also in other domains, like numerical weather simulations, where tests have shown 16-bit posits can replace 32-bit floats.
- Dylan16807 5mo ago
- shevy-java 5mo ago> a path to a ~3x hardware cost reduction Really? How long do we have to wait until that ... cost reduction hits us?
- gpm 5mo agoFor supply to meet demand. Depends very much on how aggressively producers scale and on how demand grows or shrinks. Safe to say at least a year or two. It'd be shocking if it took a decade.
- da_chicken 5mo agoAll the projections I've seen have said that the earliest we might see the curve flatten is 2030. It just takes that long to get a fab up and running.
- radialstub 5mo agoThe memory makers will not expand demand drastically. It is in the nature of their business to keep the market under-supplied, otherwise the following oversupply will kill them. Instead, supply is just rerouted from less profitable segments such as mobile and personal computing.
- jayd16 5mo agoApple could always decide to build their own fab or some such thing.
- simonh 5mo agoThat’s not the Apple way, but they might fund a supplier to build out capacity in return for priority access. The thing is they tend to only do that when they can get a technological competitive advantage. The priority access gives them a locked in competitive edge, for a while. It’s not clear there is an opportunity like that in memory.
- jayd16 5mo agoIt wasn't their way to design CPUs until it was their way.
- golem14 5mo agoDesigning and producing are separate
- nextaccountic 5mo agoApple doesn't want to enter low margin business
- ec109685 5mo agoSupply and demand always balance out. There is no way manufacturers aren’t going to compete away these inflated margins, as long as they feel like this demand is sustainable.
- overfeed 5mo agoIt sure looks like Sam Altman's masterful gambit to corner the memory market has had unforeseen consequences.
- roxolotl 5mo agoIs any of this actually unforeseen? Buying the vast majority of the world’s supply of something does have mostly predictable consequences.
- HDBaseT 5mo agoYes, but that's not what they are insinuating.
- dragonwriter 5mo ago“Unforeseen consequences” in the same way death of the target is when someone aims a loaded gun at their head and pulls the trigger.
- weitendorf 5mo agoIf you factor in Nvidia’s profit margin due to the scarcity of the current bleeding-edge chips there is a path to a much larger cost reduction still. There’s a lot to criticize Sam Altman for saying or popularizing culturally but I’ve come to think his “this is the worst it will ever be” is, in the long run, actually a very intriguing and underrated point. In a decade training LLMs to the current level of sophistication, which is in my opinion rather advanced and probably has lots of additional upside just from constructing better RL training regime independently of hardware advancement, will become just as table stakes as running a database is now. I highly recommend everyone look into the Allen Institute’s projects in GitHub and HF because they have open source training materials (including an LLM from scratch off common crawl, and some quite interesting tunes of qwen) to get a taste for what will be in the near future afternoon projects or educational material. The future is going to be wild
- oblio 5mo agoThese crazy hardware price increases will probably delay everything by at least 2-5 years. Then add at least 5-10 years for all these refinements and optimizations to permeate universally. Until everything matures, most likely the current iteration of OpenAI and Anthropic will be long gone, along with their current business models.
- refulgentis 5mo agoWell, no: manufacturers charge more than input price generally, here specifically, Nvidia wouldn’t lower prices because RAM went down.
- fittingopposite 5mo agoReally wondering what this might mean for local LLMs when RAM costs plummet...
- fittingopposite 5mo ago> either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike Or the more likely scenario that the AI bubble bursts and the hyperscalars realize they have built too many data centers.
- liccil 5mo agoWhat demand? Can't shake the notion that it's fictive considering the amount od data centers being built and GPUs sitting in containers, where they will spend quite some time before being even integrated, even more until used...
- willis936 5mo agoThis line of thinking makes sense if we're talking about opex like power usage. This is capex though and we'll be financing this overpaying for a long time after the hardware has "aged out". Not really sure there is an upside to it. Also, inference cost predictions were made before this price jump, so we really haven't started paying for it yet. Inference will not be getting cheaper.