8 ms·
We're in the brute force phase of AI – once it ends, demand for GPUs will too
- overcast 2y agoAI will go the way of ASICs, just like bitcoin.
- danielmarkbruce 2y agoAn H100 is already close to ASIC. "GPU" is just a path dependent historical naming.
- brrrrrm 2y agoIt kinda has already with fixed matrix multiplication units. But beyond that, no chance. Bitcoin is an unchanging hash algo, not a developing software
- Rinzler89 2y agoIt's already there. Have you seen the six figure AI chips that Nvidia is selling to the data center customers? Those chips are no GPUs, they can't draw a single triangle or map a single texture, they're AI accelerators all the way. People still think Nvidia is selling gaming GPUs for AI workloads like it's 2018? Google, Meta, et-all are working on their own AI chips but those chips will have to beat Nvidia's at Performance and TCO and Nvidia shows no signs of slowing down to let competitors catch up.
- kolinko 2y agoThe chips are optimised for matmuls, but not for transformer architecture per se. With dedicated ASICS, and weights hardcoded (or stored in SRAM) we could theorically get 1 token per one cycle - so millions/billions of tokens per second, not hundreds. Etched, for example claims they have a chip reaching 500k tok/s in the works. Which is still far from the theoretical max with the current techology. A similar scenario went with Bitcoin's GPU/FPGA/ASIC - the current ASICs are millions of times faster than GPUs.
- deleted 2y ago[deleted]
- throwthrowuknow 2y agoThat’s fine if you never need to improve the model, which is valid in some use cases, but for chat style interaction or even code generation you’ll regularly have to update the weights.
- kolinko 2y agoDepends on a chip architecture - etched claims 0.5M tok/s with weights that can be updated. The main constraint is with the model architecture, where it needs to be specific transformer-based model. But they claim the chip can do both Mixtral and Llama - so the constraints are not too stiff.
- matwood 2y ago> beat Nvidia's at Performance and TCO TCO, yes. Raw performance, not necessarily. TCO will attack NVDA's margins. When Meta last wrote about their cluster it was presented as power equivalent to X NVDA chips. They are already bringing their own chips into the mix.
- posix_compliant 2y agoWith Bitcoin I feel like it’s different, since the hashing algorithm would only ever change during a fork. This is rare in that it only ever happens every few years. With AI, we’re constantly training different models, which can’t be trained using asics. If we ever get to the point where we no longer need to train new models, then yeah, it will go the way of bitcoin.
- TacticalCoder 2y ago> With Bitcoin I feel like it’s different, since the hashing algorithm would only ever change during a fork. This is rare in that it only ever happens every few years. Wait what!? Did the Bitcoin hashing algorithm ever change?
- glitchc 2y agoIndeed, you can strip out a whole host of things from the GPU, the framebuffer, the Z-buffer, the transform and lighting engine, instead filling it with more CUDA cores and a higher bandwidth memory controller with a larger bus, etc. And, as it happens, that's exactly what NVidia's done with the H100: https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ https://developer.nvidia.com/blog/nvidia-hopper-architecture... It still needs to be programmable though. Can't get away from that.
- adastra22 2y agoYou don’t need general programmability for AI inference.
- glitchc 2y agoThe money's in the training, not the inference. If you look at Apple and Google, they already have their own hardware for inference in their smartphones. They don't need NVidia for that.
- mupuff1234 2y agoApple and Google use TPUs for training. https://www.cnbc.com/2024/07/29/apple-says-its-ai-models-were-trained-on-googles-custom-chips-.html https://www.cnbc.com/2024/07/29/apple-says-its-ai-models-wer...
- 2y ago
- vidarh 2y agoGroq is one example (NOT Musk's Grok), though currently focused only on inference, I think.
- kkielhofner 2y ago> just like bitcoin The problem with this comparison is Bitcoin has basically just been SHA256 for 15 years and likely will continue to be for some time. Transformers have been mostly dominant for at least several years but there are still other archs (CNN, RNN, etc) in various use-cases and we're already seeing nearly-fundamental changes in Transformers and "emerging" approaches like Mamba, RWKV, hybrids, etc. Transformers have shown remarkable versatility and adaptability (that's their whole thing) but it's already creaking and showing its age. Startups building Transformer-specific silicon are playing a very risky game that is already somewhat problematic now and almost certainly won't end well. AI is much newer, much more vast, and moving much more quickly. The ASIC design, tape out, manufacture, software ecosystem, actually getting to market, etc cycle is fundamentally too long and I suspect even the Transformer-specific silicon we see now will be viewed as a major blunder in the relatively near future: "Oh yeah, remember those graveyard companies that did transformer silicon back in the first AI hype round?" I cannot see how anything other than GPGPU, TPU, NPU, etc (or similar "generic" approaches) will have legs.
- dboreham 2y agoAI generated article?
- darby_nine 2y agoI really wish the title would use "will scale with demand more slowly" rather than saying demand will end, which is trivially false.
- adastra22 2y agoIf the market overbuys GPUs in anticipation of further scaling and that doesn’t occur, the second-hand market will eat up the primary market entirely. Demand for NEW GPUs will plummet to near zero.
- SpicyLemonZest 2y agoI don't really buy it. The advantage of massively parallel operations seems to me to be fundamental in the architecture of modern AI systems, not something that could eventually be optimized away through an "elegant way of programming". It feels like hypothesizing some clever technique that would let you run graphics through your CPU.
- HarHarVeryFunny 2y agoHis argument seems to be more about the limited utility of GenAI, specifically LLMs (vs image generation), than about more efficient AI techniques. We do seem to be at peak LLM hype right now, with C-suite execs thinking of "GenAI" as some sort of proto-AGI that can do magic, and OpenAI/etc still claiming that AGI is just around the corner if only investors will foot the bill for $100B data centers to train ever bigger models. It's not clear what level of datacenter/GPU investment is needed for inference vs training - all the talk of massive GPU clusters seems to be about training needs, not inference. As far as efficiency, presumably we'll eventually switch to smarter (brain-like) dataflow type designs with incremental learning, but for time being we're stuck synchronously pumping 100K contexts through hundreds of transformer layers, and trillion token pre-training runs. Brute force indeed!
- mensetmanusman 2y agoYou will need 10-100x the number of GPUs to get video working. If GPUs crash in price, video will take off and then GPUs will be scarce again.
- hiddencost 2y agoThat's literally never happened. Machine learning is a trade off between model size (training cost), model run time (inference cost), and quality. When some task is solved (e.g., hot word detection or speech to text), it becomes a commodity and some harder task becomes the priority.
- thecupisblue 2y agoAh yes, because LLMs and Generative AI are the peak of AI technology. And in these two years they have been popular, we should have already seen trillion gazillion dollar revenues. While yes, it is "brute force" and we'll probably find a way (and already have) to run it on vanilla machines, there is so much more to AI than LLM's and generating photos. And even LLM's and photo generation open up a bajillion usecases that would have taken years of research and development before. But nobody focuses on these - instead, they focus on S&P 500 companies and how they haven't earned much with GenAI. Because these companies are usually known as the peak of human creativity, imagination and are ready to jump on a new technology without much red tape in it. Honestly, it's been like two-three years tops. Even talking to tech startup CEO's I don't get the feeling they remotely understand the technology or application, as 90% of things I've heard them say is "oooh we could make a chatbot!" or "let's replace developers with it - oh it can't one shot generate my whole codebase? pft that sucks". If these folks don't know how to use it, surely Jim VP of Engineering #62 at ACME & CO that hasn't used any tech except ERP's for the last 10 years will have an idea how to.
- perching_aix 2y agoDo non-managers / not-business-owners have a better way of using these technologies? Or who exactly are you envisioning as possessing that supposed pinnacle of human creativity?
- thecupisblue 2y ago"non-managers / not-business-owners" usually do have a better way of using these technologies, unless the former have actually worked in the field or in the frontlines and then advanced to managers. It might sound condescending, but a lot, and I mean a loooooot of product managers, product heads, product owners, VP's, CEO's and such don't have a clue what they're doing or experience in the real world. One might be "oh but why are they CEO then", but hell, corporate incompetence is a real thing. I've met dozens, if not hundreds of PMs/POs who were hired based on "oh they have organisational skills and aren't an autist when it comes to talking". >"Or who exactly are you envisioning as possessing that supposed pinnacle of human creativity" Creative people. It's okay to say some people aren't creative. Most ticket-dragging meeting-slacking Jira people aren't it. I've met devs who had brilliant ideas on AI incorporation, only to be dismissed by the higher ups because they didn't understand it. I've talked to designers who's ideas could save hundreds of man-hours if implemented. But do you really think an average VP knows that they can use their existing component library to train an MMM to output pseudo-code from screenshot and then translate that into their real, existing components? Or that an LLM could be hooked up to auto-correct the mistakes in the input of tens of thousands workers that they actually have people check, wasting human souls on what is basically input formatting? Or shit, that it could even translate John from Warehouse's data directly into monthly reports without him having to go around asking stuff and wasting everyone's time? No, these people usually don't have a clue about the real-world process of actual work that goes on so, so of course they have a problem identifying leverage spots in it.
- habitue 2y agoI disagree, I think the fluke was the era in which we didn't have enough work to do for CPU/GPU to be at 100% 24/7. Think of this like leaving money on the table: compute should always be useful, why aren't those cores pegged to max all the time? I think it was literally lack of imagination. We were like "well, I automated most of the paper pushing we used to do in the office, guess my job is done!" and this occupied 0.001% of a computer's time. We invented all sorts of ways for people to only pay for that tiny slice of active time (serverless, async web frameworks, etc). Now we're in an era where we can actually use the computers we've built. I don't think we're going back
- dgacmu 2y agoEven once you have a core sitting in a datacenter, the value you derive from the computations must exceed the power cost (and the SWE cost to run that computation). Otherwise you're better off turning it off. I have a stack of 1080ti and Titan V GPUs that are testament to this. :-) (which, admittedly, I should sell)
- eli_gottlieb 2y agoWell if you're looking to sell, get in touch!
- habitue 2y agoThis is where the lack of imagination comes in (not you in particular, this is everyone right now). I'm postulating something non-obvious and pretty contentious, but I think compute should always be more valuable than the cost of the power it consumes.
- yifanl 2y agoThat's an obviously wrong statement: at some point of power consumption, you've burned the earth to ashes, and there's no returns on compute that can pay that off.
- 2y ago
- janalsncm 2y agoEven if generative AI went to zero there are still tons of GPU-hungry ML methods with tons of business applications. GPU demand isn’t going away.
- onlyrealcuzzo 2y agoIf the demand is flat and GPU performance doubles every 2 years, you don't need that many years before sales essentially go to zero. In just 4, you're down 75%. Demand can't "not go anywhere". Demand has to continue to go through the roof for sales to stay where they're at. For sales to continue increasing at the rate they were, demand would have to be insane.
- dartos 2y ago> and GPU performance doubles every 2 years That’s a big assumption.
- onlyrealcuzzo 2y agoSee Huang's Law: https://en.wikipedia.org/wiki/Huang%27s_law https://en.wikipedia.org/wiki/Huang%27s_law
- dartos 2y agoFrom that same Wikipedia page > There has been criticism. Journalist Joel Hruska writing in ExtremeTech in 2020 said "there is no such thing as Huang's Law", calling it an "illusion" that rests on the gains made possible by Moore's law; and that it is too soon to determine a law exists.[9] The research nonprofit Epoch has found that, between 2006 and 2021, GPU price performance (in terms of FLOPS/$) has tended to double approximately every 2.5 years, much slower than predicted by Huang's law Regardless, there is a ceiling to how fast hardware can be. Can’t just double forever.
- myrmidon 2y agoDemand being independent of price/performance is a oversimplifying assumption that is very obviously flawed, and this does not even require continuously discovering novel applications -- there is already plenty of stuff that would get viable with lower costs/better availability of computing power right now.
- emehex 2y agoMay I interest you in Jevons paradox: > In economics, the Jevons paradox occurs when technological progress increases the efficiency with which a resource is used (reducing the amount necessary for any one use), but the falling cost of use induces increases in demand enough that resource use is increased, rather than reduced. Source: https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
- rachofsunshine 2y agoAs a concrete application: software development has gotten infinitely easier in recent decades. Better IDEs, fewer performance constraints, better virutalization, less worries about on-prem deployments, autocomplete, easy version control, you name it. Any given engineer should be orders of magnitude more productive - yet demand has (recent slump aside) only grown.
- meiraleal 2y agoProductivity which is cancelled by the procrastination and mental issues social media brought to us all.
- visarga 2y agoAnd computers have become 1 million times better in the last 2-3 decades. Higher frequency, more RAM, more bandwidth, more network peers. And yet we make even more of them. > "I think there is a world market for maybe five computers." Thomas Watson, president of IBM, 1943
- ForHackernews 2y agohttps://en.wikipedia.org/wiki/Thomas_J._Watson#Famous_attribution https://en.wikipedia.org/wiki/Thomas_J._Watson#Famous_attrib...
- dghlsakjg 2y agoWhich, mythological provenance aside, was a very accurate statement at the time. In 1943 there were probably only a few governments capable of paying the extreme cost, and having a valid use case, for a "computer" in 1943. If you asked him what he thought the market was for a device that cost a month's wages, and could connect to anywhere on earth, do infinite math, remember anything perfectly, and entertain the whole household was, he probably would have had a different answer.
- lores 2y agoWe've always been in the brute force phase of AI. Much of AI is throw things at the wall and see what sticks.
- dartos 2y agoI mean that how machine learning works, right? Millions or billions of iterations to tune some algorithms to output some data which satisfies a success metric.
- danielmarkbruce 2y agoAI should be renamed "Brute Force Statistical Computing". BFSC. It is what it is.
- nick3443 2y agoIt's stochastic gradient descent, so slightly better than blind brute force
- danielmarkbruce 2y agoThere are various optimizers... but sure, it's slightly better than brute force. SBTBFSC.
- lores 2y agoIndeed, and "hyperparameter tuning", i.e. "try every combination but give that an impressive name so people don't raise their eyebrow at us"
- marcosdumay 2y agoThere's nothing on machine learning that makes convergence inherently hard. And there's certainly something we are doing wrong that makes it so data-intensive.
- dartos 2y ago
- bluGill 2y agoGPUs have been on most computers for decades now. Vector operations have long been known as useful for a lot of different tasks. Many of those tasks are long running enough that shunting them off to a different core as long made sense. Thus GPUs have been in everything and manufactures of computers have long been trying to figure out how to use those GPUs for workloads that don't need the full power for graphics. For some tasks GPUs are better, for others CPUs with vector operations are better. There is enough room for both on modern computers and this doesn't look to change.
- amelius 2y agoWe're also in the alchemy phase of AI.
- CharlieDigital 2y agoWe're really at the cusp of gen AI and we've barely scratched the surface. Two Reddit threads really highlight this. - ~10 years ago: https://www.reddit.com/r/StableDiffusion/comments/y9zxj1/you_think_thats_insane_heres_what_an_aigenerated/ https://www.reddit.com/r/StableDiffusion/comments/y9zxj1/you... - Today: https://www.reddit.com/r/StableDiffusion/comments/1f0b45f/flux_is_a_gamechanger_for_character_wardrobe/ https://www.reddit.com/r/StableDiffusion/comments/1f0b45f/fl... The upgrade in throughput from GPT-4 to GPT-4o and GPT-4o Mini actually unlocked use cases for the startup I'm at. People that think demand for GPU compute capacity is going to decrease are probably wrong in the same way that people who thought the demand for faster processors and more RAM would wane were wrong. We are just barely at the start of finding the use cases and how to eat those GPU cycles. > The need for specialist hardware, he observed, is a sign of the "brute force" phase of AI, in which programming techniques are yet to be refined and powerful hardware is needed. "If you cannot find the elegant way of programming … it [the AI application] dies," he added. The thing is that even if there is an elegant and efficient programmatic/algorithmic solution, having more and faster hardware only makes it better and pushes the limits even more.
- dartos 2y ago> We're really at the cusp of gen AI and we've barely scratched the surface. What makes you say that? I don’t really see a trend of AI generated content getting better, just more players in the space. I think we’re at the peak of AI gen, I doubt we’ll see much improvement in quality (it’s already pretty good and it seems like all the low hanging fruit is gone), just more specialized models. Maybe some better tooling to give artists more control having seen it grow more and more since 2016 when GANs started making fairly realistic human faces, this seems like the end goal already.
- CharlieDigital 2y ago> I don’t really see a trend of AI generated content getting better You see that second link as the endpoint? That there's nowhere to go from there? How about you can have a holodeck type experience with Apple Vision Pro? Literally generate any scenario you want? Download generated scenarios and customize it however you want in real time? Entire animation workflows changed from animating models to using voice and text to describe scenes and actions. Lowering the barrier of digital film making to the same level and ease of use as photo editing apps today -- even easier. You really think that the second link is the peak of gen AI? You really think that nothing else and no more major industry shifts are going to happen when gen AI gets cheaper, faster, algorithms get better, and hardware gets more powerful?
- daveguy 2y agoThe premise of this whole article is that once general purpose computing can do what the GPUs can then demand for GPUs will drop. That is a fundamentally flawed assumption. The ability to parallelize operations using a GPU will always be available and GPU development will continue. Hardware tech (process nodes, etc) that improves CPUs will also improve GPUs. Maybe we will reach a peak demand, but not until individual GPUs and CPUs are in millisecond token inference range. And that won't happen for a long time. The author is erroneously conflating GPU and ASIC development. To be clear, I agree that LLMs are not anywhere close to AGI and I don't think they ever will be (just a component). But that doesn't mean they aren't useful enough to chew up a lot of compute for the foreseeable future.
- HarHarVeryFunny 2y ago> To be clear, I agree that LLMs are not anywhere close to AGI and I don't think they ever will be (just a component). But that doesn't mean they aren't useful enough to chew up a lot of compute for the foreseeable future. Sure, but the question is how much compute? What if we're reaching the asymptotic limit of scaling (but still with some post-training and dataset curation gains to be had), and existing datacenters go from being used for training to inference instead. How long before another data center needs to be built or updated with latest NVIDIA GPUs? Is this (LLM-based AI) just a GPU-upgrade market, or still one growing explosively ?
- danjl 2y agoThe reasons GPUs are useful is that they allow us to stay closer Moore's curve. Power is the problem, not AI algorithms. AI algorithms are a tensor processing best case right now. It took us decades to develop widely parallel algorithms for graphics rendering. AI will develop new algorithms. And algorithms will be optimized to be more sparse, which will make them less of an ideal case for the widely parallel hardware. But we will likely need both serial and widely parallel cores to handle any future algorithm.
- janwas 2y agoBrute force hardware locks us into the local minimum of matmul. Stella Nera demonstrated better power efficiency than GPUs, using approx matmul. Maybe the next generation of special purpose HW can help, too. But they come after 2-4 years. Who knows what we will want then? I'd rather have flexible HW.
- baal80spam 2y agoSome people just can't stand that this (supposed) bubble won't pop.
- boringg 2y agoBetter question is who is going to fill in the gap between supply and demand and how fast will those prices drop.
- throwaway4aday 2y agoI'm going to tap the sign again: - [X] Text - [X] Images - [X] Audio - [ ] Videos (in progress) - [ ] 3D Meshes and Textures (in progress) - [ ] Genetics (in progress) - [ ] Physics Simulation (in progress) - [ ] Mathematics - [ ] Logic and Algorithms aka Planning and Optimization - [ ] Reasoning - [ ] Emotion - [ ] Consciousness We still have a lot of data to crunch but it's not nearly enough so we're also going to have to collect and generate a lot more of it. Some of these items require data that we don't even know how to collect yet. Barring some kind of disastrous event, draconian regulation, or politically/culturally motivated demonization of ML I don't see GPU demand dropping any time soon.
- danielmarkbruce 2y agoPlus we will re-crunch it a bajillion times and run inference a bajillion times.
- pipo234 2y agoExtrapolation is dangerous. People tend to overestimate what's possible in a year while underestimating the possibilities next decade. So far we're mostly seeing huge investments hoping for a short term goals while we're not sure whether long term goals (like: Logic and Algorithms aka Planning and Optimization) would even benefit from more compute. Maybe, yes. Shedding some hindsight on earlier extrapolations — The billions pored into the metaverse or self-driving didn't yield the results we expected in the period we expected.
- throwaway4aday 2y agoWhile I agree that we don't want to extrapolate too much I disagree that this type of exploration may not benefit from more compute. We won't know until we try and since we have what seems to be a very generalizable architecture it makes sense to take the brute force approach of creating models of that data by scaling the amount of data and the amount of compute we dedicate to it. If it turns out not to work then we've learned something. As it turns out, Logic and Algorithms has seen some early success using Transformers (Searchformer) https://arxiv.org/abs/2402.14083 https://arxiv.org/abs/2402.14083
- ravenstine 2y agoEven if there is an AI bubble and it pops, lowering the demand for GPUs, I don't see the overall demand for GPUs or even AI becoming less relevant. The lowering of demand would have mostly to do with folks getting off the hype train. Fundamentally, GPUs are about parallel processing, and that's not going obsolete. But if demand for GPUs goes down, that cam be a very good thing because that might also bring down their prices. I would call the end of this "brute forcing" more of a healthy market correction and less of "see, I told you AI is bullshit."
- blueyes 2y agoThis is dumb. It's kind of like saying: "once more processing power is no longer useful, we will no longer need our large human brains." SOTA will remain at the edge of what compute can produce for a long time to come. SOTA is a moving frontier, and there will be demand for incrementally smarter models, because they save you time by making fewer mistakes. No matter how much more efficient algorithmic innovations make ML model training, compute will make those algorithms smarter. It's a coefficient.
- SoftTalker 2y agoCompanies like Gartner must be panicking. LLMs can easily spit out the buzzwords and MBA-textbook advice that their analysts currently produce.
- marcosdumay 2y agoPeople do not hire Gartner for the content they provide. I don't know why people hire them. But I know for sure it's not because of the content.
- deleted 2y ago[deleted]
- BiteCode_dev 2y agoOnce it ends, we will buy more GPUs because any small website will want its own. I know that as soon as I can output 100req/s on the cheap on a llama-level model I will put it EVERYWHERE. And my clients too. DMCA handling? Content flagging alerts? Fuzzy categorization? Natural UI for end user complex queries? All LLM baby. And much, much more.
- devinprater 2y agoI think of it like speech synthesizers. First they were their own machines, then cards you plug into a computer, then once people figured out how to mash human speech together, they were, in some cases, a good 1.5 GB. Now, Siri voices, which are tons better than the concatinative models, used with the VoiceOver screen reader are a good 70 MB, Google TTS, even though it's awful and laggy with TalkBack, offline voices are a good 30 MB for a language pack, and in iOS 18, we can use our own voices as VoiceOver voices. So I think eventually we'll figure out how to run amazing AI stuff, even better than today, on our devices. And I think tons more people are working on LLM's than were ever working on TTS systems.
- ein0p 2y agoDoubtful. Everyone is so GPU starved right now that many research directions can’t even be pursued. That’s why almost everyone is basically training the same architecture with minor variations. Once/if that starvation ends, research will dramatically expand.
- kjellsbells 2y agoTwo thought experiments I like to consider are: - if a company, say, AMD, found a way to produce GPUs at a fraction (say, x=10%) of the price of NVDA, would that increase aggregate demand, or keep it about the same (substitution for nVidia)? Would the price difference be enough to incentivize creation of a CUDA- alternative ecosystem? If not, what does the fraction x need to be? - Very reductively, GPUs seem to be universally needed because they are better at manipulating matrices than alternatives (eg, inverting a matrix, finding dot product, cosine similarity, etc). Are there alternative approaches that could come to market in the next 2-3 years that could be better, or better-per-$, than the current approach of just building bigger GPUs?
- Havoc 2y agoWe’re using gaming GPUs to do this. (Kinda). And the next gen asic like stuff is even more specialised. Really don’t see why he thinks brute force is going away. I mean human brains have billions of neurons too after all.
- xpuente 2y agoProof of that this is wrong, is that our brains require <20W of power. Non-brute force AI will be very efficient. Simple operations (int add/cmp) and no float matrix multiplications and keep moving around the same data all the time from memory to the ALU. GPUs will be overthrown by simple PIM.