> My understanding is that the generation of responses (the inference) is what most of the compute is spent upon (as opposed to the training). no, training tak

> My understanding is that the generation of responses (the inference) is what most of the compute is spent upon (as opposed to the training). no, training takes hundreds/thousands of high-end HPC gpus for days/weeks/months/years. Inference runs in seconds. The argument around "inference is more important/will be a larger market" revolves around the total volume of inference performed being larger than the amount to train. A million consumers perform inference, but you only train it once. The problem is the training really never stops, it's not like you train a model and you're done. Even if there were no new innovations it would be worth continuing to train models for years even just fitting the models we've got to the data better. LORAs and QLORAs to train. Etc. As you adopt it more widely, the amount of things we need to train will continue to increase anyway. There is no particular reason to assume the growth of training needs will ever stop, or even decrease below current growth rates (TOPS, not $). It will certainly not continue at 300% margins forever, of course. But I just don't think the assertion that training needs will decrease is supportable - this seems like a Jevon's Paradox moment, the increased utility and accessibility of this will drive an enormous increase in usage, both training and inference. It's also a misleading statement around revenue. Yes, there will be a lot of inference that's done... and it's trivial to implement and will be in almost every single device etc. Training is really the only thing it's feasible to build an ecosystem/moat around, because inference is going to be done by everyone, and will have almost zero margin for either standalone devices or the IP. Training is the interesting and profitable part of the market, and new discoveries go from a "research -> training/scale-out -> inference" pipeline that tends to favor NVIDIA because they have the ecosystem for the research etc. It has been the flexibility and programmability of the GPGPU model that has won so far - you can certainly get a lot of TOPS on Trainium or whatever, but if they can't train the techniques currently used then it doesn't matter. They have spent an unfathomable sum of money (tens of billions) fostering that ecosystem, sponsoring academics and researchers and conferences and paying for people to write libraries and tuning their software etc. There are millions of man-hours of work that need to be done even if you fully understand exactly the minimal combination of pieces to get there, and that doesn't even begin to replicate the mindshare factors etc. Intel increasingly has a perfectly viable software ecosystem, but that doesn't matter if nobody is using it. NVIDIA and Apple are really the only 2 ecosystems anyone uses, in the sense of being a place where work takes place on the platform for the sake of being used by people on that platform. Everything else is just a port-of-convenience for access to hardware and nobody cares a whit about building a long-term community around SyCL vs ROCm as a thing in itself. That's table stakes to build an ecosystem, not the ecosystem itself. Having a complete, viable software ecosystem was the ask 10 years ago, to be competitive in 2024 you need the people using it because we're no longer in the blue-sky/academic-research stage here. It's like thinking you can be a reddit competitor just because you implemented lemmy - the thing that makes reddit reddit is not the source code, that's table stakes, not a platform. Building a mapping layer over the top might help, but it also removes any incentive for the meta-layer to care about you as anything more than a "utility". Literally the players that are going to succeed are the ones like NVIDIA, Apple, and Sony where there is a first-party culture and an ecosystem that is intellectually self-sustaining as a target in itself (and not a utility/build target for someone else). Maybe microsoft to a lesser extent (they are in a good place given their control of the dominant client OS). The "iphone bubble" never popped - yearly revenue increased more or less continuously for a decade, even during the Great Recession it went up. Even though the smartphone market did see many other viable players emerge, Apple retained a plurality control of the market and remains the single largest and most influential player in design as well as direct market influence. Saying something is a "bubble" is begging the question, maybe NVIDIA just popularized a whole new device/field (GPGPU) in the same way Apple did and the market grows underneath them. People are blindly making the assertions that the revenue MUST be unsustainable because "well I don't see any value!" and then building this whole little chain of logic on some things that are ultimately propositions/assertions and not fait-accompli.