10 ms·

Nvidia’s AI supremacy is only temporary

172 points by sebg 3y ago

politelemon 3y ago
Thanks for sharing. I don't know if these predictions will pan out or not but it would make me very happy if inference becomes more accessible and does not stay in the (current) divide of haves and have-nots in terms of hardware and paywalls. The possibility of inference on crappy hardware would open up a lot more possibilities, many of which we haven't dreamed of yet.
- imhoguy 3y ago
  Inference on crappy hardware will bring crappy results. However upcoming SoC solutions with ML capabilities will definitely make a difference. E.g. RPi + Sony Aitrios on single board may bring interesting embedded applications: https://www.prnewswire.com/news-releases/raspberry-pi-receives-strategic-investment-from-sony-semiconductor-solutions-corporation-301795015.html https://www.prnewswire.com/news-releases/raspberry-pi-receiv...
orliesaurus 3y ago
I have no idea what inference means but I hope it happens - and perhaps it will happen. That being said - things like Sun (i.e. solaris workstations) or Intel (for desktop or recently in the last 10ish years, servers) had the world under their thumb for 10+ years. Thus Nvidia might have quite a good reign ahead of themselves - even if it will eventually fade, like everyone else.
- tinco 3y ago
  With inference they mean the dominance in purchasing will change from the producers to the consumers of machine learning models. Right now everyone is buying hardware to produce machine learning models (aka training) and at some point the author predicts the market will shift to buying hardware to consume (run inference) machine learning models. I don't think I agree this is a significant shift that is guaranteed to happen. It might happen that we will go over some sort of hump where there's less training happening than there was at the top of the hump, but who knows when that hump will be? It's such a new field and there's so many low hanging fruit improvements to be made. We could train new models for years and have steady significant improvements every time, even if there's no fundamental breakthrough developments on the horizon. And even if there was a cooldown on new training, training is so many orders of magnitude more expensive than inference that the inference demand would have to be extreme in the face of a very unrealistically rate of training for inference to be dominant.
  - pixl97 3y ago
    Yea, if the author is only thinking about text data, then maybe they'd have a point. But the world in which 'intelligence' exists only a tiny bit textual. Visual then audio data represent most of what humans interpret. And who knows what continuous learning will look like. If you believe we are moving towards the more 'star trek' like future of AI where AI observes and interprets the world as humans see it and experience it, a massive amount of compute is still needed for the foreseeable future. If you believe we are capping out on AI capability soon for some time, then you'll see AI as more of part of the "IBM toolkit" offered as an additional compute service and it will more likely 'fit' in our existing computer architectures.
- krallistic 3y ago
  "Inference" - getting the predictions out of the model. While training you need to run: Input -> Model -> Output (Prediction) - Compare with True Output (Label) -> Backpropagation of Loss through the Model. Which can highly batched & pipelined. (And you have to batch to train in any reasonable amount of times, and GPUs shine in batch regime) When a single user request comes in, you just want the prediction of that single input, so no backprogation and no batching. Which is more CPU friendly.
  - syockit 3y ago
    Wow, now I learned something new. So even though statistics and machine learning overlap each other a lot, a word as simple as inference have totally different meanings. In statistics, it usually refers to determining the influence of an input, for a multi-input model. Getting predictions is simply called prediction.
IllustriousCrow 3y ago
[flagged]
YetAnotherNick 3y ago
How did author just assumed that CPU are competetive for inference. Maybe yes if you just want to run 7 billion parameters model with batch size of 1, but with batching(including continuos batching of vllm), GPU have 2 order higher throughput. And even assuming moore's law is well alive, it will take decade to reach current GPU throughput. There is no way companies will shift to CPU for inference.
- melling 3y ago
  Training can take hours, days, weeks, or months. Doesn’t the inference part only require seconds? Since it requires a fraction of the computation , can’t CPU’s be optimized for that? A few matrix multiplications
  - rerx 3y ago
    GPUs will be better optimized for large matrix multiplications than CPUs, by design. And you need to the inference again and again, not just a single time (like your training).
    - dist-epoch 3y ago
      Desktop CPUs have integrated GPUs, so it's more complicated. If I infer on the GPU inside my CPU, how do you count that?
      - rerx 3y ago
        True. Memory bandwidth may be the most limiting factor.
  - dahart 3y ago
    Inference is only seconds on a GPU, but have a look at flops of modern GPUs to CPUs - matrix multiplications differ by two orders of magnitude. Seconds on the GPU is minutes on the CPU. And don’t forget inference needs to scale in the data center, it needs to run repeatedly for many users.
  - 3y ago
KaoruAoiShiho 3y ago
Err everything sounds right except for #2, wtf did he just say about CPUs LOL?
bryanlarsen 3y ago
While it's highly likely that in the future Nvidia's market share will be lower than it is now since there's basically only one direction share can go when it is currently ~99% (WAG). However it seems to me that the market will be larger in the future than it is now. IOW a smaller share of a larger pie. Not necessarily bad for Nvidia.
rerx 3y ago
I don't think the article fully applies to large language models (LLMs). > Inference will Dominate, not Training This rings true. While LLMs will be fine-tuned by many, fewer companies will train their own independent foundation models from scratch (which doesn't require a "few GPUs", but hundreds with tight interconnect). The inference cost of running these in applications will dominate in these companies. > CPUs are Competitive for Inference I disagree for LLMs. Running the inference still takes a lot of the type of compute that GPUs are optimized for. If you want to respond to your customers' requests with acceptable latency (and achieve some throughput), you will want to use GPUs. For "medium-sized" LLMs you won't need NVLink-level interconnect speeds between your GPUs, though.
pixl97 3y ago
Much like Nvidia's actual GPU supremacy is only temporary... it's just lasted a very long time and doesn't show any signs of stopping. I personally think we're on a very long path AI improvement. At least to me the idea we're going to train an AI and stay on it's core for any significant amount of time doesn't seem likely to me. Continuous learning and feedback improvements are just one avenue we will take. Others will be expanding into multimodal models that self learn based on comparing what they generate with multiple forms of senses.
runako 3y ago
If anything, this post suggests Nvidia has a long supremacy ahead. In particular, the author lays out what is likely to be a durable network in favor of Nvidia: - best in breed software - industry standard used and preferred by most practitioners - better (faster) hardware Notably, this is a similar combination to that which led Wintel to be a durable duopoly for decades, with the only likely end the mass migration to other modes of compute. Regarding the "what will change" category, 2 of the bullet points essentially argue that the personnel he cites as being part of the lock-in will decide to no longer bias for Nvidia, primarily for cost reasons. A third point similarly leans on cost reasons. Nowhere in the analysis does the author account for the historical fact that typically the market leader is best positioned to also be the low-cost leader if strategically desired. It is unlikely that a public company like Intel or AMD or (soon) Arm would enter the market explicitly to race to zero margins. (See also: the smartphone market.) Nvidia also could follow the old Intel strategy and sell its high-end tech for training and its older (previously) high-end tech for inference, allowing customers to use a unified stack across training & inference at different price points. Training customers pay for R&D & profit margin; lower-price inference customers provide a strategic moat.
- dsizzle 3y ago
  One of his points is that NVIDIA is unlikely to maintain its current high margins, and becoming a low-cost leader would lead to lower margins, so that part is consistent.
- tgma 3y ago
  This. Plus, in the high end AI world you are going to need to build a big machine not just a single chip on a PCIe card. They basically have a monopoly on high end RDMA fabric via Mellanox.
- sottol 3y ago
  Research doesn't really have the sunk-cost that industry does. New students are willing to try new things and supervisors don't necessarily need to reign them in. I wonder what is holding AMD back in research? Their cards seem much less costly. I would have figured a nifty research student would figure out quickly how to port torch and run twice as many gpus with his small budget to eek out a bit more performance.
airocker 3y ago
Inference will not work on cpu, I see 100 percent cpu frequently when doing cpu inference and we had to change to use GPUs for a large client.
breadwinner 3y ago
There are a lot of Nvidia chips being bought because of the hype. Saudi Arabia and UAE have decided to become AI powerhouses and the way to do that, of course, is to buy lots of Nvidia chips [1]. So has the UK government, and they are buying $130 million worth of chips [2]. There will be lots of disappointment, and the hype will die down. [1] https://www.ft.com/content/c93d2a76-16f3-4585-af61-86667c5090ba https://www.ft.com/content/c93d2a76-16f3-4585-af61-86667c509... [2] https://www.theguardian.com/business/2023/aug/20/uk-global-race-produce-ai-chips https://www.theguardian.com/business/2023/aug/20/uk-global-r...
- brucethemoose2 3y ago
  The Saudis also bought a Cerebras datacenter.
- rmbyrro 3y ago
  Why do you think it's a hype and why it'll die down? I'm now a heavy user of AI personally & professionally (as a dev). The two work projects I'm involved with are increasing by a lot the usage of GPU to apply LLM tech. I don't see this coming back. The market growth rate will slow down, but it'll continue to grow (and not come back) for quite a few years, I think. When it starts to slow (growth rate, not market), I guess there'll be other breakthroughs in AI like GPT that'll renew the trend.
  - paulddraper 3y ago
    Hype. In 80% of cases people overestimate the usefulness of AI. Kinda like the dotcom bubble. Over the long long term it will be enormous, but they're many growing pains until then
    - josephg 3y ago
      > In 80% of cases people overestimate the usefulness of AI. Looking at my chatgpt history, my partner and I seem to average about 3 conversations a day. We would use it a lot more than that if we had a way to invoke it with our voices, like Siri. Our usage is increasing over time as we figure out the sort of questions it’s good at answering. I’m not saying all the hype is justified, but if anything I think people underestimate how useful AI can already be in their lives. It just takes some learning to figure out how and when to use it. This is markedly different from both web3 and VR. It’s 2023 and I still make purchases with my Visa card and play most video games with mouse and keyboard (while my quest - cool as it is - gathers dust).
SuchAnonMuchWow 3y ago
What the OP is missing in its "today" analysis is that that cloud platform are choosing nvidia right now since its the most mature compute platform, and so the software for using GPU on the cloud will be written more and more in cuda / using nvidia libs: it will become a defacto standard, and nvidia will entrench themselves that way.
- crop_rotation 3y ago
  There is not much general purpose low level GPU software that is in demand. Most GPU software is using few specific high level ML libraries
waffletower 3y ago
I don't see CPUs being competitive for low-latency inference in the web accessible SaaS ('software as a service') space. They certainly can be attractive for specialized backend applications where batch (in the macro-scheduling sense) processing can be utilzed. The author also neglects the attention that other GPU makers are investing in improving their software stacks, particularly AMD, to compete directly with Nvidia.
MPSimmons 3y ago
We _must_ build techniques to continue training existing models, and we have to figure out how to do it in a relatively ubiquitous way. The underlying data that a model is trained on becomes obsolete relatively quickly - I am constantly running into problems with GPT-4 while trying to solve technical problems, because it's cutoff was 2021 and a ton of code has changed since then, rendering much of the knowledgebase useless. The "paste in the current docs as context" trick only scales so far. This is doubly so for large corporations who will be using "inference" on internal datasets. Training can't be a one-time thing. New documents and state must constantly be added to the weights in order for this technology to be useful in the long run. We need to figure out a way to do this that doesn't constantly make models forget about old training or dramatically overweight recent knowledge.
crop_rotation 3y ago
The problem is Nvidia has no real moat. Being better technically and having better software is not enough long term for ridiculous profits, unless you have an extreme lock in. And in Nvidia's case, it is not even as if there is a wide variety of external software that has to run on any alternative. Most companies just need to run <5 ML frameworks and moving that inference to something cheaper doesn't sound too hard, and in any case puts a ceiling on what Nvidia can charge. Training will be harder, but at some level of expenditure threshold crossing, there will be enough money pumped into it by atleast the big clouds to put a ceiling on Nvidia margins there too. The comparisons with Nvidia's long term GPU dominance are misguided. GPUs were not making anywhere near the amount of money to put an extreme pressure from all sides. When you are on track to make MSFT level money without MSFT level moat, expect pressure from all sides trying to take any available slice.
- deleted 3y ago
  [deleted]
- pixl97 3y ago
  Nvidia has just as much of moat as Intel does on processors. Yes, Intels dominance has subsided some recently, but even then the 'stickiness' of businesses and datacenters to stay with Intel is pretty extreme. Nvidia does a lot of work in performance, stability, and libraries that other vendors will have to complete with.
  - crop_rotation 3y ago
    Intel ran general purpose software, that's why it was dominating. Any seamless alternative had to take the vast array of x86 applications and give better perf/price. An Nvidia alternative doesn't have to run all CUDA applications out there to make a dent. It just has to run LLM infernece to make a serious dent in Nvidia earnings.
    - pixl97 3y ago
      "Intel ran general purpose software," Intel did not actually want to do this, and only between a series of lawsuits and licensing deals did it happen. AMD was a competitor to Intel for decades and only has really recently made a dent. At the end of the day delivering products and having a ecosystem developers can use are important. Nvidia's competitors have not delivered on that yet.
propercoil 3y ago
lol have fun building a CUDA competitive. Let us know when you have the drivers.
- ksec 3y ago
  Exactly. It is also hard to grasp why this is needed to be stated upfront in 2023. Do people still believe next year will be the year of Linux Desktop?
dathinab 3y ago
Some additional interesting points: - The recent LLM are so huge that even inference cost is quite expensive. Companies which want to enrich e.g. search with AI but don't need full "chat" capabilities are already looking for alternatives which are cheaper to run even if a bit worse in capabilities (ignoring training cost). - For the same reason specialized hardware for inference has been a thing for quite a while and is currently becoming more mainstream. E.g. google cloud edge TPUs are for mainly inference, so are many phones AI/Neural cores. I also wouldn't be surprised if the main focus for e.g. the recent AI cores in AMD graphic cards would be inference through you can use them for more then that. - Both AMD and Intel might be less behind then it seems when it comes to training and especially inference. E.g. AMD has been selling somewhat successful GPU compute, just not to the general public. With OpenCL being semi abandoned this lead to them having close to no public mind share. Through with ROCm slowly moving to public availability and AI training being more consolidated on the internal architectures it uses this might change very well. Sure for research, especially of unusual AI architectures, Nvidea will probably still win for a long time. But for "daily" LLM training they probably soon will have serious competition, even more so for inference. Similar Intels new dedicated GPU architectures was made with AI training and inference in mind, so at least for inference I'm pretty sure they soon will be competitive, too. - AI training has also become increasingly more professional, with increasingly more often a small number of quite high level frameworks being used. That means that instead of having to make every project work well with your GPU you now can focus on a few high level frameworks. Similar AI architectures widely used differ less extrema then in the past and have less often big changes. Putting both together it means it's today much easier to create hardware+driver which works for well most cases. Which can be good enough to compete. Even with all that said Nvidea has massive mind share which will give them a huge boost and when it comes to bleeding edge/exotic AI research (not just the next generation of LLMs) they probably still will win out hugely. But LLMs is where the current money is, and as far as it seems generational improvements do not come with any massive conceptually architectural changes, but just better composition of the same (by now kinda old) building blocks.
Dritzzka 3y ago
I have a feeling some security researcher is going to come out of the woodwork with a A.I security flaw in Nvidia Drivers or Hardware.
yalogin 3y ago
For Nvidia to lose their supremacy creating a model shouldn’t require large dedicated hardware and instead commodity CPUs should suffice. This is similar to how Facebook and google created the networking stack on commodity hardware. This I can see happening by massively parallelizing training Unfortunately the article doesn’t talk about any innovation in ML at all.
brown 3y ago
“Training costs scale with the number of researchers, inference costs scale with the number of users”. This is interesting, but I think I disagree? I'm most excited about a future where personalized models are continuously training on my own private data.
- Sohcahtoa82 3y ago
  How can you disagree with that statement? Training takes significantly more processing power than inference, and typically only the researchers will be doing the training, so it makes sense that training costs scale with the number of researchers, as each researcher needs access to their own system powerful enough to perform training. Inference costs scaling with the number of users is a no-brainer. I'm pretty dumbfounded how you can just dismiss both statements without giving any reasoning as to why. EDIT: > I'm most excited about a future where personalized models are continuously training on my own private data. This won't be as common as you think.
  - visarga 3y ago
    > typically only the researchers will be doing the training Citizen LLM developers are becoming a thing. Everyone trains (mostly fine-tunes) models today.
    - Sohcahtoa82 3y ago
      Non-technical people will not be fine-tuning models. A service targeted at the masses is unlikely to fine-tune a per-user model. It wouldn't scale without being astronomically expensive.
- mandevil 3y ago
  We will need at least one- if not several- research and data capture breakthroughs to get to that point. One person just doesn't create enough data to effectively train models with our current techniques, no matter what kind of silicon you have. It might be possible, but research and data breakthroughs are much harder to predict than chip and software developer ergonomics improvements. Sometimes the research breakthroughs just never happen.
lukeschlather 3y ago
"Inference costs will dominate" seems very short-sighted. I've kind of laughed at any startup saying "oh we will use AI to do <X>" for years but seeing what LLMs can do suddenly hardware seems like the limiting factor. I can think of endless applications if I had a few GPUs with a petabyte of onboard ram each that also run about 10000x the FLOPs of current GPUs and a few petabytes of storage for training data. I would be training models left and right to tackle whatever problem I thought of. Of course, it's hard to say if such hardware will be available in my lifetime at a price point that I can get for personal use. In the meantime providing hardware for training will still underpin massive businesses. And especially as hardware costs come down I suspect "fine-tuning" will become more and more common and there will even be use cases where running inference on a large number of tokens looks a lot more like fine-tuning, which is to say you're going to want the best GPU you can find and CPUs are just not going to work very well if at all.
- 015a 3y ago
  One thing I think about: over time, more and more training will probably move closer to users devices. - Client-side training carries a lot of financial advantages; you can push the cost of the silicon, storage, and electricity onto the user. - There's privacy benefits, which while not a major driver in adoption is something people think about. - Apple does this already. They're going to keep doing this. When Apple makes a decision, it instantly impacts a massive plurality of the human population in a way that no other company can; and, it tangentially influences other companies. I think you're right that "inference costs will dominate" is a short-sighted take. But: I think the better point is to think about where training will happen. Nvidia is weirdly poorly positioned to have a strong hand in client-side training. They don't have a cost-efficient and electricity-efficient strategy in any of their products; except for Tegra, which has seen zero consumer uptake outside of the Nintendo Switch. There's no hundred-billion-dollar client side AI training strategy anywhere approximate to the RTX 3070 in my gaming Windows PC, that ain't happening. I'm doubtful they can make that pivot; there's a lot of entrenched interest, and legitimately great products, from the existing computer & smartphone manufacturers. Apple has their chips. Google has their chips, and a really strong relationship with Samsung. Microsoft will be an ally, but they have very little power toward convincing their Windows users that a $1400 laptop is better than a $800 one because it has local AI training capability. But, I mean: server-side training is still going to be huge, and Nvidia will still be an extremely successful company. Its just when you consider their percent ownership of the net total of all AI training that will happen in 2030; its going to drop, and the biggest factor behind that drop isn't going to be AMD; its going to be client-side training on chips made by Apple, Google, and others.
choppaface 3y ago
For background, Pete was a founder of jetpac which scraped millions of images from Instagram to use as content in their company which Google bought. [1] This essay’s bold claims about nvidia are like jetpac: something shortsighted, flashy, and designed to make Pete money. Several flags in the essay: “Machine learning is focused on training, not inference” Nope! There are many start-ups that do large-scale inference in the cloud, and have been long before Transformers existed. Some of said companies are customers of e.g. Roboflow and Determined.ai etc. Sure it’s not Google-scale, as Pete has been in Tensorflow land for the past few years. “Researchers have the Purchasing Power.” False! Some can afford a 2-GPU machine, but Pete’s employer and many other large companies have shifted the attention of researchers to problems that require large clusters. It’s almost impossible to publish now (e.g. reproduce results and do something new) without Google’s network (hardware money and people) having a hand in it. The rest of the essay outlines a thesis that an inference-focused product (where Pete invests himself) will disrupt nvidia. Investors take note! Googler is almost done with his handcuffs! There are many risks to nvidia’s moat (they failed to get Arm after all) but this piece is about Pete trying to find investors, not about Nvidia. [1] e.g. https://petewarden.com/2018/05/28/why-you-need-to-improve-your-training-data-and-how-to-do-it/ https://petewarden.com/2018/05/28/why-you-need-to-improve-yo... Among others
JackFr 3y ago
So where are the biggest GPU crypto-miners, and have they turned their GPU's away from crypto to training models? Because when that happens, I can only assume that time-travel is about to be invented. It just makes sense that someone from the future went back in time and created a crypto craze to rival tulip mania, but to ensure that their operation could finance enough GPU's to eventually invent time travel.
mgreg 3y ago
Thanks to the folks at MLCommons we have some benchmarks and data to evaluate and track inference performance published today. Includes results from GPUs, TPUs, and CPUs as well as some power measurements across several ML use cases including LLMs. "This benchmark suite measures how fast systems can process inputs and produce results using a trained model. Below is a short summary of the current benchmarks and metrics. Please see the MLPerf Inference benchmark paper for a detailed description of the motivation and guiding principles behind the benchmark suite." https://mlcommons.org/en/inference-datacenter-31/ https://mlcommons.org/en/inference-datacenter-31/ For example the latest TPU (v5) from Google scores 7.13 queries per second with an LLM. Looking at GCP that server runs $1.2 / hour on demand. On Azure an H100 scores 84.22 queries per second with an LLM. Couldn't find the price for that but an A100 costs $27.197 per hour so no doubt the H100 will be more expensive than that. 7.13 / $1.2 = 5.94 queries/second/$ 84.22 / $27.197 (A100 Pricing) = 3.09 queries/second/$ [edited to include GCP TPU v5 and Nvidia H100 relative performance info for LLM Inference]
seydor 3y ago
"Temporary" is a relative term
gcapu 3y ago
The article has good arguments for Nvidia losing market share, not losing supremacy. The title is mostly clickbait.
yieldcrv 3y ago
Just like Portugal’s supremacy was only temporary for 400 years
- selimthegrim 3y ago
  Who’s the Pope of AI?
rwalle 3y ago
> anyone with experience has their pick of job offers right now I have to say that this is absolutely not true, especially for those with less experience -- new graduates or people with few years of working experience (and without a PhD or many papers)
kkielhofner 3y ago
Article is hit and miss. Mostly correct about inference. Snapchat has been running small ML on mobile devices for years for stuff like face filters, etc. Same for those features on iOS that do facial recognition to match pictures of friends with your contacts. Start to pay attention and you’ll realize your phone is doing things like object detection, classification, face tracking, voice assistant wake word detection, some on device speech and command recognition, and a wild array of other ML tasks. LLMs are sucking all of the oxygen out of the room but the overwhelming majority of end-user use of AI isn’t generative and won’t be for a long time if ever. The article is correct in saying inference, inference, inference. The future of inference is smaller application and use-case specific models deployed to edge. Many of these applications just don’t work with the latency of networks to cloud. Imagine face tracking for a Snapchat filter if it involved streaming video to a cloud for inference. Yeah, not happening. The hosting costs are also astronomical, big inference hardware is only getting harder to get and Nvidia only has so much manufacturing capacity. Leave the H100s up to Meta, OpenAI, etc that are training massive multi-billion parameter LLMs from scratch. Or people renting them in small batches to do finetuning of “smaller” models, etc. This is also getting chipped at - with the unified memory of Apple Silicon you can get the RAM of 2 A/H100s today for less than the cost of used 80GB A100s. With an entire computer (Mac Pro), new and under warranty. Nvidia still wins on TFLOPS but expect M3/M4/whatever to close the gap on this by leaps and bounds. Again, not going up against Meta’s 15k H100s but all anyone else will ever need. Back to the mobile/edge strategy, Xcode includes what is basically ML training and tuning functionality built in. You can literally train an object recognition model by dragging and dropping pictures, encrypt it, and bundle with your app all within Xcode. App developers are doing ML and barely even noticing. This is in latest Xcode, you can bet your bottom dollar Apple will be putting their significant resources to embracing all of this. Train your model on your Mac, bundle it with your app, scale to infinitely for $0 in hosting costs because the model is running on the user’s device. No Nvidia in sight. In terms of architecture you can still offload the big stuff to datacenter but at increasingly receding rates. I personally think the capability and demand for ML/AI will quickly reach a point where Nvidia and clouds just cannot meet demand for the user base and breadth and scope of the functionality they will increasingly expect. ChatGPT has an estimated 100 MAU. Very impressive but Snapchat alone is 1 billion. Capacity, hardware advancements, and the economics of “host everything on big Nvidia” just doesn’t work out. Google has been putting TPU (lite) silicon in Pixel devices since roughly 2021. Apple with neural engine since the iPhone X in 2017… If you’ve been paying attention to these moves from Google and Apple over the last several years you would have seen this coming. They have not been caught flat-footed on this as so many people, press, etc think. Granted there will always be demand for the big datacenter stuff and Nvidia won’t be hurting anytime soon but expect to see demand for Nvidia hardware and cloud GPU usage to drop more and more as this approach eats more and more.
Havoc 3y ago
I’d say this substantially underestimates first move advantage. Early tech dominating the market often has significant staying power. See x86
labrador 3y ago
Training a model and deploying it to consumers to infer from it are two different things. NVidia will remain in demand for training, while deployment will use cheaper hardware on the final model to serve up to users. What am I missing?
cameldrv 3y ago
We will see. I thought this in 2015 when AI for computer vision was starting to heat up and NVIDIA was the clear leader. I was certain AMD would put in the $20 million or so it would take to catch up with CUDA and CuDNN at that time. Based on that analysis, I decided that NVIDIA was overpriced as a stock. Whoops.
AtomicOrbital 3y ago
just wait until Tesla makes its DoJo into a software-as-a-service

172 points by sebg 3y ago

Thanks for sharing. I don't know if these predictions will pan out or not but it would make me very happy if inference becomes more accessible and does not stay in the (current) divide of haves and have-nots in terms of hardware and paywalls. The possibility of inference on crappy hardware would open up a lot more possibilities, many of which we haven't dreamed of yet.

imhoguy 3y ago
Inference on crappy hardware will bring crappy results. However upcoming SoC solutions with ML capabilities will definitely make a difference. E.g. RPi + Sony Aitrios on single board may bring interesting embedded applications: https://www.prnewswire.com/news-releases/raspberry-pi-receives-strategic-investment-from-sony-semiconductor-solutions-corporation-301795015.html https://www.prnewswire.com/news-releases/raspberry-pi-receiv...

orliesaurus 3y ago

I have no idea what inference means but I hope it happens - and perhaps it will happen. That being said - things like Sun (i.e. solaris workstations) or Intel (for desktop or recently in the last 10ish years, servers) had the world under their thumb for 10+ years. Thus Nvidia might have quite a good reign ahead of themselves - even if it will eventually fade, like everyone else.

tinco 3y ago
With inference they mean the dominance in purchasing will change from the producers to the consumers of machine learning models. Right now everyone is buying hardware to produce machine learning models (aka training) and at some point the author predicts the market will shift to buying hardware to consume (run inference) machine learning models. I don't think I agree this is a significant shift that is guaranteed to happen. It might happen that we will go over some sort of hump where there's less training happening than there was at the top of the hump, but who knows when that hump will be? It's such a new field and there's so many low hanging fruit improvements to be made. We could train new models for years and have steady significant improvements every time, even if there's no fundamental breakthrough developments on the horizon. And even if there was a cooldown on new training, training is so many orders of magnitude more expensive than inference that the inference demand would have to be extreme in the face of a very unrealistically rate of training for inference to be dominant.
- pixl97 3y ago
  Yea, if the author is only thinking about text data, then maybe they'd have a point. But the world in which 'intelligence' exists only a tiny bit textual. Visual then audio data represent most of what humans interpret. And who knows what continuous learning will look like. If you believe we are moving towards the more 'star trek' like future of AI where AI observes and interprets the world as humans see it and experience it, a massive amount of compute is still needed for the foreseeable future. If you believe we are capping out on AI capability soon for some time, then you'll see AI as more of part of the "IBM toolkit" offered as an additional compute service and it will more likely 'fit' in our existing computer architectures.
krallistic 3y ago
"Inference" - getting the predictions out of the model. While training you need to run: Input -> Model -> Output (Prediction) - Compare with True Output (Label) -> Backpropagation of Loss through the Model. Which can highly batched & pipelined. (And you have to batch to train in any reasonable amount of times, and GPUs shine in batch regime) When a single user request comes in, you just want the prediction of that single input, so no backprogation and no batching. Which is more CPU friendly.
- syockit 3y ago
  Wow, now I learned something new. So even though statistics and machine learning overlap each other a lot, a word as simple as inference have totally different meanings. In statistics, it usually refers to determining the influence of an input, for a multi-input model. Getting predictions is simply called prediction.

IllustriousCrow 3y ago

[flagged]

YetAnotherNick 3y ago

How did author just assumed that CPU are competetive for inference. Maybe yes if you just want to run 7 billion parameters model with batch size of 1, but with batching(including continuos batching of vllm), GPU have 2 order higher throughput. And even assuming moore's law is well alive, it will take decade to reach current GPU throughput. There is no way companies will shift to CPU for inference.

melling 3y ago
Training can take hours, days, weeks, or months. Doesn’t the inference part only require seconds? Since it requires a fraction of the computation , can’t CPU’s be optimized for that? A few matrix multiplications
- rerx 3y ago
  GPUs will be better optimized for large matrix multiplications than CPUs, by design. And you need to the inference again and again, not just a single time (like your training).
  - dist-epoch 3y ago
    Desktop CPUs have integrated GPUs, so it's more complicated. If I infer on the GPU inside my CPU, how do you count that?
    - rerx 3y ago
      True. Memory bandwidth may be the most limiting factor.
- dahart 3y ago
  Inference is only seconds on a GPU, but have a look at flops of modern GPUs to CPUs - matrix multiplications differ by two orders of magnitude. Seconds on the GPU is minutes on the CPU. And don’t forget inference needs to scale in the data center, it needs to run repeatedly for many users.
- 3y ago

KaoruAoiShiho 3y ago

Err everything sounds right except for #2, wtf did he just say about CPUs LOL?

bryanlarsen 3y ago

While it's highly likely that in the future Nvidia's market share will be lower than it is now since there's basically only one direction share can go when it is currently ~99% (WAG). However it seems to me that the market will be larger in the future than it is now. IOW a smaller share of a larger pie. Not necessarily bad for Nvidia.

rerx 3y ago

I don't think the article fully applies to large language models (LLMs). > Inference will Dominate, not Training This rings true. While LLMs will be fine-tuned by many, fewer companies will train their own independent foundation models from scratch (which doesn't require a "few GPUs", but hundreds with tight interconnect). The inference cost of running these in applications will dominate in these companies. > CPUs are Competitive for Inference I disagree for LLMs. Running the inference still takes a lot of the type of compute that GPUs are optimized for. If you want to respond to your customers' requests with acceptable latency (and achieve some throughput), you will want to use GPUs. For "medium-sized" LLMs you won't need NVLink-level interconnect speeds between your GPUs, though.

pixl97 3y ago

Much like Nvidia's actual GPU supremacy is only temporary... it's just lasted a very long time and doesn't show any signs of stopping. I personally think we're on a very long path AI improvement. At least to me the idea we're going to train an AI and stay on it's core for any significant amount of time doesn't seem likely to me. Continuous learning and feedback improvements are just one avenue we will take. Others will be expanding into multimodal models that self learn based on comparing what they generate with multiple forms of senses.

runako 3y ago

If anything, this post suggests Nvidia has a long supremacy ahead. In particular, the author lays out what is likely to be a durable network in favor of Nvidia: - best in breed software - industry standard used and preferred by most practitioners - better (faster) hardware Notably, this is a similar combination to that which led Wintel to be a durable duopoly for decades, with the only likely end the mass migration to other modes of compute. Regarding the "what will change" category, 2 of the bullet points essentially argue that the personnel he cites as being part of the lock-in will decide to no longer bias for Nvidia, primarily for cost reasons. A third point similarly leans on cost reasons. Nowhere in the analysis does the author account for the historical fact that typically the market leader is best positioned to also be the low-cost leader if strategically desired. It is unlikely that a public company like Intel or AMD or (soon) Arm would enter the market explicitly to race to zero margins. (See also: the smartphone market.) Nvidia also could follow the old Intel strategy and sell its high-end tech for training and its older (previously) high-end tech for inference, allowing customers to use a unified stack across training & inference at different price points. Training customers pay for R&D & profit margin; lower-price inference customers provide a strategic moat.

dsizzle 3y ago
One of his points is that NVIDIA is unlikely to maintain its current high margins, and becoming a low-cost leader would lead to lower margins, so that part is consistent.
tgma 3y ago
This. Plus, in the high end AI world you are going to need to build a big machine not just a single chip on a PCIe card. They basically have a monopoly on high end RDMA fabric via Mellanox.
sottol 3y ago
Research doesn't really have the sunk-cost that industry does. New students are willing to try new things and supervisors don't necessarily need to reign them in. I wonder what is holding AMD back in research? Their cards seem much less costly. I would have figured a nifty research student would figure out quickly how to port torch and run twice as many gpus with his small budget to eek out a bit more performance.

airocker 3y ago

Inference will not work on cpu, I see 100 percent cpu frequently when doing cpu inference and we had to change to use GPUs for a large client.

breadwinner 3y ago

There are a lot of Nvidia chips being bought because of the hype. Saudi Arabia and UAE have decided to become AI powerhouses and the way to do that, of course, is to buy lots of Nvidia chips [1]. So has the UK government, and they are buying $130 million worth of chips [2]. There will be lots of disappointment, and the hype will die down. [1] https://www.ft.com/content/c93d2a76-16f3-4585-af61-86667c5090ba https://www.ft.com/content/c93d2a76-16f3-4585-af61-86667c509... [2] https://www.theguardian.com/business/2023/aug/20/uk-global-race-produce-ai-chips https://www.theguardian.com/business/2023/aug/20/uk-global-r...

brucethemoose2 3y ago
The Saudis also bought a Cerebras datacenter.
rmbyrro 3y ago
Why do you think it's a hype and why it'll die down? I'm now a heavy user of AI personally & professionally (as a dev). The two work projects I'm involved with are increasing by a lot the usage of GPU to apply LLM tech. I don't see this coming back. The market growth rate will slow down, but it'll continue to grow (and not come back) for quite a few years, I think. When it starts to slow (growth rate, not market), I guess there'll be other breakthroughs in AI like GPT that'll renew the trend.
- paulddraper 3y ago
  Hype. In 80% of cases people overestimate the usefulness of AI. Kinda like the dotcom bubble. Over the long long term it will be enormous, but they're many growing pains until then
  - josephg 3y ago
    > In 80% of cases people overestimate the usefulness of AI. Looking at my chatgpt history, my partner and I seem to average about 3 conversations a day. We would use it a lot more than that if we had a way to invoke it with our voices, like Siri. Our usage is increasing over time as we figure out the sort of questions it’s good at answering. I’m not saying all the hype is justified, but if anything I think people underestimate how useful AI can already be in their lives. It just takes some learning to figure out how and when to use it. This is markedly different from both web3 and VR. It’s 2023 and I still make purchases with my Visa card and play most video games with mouse and keyboard (while my quest - cool as it is - gathers dust).

SuchAnonMuchWow 3y ago

What the OP is missing in its "today" analysis is that that cloud platform are choosing nvidia right now since its the most mature compute platform, and so the software for using GPU on the cloud will be written more and more in cuda / using nvidia libs: it will become a defacto standard, and nvidia will entrench themselves that way.

crop_rotation 3y ago
There is not much general purpose low level GPU software that is in demand. Most GPU software is using few specific high level ML libraries

waffletower 3y ago

I don't see CPUs being competitive for low-latency inference in the web accessible SaaS ('software as a service') space. They certainly can be attractive for specialized backend applications where batch (in the macro-scheduling sense) processing can be utilzed. The author also neglects the attention that other GPU makers are investing in improving their software stacks, particularly AMD, to compete directly with Nvidia.

MPSimmons 3y ago

We _must_ build techniques to continue training existing models, and we have to figure out how to do it in a relatively ubiquitous way. The underlying data that a model is trained on becomes obsolete relatively quickly - I am constantly running into problems with GPT-4 while trying to solve technical problems, because it's cutoff was 2021 and a ton of code has changed since then, rendering much of the knowledgebase useless. The "paste in the current docs as context" trick only scales so far. This is doubly so for large corporations who will be using "inference" on internal datasets. Training can't be a one-time thing. New documents and state must constantly be added to the weights in order for this technology to be useful in the long run. We need to figure out a way to do this that doesn't constantly make models forget about old training or dramatically overweight recent knowledge.

crop_rotation 3y ago

The problem is Nvidia has no real moat. Being better technically and having better software is not enough long term for ridiculous profits, unless you have an extreme lock in. And in Nvidia's case, it is not even as if there is a wide variety of external software that has to run on any alternative. Most companies just need to run <5 ML frameworks and moving that inference to something cheaper doesn't sound too hard, and in any case puts a ceiling on what Nvidia can charge. Training will be harder, but at some level of expenditure threshold crossing, there will be enough money pumped into it by atleast the big clouds to put a ceiling on Nvidia margins there too. The comparisons with Nvidia's long term GPU dominance are misguided. GPUs were not making anywhere near the amount of money to put an extreme pressure from all sides. When you are on track to make MSFT level money without MSFT level moat, expect pressure from all sides trying to take any available slice.

deleted 3y ago
[deleted]
pixl97 3y ago
Nvidia has just as much of moat as Intel does on processors. Yes, Intels dominance has subsided some recently, but even then the 'stickiness' of businesses and datacenters to stay with Intel is pretty extreme. Nvidia does a lot of work in performance, stability, and libraries that other vendors will have to complete with.
- crop_rotation 3y ago
  Intel ran general purpose software, that's why it was dominating. Any seamless alternative had to take the vast array of x86 applications and give better perf/price. An Nvidia alternative doesn't have to run all CUDA applications out there to make a dent. It just has to run LLM infernece to make a serious dent in Nvidia earnings.
  - pixl97 3y ago
    "Intel ran general purpose software," Intel did not actually want to do this, and only between a series of lawsuits and licensing deals did it happen. AMD was a competitor to Intel for decades and only has really recently made a dent. At the end of the day delivering products and having a ecosystem developers can use are important. Nvidia's competitors have not delivered on that yet.

propercoil 3y ago

lol have fun building a CUDA competitive. Let us know when you have the drivers.

ksec 3y ago
Exactly. It is also hard to grasp why this is needed to be stated upfront in 2023. Do people still believe next year will be the year of Linux Desktop?

dathinab 3y ago

Some additional interesting points: - The recent LLM are so huge that even inference cost is quite expensive. Companies which want to enrich e.g. search with AI but don't need full "chat" capabilities are already looking for alternatives which are cheaper to run even if a bit worse in capabilities (ignoring training cost). - For the same reason specialized hardware for inference has been a thing for quite a while and is currently becoming more mainstream. E.g. google cloud edge TPUs are for mainly inference, so are many phones AI/Neural cores. I also wouldn't be surprised if the main focus for e.g. the recent AI cores in AMD graphic cards would be inference through you can use them for more then that. - Both AMD and Intel might be less behind then it seems when it comes to training and especially inference. E.g. AMD has been selling somewhat successful GPU compute, just not to the general public. With OpenCL being semi abandoned this lead to them having close to no public mind share. Through with ROCm slowly moving to public availability and AI training being more consolidated on the internal architectures it uses this might change very well. Sure for research, especially of unusual AI architectures, Nvidea will probably still win for a long time. But for "daily" LLM training they probably soon will have serious competition, even more so for inference. Similar Intels new dedicated GPU architectures was made with AI training and inference in mind, so at least for inference I'm pretty sure they soon will be competitive, too. - AI training has also become increasingly more professional, with increasingly more often a small number of quite high level frameworks being used. That means that instead of having to make every project work well with your GPU you now can focus on a few high level frameworks. Similar AI architectures widely used differ less extrema then in the past and have less often big changes. Putting both together it means it's today much easier to create hardware+driver which works for well most cases. Which can be good enough to compete. Even with all that said Nvidea has massive mind share which will give them a huge boost and when it comes to bleeding edge/exotic AI research (not just the next generation of LLMs) they probably still will win out hugely. But LLMs is where the current money is, and as far as it seems generational improvements do not come with any massive conceptually architectural changes, but just better composition of the same (by now kinda old) building blocks.

Dritzzka 3y ago

I have a feeling some security researcher is going to come out of the woodwork with a A.I security flaw in Nvidia Drivers or Hardware.

yalogin 3y ago

For Nvidia to lose their supremacy creating a model shouldn’t require large dedicated hardware and instead commodity CPUs should suffice. This is similar to how Facebook and google created the networking stack on commodity hardware. This I can see happening by massively parallelizing training Unfortunately the article doesn’t talk about any innovation in ML at all.

brown 3y ago

“Training costs scale with the number of researchers, inference costs scale with the number of users”. This is interesting, but I think I disagree? I'm most excited about a future where personalized models are continuously training on my own private data.

Sohcahtoa82 3y ago
How can you disagree with that statement? Training takes significantly more processing power than inference, and typically only the researchers will be doing the training, so it makes sense that training costs scale with the number of researchers, as each researcher needs access to their own system powerful enough to perform training. Inference costs scaling with the number of users is a no-brainer. I'm pretty dumbfounded how you can just dismiss both statements without giving any reasoning as to why. EDIT: > I'm most excited about a future where personalized models are continuously training on my own private data. This won't be as common as you think.
- visarga 3y ago
  > typically only the researchers will be doing the training Citizen LLM developers are becoming a thing. Everyone trains (mostly fine-tunes) models today.
  - Sohcahtoa82 3y ago
    Non-technical people will not be fine-tuning models. A service targeted at the masses is unlikely to fine-tune a per-user model. It wouldn't scale without being astronomically expensive.
mandevil 3y ago
We will need at least one- if not several- research and data capture breakthroughs to get to that point. One person just doesn't create enough data to effectively train models with our current techniques, no matter what kind of silicon you have. It might be possible, but research and data breakthroughs are much harder to predict than chip and software developer ergonomics improvements. Sometimes the research breakthroughs just never happen.

lukeschlather 3y ago

"Inference costs will dominate" seems very short-sighted. I've kind of laughed at any startup saying "oh we will use AI to do <X>" for years but seeing what LLMs can do suddenly hardware seems like the limiting factor. I can think of endless applications if I had a few GPUs with a petabyte of onboard ram each that also run about 10000x the FLOPs of current GPUs and a few petabytes of storage for training data. I would be training models left and right to tackle whatever problem I thought of. Of course, it's hard to say if such hardware will be available in my lifetime at a price point that I can get for personal use. In the meantime providing hardware for training will still underpin massive businesses. And especially as hardware costs come down I suspect "fine-tuning" will become more and more common and there will even be use cases where running inference on a large number of tokens looks a lot more like fine-tuning, which is to say you're going to want the best GPU you can find and CPUs are just not going to work very well if at all.

015a 3y ago
One thing I think about: over time, more and more training will probably move closer to users devices. - Client-side training carries a lot of financial advantages; you can push the cost of the silicon, storage, and electricity onto the user. - There's privacy benefits, which while not a major driver in adoption is something people think about. - Apple does this already. They're going to keep doing this. When Apple makes a decision, it instantly impacts a massive plurality of the human population in a way that no other company can; and, it tangentially influences other companies. I think you're right that "inference costs will dominate" is a short-sighted take. But: I think the better point is to think about where training will happen. Nvidia is weirdly poorly positioned to have a strong hand in client-side training. They don't have a cost-efficient and electricity-efficient strategy in any of their products; except for Tegra, which has seen zero consumer uptake outside of the Nintendo Switch. There's no hundred-billion-dollar client side AI training strategy anywhere approximate to the RTX 3070 in my gaming Windows PC, that ain't happening. I'm doubtful they can make that pivot; there's a lot of entrenched interest, and legitimately great products, from the existing computer & smartphone manufacturers. Apple has their chips. Google has their chips, and a really strong relationship with Samsung. Microsoft will be an ally, but they have very little power toward convincing their Windows users that a $1400 laptop is better than a $800 one because it has local AI training capability. But, I mean: server-side training is still going to be huge, and Nvidia will still be an extremely successful company. Its just when you consider their percent ownership of the net total of all AI training that will happen in 2030; its going to drop, and the biggest factor behind that drop isn't going to be AMD; its going to be client-side training on chips made by Apple, Google, and others.

choppaface 3y ago

For background, Pete was a founder of jetpac which scraped millions of images from Instagram to use as content in their company which Google bought. [1] This essay’s bold claims about nvidia are like jetpac: something shortsighted, flashy, and designed to make Pete money. Several flags in the essay: “Machine learning is focused on training, not inference” Nope! There are many start-ups that do large-scale inference in the cloud, and have been long before Transformers existed. Some of said companies are customers of e.g. Roboflow and Determined.ai etc. Sure it’s not Google-scale, as Pete has been in Tensorflow land for the past few years. “Researchers have the Purchasing Power.” False! Some can afford a 2-GPU machine, but Pete’s employer and many other large companies have shifted the attention of researchers to problems that require large clusters. It’s almost impossible to publish now (e.g. reproduce results and do something new) without Google’s network (hardware money and people) having a hand in it. The rest of the essay outlines a thesis that an inference-focused product (where Pete invests himself) will disrupt nvidia. Investors take note! Googler is almost done with his handcuffs! There are many risks to nvidia’s moat (they failed to get Arm after all) but this piece is about Pete trying to find investors, not about Nvidia. [1] e.g. https://petewarden.com/2018/05/28/why-you-need-to-improve-your-training-data-and-how-to-do-it/ https://petewarden.com/2018/05/28/why-you-need-to-improve-yo... Among others

JackFr 3y ago

So where are the biggest GPU crypto-miners, and have they turned their GPU's away from crypto to training models? Because when that happens, I can only assume that time-travel is about to be invented. It just makes sense that someone from the future went back in time and created a crypto craze to rival tulip mania, but to ensure that their operation could finance enough GPU's to eventually invent time travel.

mgreg 3y ago

Thanks to the folks at MLCommons we have some benchmarks and data to evaluate and track inference performance published today. Includes results from GPUs, TPUs, and CPUs as well as some power measurements across several ML use cases including LLMs. "This benchmark suite measures how fast systems can process inputs and produce results using a trained model. Below is a short summary of the current benchmarks and metrics. Please see the MLPerf Inference benchmark paper for a detailed description of the motivation and guiding principles behind the benchmark suite." https://mlcommons.org/en/inference-datacenter-31/ https://mlcommons.org/en/inference-datacenter-31/ For example the latest TPU (v5) from Google scores 7.13 queries per second with an LLM. Looking at GCP that server runs $1.2 / hour on demand. On Azure an H100 scores 84.22 queries per second with an LLM. Couldn't find the price for that but an A100 costs $27.197 per hour so no doubt the H100 will be more expensive than that. 7.13 / $1.2 = 5.94 queries/second/$ 84.22 / $27.197 (A100 Pricing) = 3.09 queries/second/$ [edited to include GCP TPU v5 and Nvidia H100 relative performance info for LLM Inference]

seydor 3y ago

"Temporary" is a relative term

gcapu 3y ago

The article has good arguments for Nvidia losing market share, not losing supremacy. The title is mostly clickbait.

yieldcrv 3y ago

Just like Portugal’s supremacy was only temporary for 400 years

selimthegrim 3y ago
Who’s the Pope of AI?

rwalle 3y ago

> anyone with experience has their pick of job offers right now I have to say that this is absolutely not true, especially for those with less experience -- new graduates or people with few years of working experience (and without a PhD or many papers)

kkielhofner 3y ago

Article is hit and miss. Mostly correct about inference. Snapchat has been running small ML on mobile devices for years for stuff like face filters, etc. Same for those features on iOS that do facial recognition to match pictures of friends with your contacts. Start to pay attention and you’ll realize your phone is doing things like object detection, classification, face tracking, voice assistant wake word detection, some on device speech and command recognition, and a wild array of other ML tasks. LLMs are sucking all of the oxygen out of the room but the overwhelming majority of end-user use of AI isn’t generative and won’t be for a long time if ever. The article is correct in saying inference, inference, inference. The future of inference is smaller application and use-case specific models deployed to edge. Many of these applications just don’t work with the latency of networks to cloud. Imagine face tracking for a Snapchat filter if it involved streaming video to a cloud for inference. Yeah, not happening. The hosting costs are also astronomical, big inference hardware is only getting harder to get and Nvidia only has so much manufacturing capacity. Leave the H100s up to Meta, OpenAI, etc that are training massive multi-billion parameter LLMs from scratch. Or people renting them in small batches to do finetuning of “smaller” models, etc. This is also getting chipped at - with the unified memory of Apple Silicon you can get the RAM of 2 A/H100s today for less than the cost of used 80GB A100s. With an entire computer (Mac Pro), new and under warranty. Nvidia still wins on TFLOPS but expect M3/M4/whatever to close the gap on this by leaps and bounds. Again, not going up against Meta’s 15k H100s but all anyone else will ever need. Back to the mobile/edge strategy, Xcode includes what is basically ML training and tuning functionality built in. You can literally train an object recognition model by dragging and dropping pictures, encrypt it, and bundle with your app all within Xcode. App developers are doing ML and barely even noticing. This is in latest Xcode, you can bet your bottom dollar Apple will be putting their significant resources to embracing all of this. Train your model on your Mac, bundle it with your app, scale to infinitely for $0 in hosting costs because the model is running on the user’s device. No Nvidia in sight. In terms of architecture you can still offload the big stuff to datacenter but at increasingly receding rates. I personally think the capability and demand for ML/AI will quickly reach a point where Nvidia and clouds just cannot meet demand for the user base and breadth and scope of the functionality they will increasingly expect. ChatGPT has an estimated 100 MAU. Very impressive but Snapchat alone is 1 billion. Capacity, hardware advancements, and the economics of “host everything on big Nvidia” just doesn’t work out. Google has been putting TPU (lite) silicon in Pixel devices since roughly 2021. Apple with neural engine since the iPhone X in 2017… If you’ve been paying attention to these moves from Google and Apple over the last several years you would have seen this coming. They have not been caught flat-footed on this as so many people, press, etc think. Granted there will always be demand for the big datacenter stuff and Nvidia won’t be hurting anytime soon but expect to see demand for Nvidia hardware and cloud GPU usage to drop more and more as this approach eats more and more.

Havoc 3y ago

I’d say this substantially underestimates first move advantage. Early tech dominating the market often has significant staying power. See x86

labrador 3y ago

Training a model and deploying it to consumers to infer from it are two different things. NVidia will remain in demand for training, while deployment will use cheaper hardware on the final model to serve up to users. What am I missing?

cameldrv 3y ago

We will see. I thought this in 2015 when AI for computer vision was starting to heat up and NVIDIA was the clear leader. I was certain AMD would put in the $20 million or so it would take to catch up with CUDA and CuDNN at that time. Based on that analysis, I decided that NVIDIA was overpriced as a stock. Whoops.

AtomicOrbital 3y ago

just wait until Tesla makes its DoJo into a software-as-a-service