Article is hit and miss. Mostly correct about inference. Snapchat has been running small ML on mobile devices for years for stuff like face filters, etc. Same

Article is hit and miss. Mostly correct about inference. Snapchat has been running small ML on mobile devices for years for stuff like face filters, etc. Same for those features on iOS that do facial recognition to match pictures of friends with your contacts. Start to pay attention and you’ll realize your phone is doing things like object detection, classification, face tracking, voice assistant wake word detection, some on device speech and command recognition, and a wild array of other ML tasks. LLMs are sucking all of the oxygen out of the room but the overwhelming majority of end-user use of AI isn’t generative and won’t be for a long time if ever. The article is correct in saying inference, inference, inference. The future of inference is smaller application and use-case specific models deployed to edge. Many of these applications just don’t work with the latency of networks to cloud. Imagine face tracking for a Snapchat filter if it involved streaming video to a cloud for inference. Yeah, not happening. The hosting costs are also astronomical, big inference hardware is only getting harder to get and Nvidia only has so much manufacturing capacity. Leave the H100s up to Meta, OpenAI, etc that are training massive multi-billion parameter LLMs from scratch. Or people renting them in small batches to do finetuning of “smaller” models, etc. This is also getting chipped at - with the unified memory of Apple Silicon you can get the RAM of 2 A/H100s today for less than the cost of used 80GB A100s. With an entire computer (Mac Pro), new and under warranty. Nvidia still wins on TFLOPS but expect M3/M4/whatever to close the gap on this by leaps and bounds. Again, not going up against Meta’s 15k H100s but all anyone else will ever need. Back to the mobile/edge strategy, Xcode includes what is basically ML training and tuning functionality built in. You can literally train an object recognition model by dragging and dropping pictures, encrypt it, and bundle with your app all within Xcode. App developers are doing ML and barely even noticing. This is in latest Xcode, you can bet your bottom dollar Apple will be putting their significant resources to embracing all of this. Train your model on your Mac, bundle it with your app, scale to infinitely for $0 in hosting costs because the model is running on the user’s device. No Nvidia in sight. In terms of architecture you can still offload the big stuff to datacenter but at increasingly receding rates. I personally think the capability and demand for ML/AI will quickly reach a point where Nvidia and clouds just cannot meet demand for the user base and breadth and scope of the functionality they will increasingly expect. ChatGPT has an estimated 100 MAU. Very impressive but Snapchat alone is 1 billion. Capacity, hardware advancements, and the economics of “host everything on big Nvidia” just doesn’t work out. Google has been putting TPU (lite) silicon in Pixel devices since roughly 2021. Apple with neural engine since the iPhone X in 2017… If you’ve been paying attention to these moves from Google and Apple over the last several years you would have seen this coming. They have not been caught flat-footed on this as so many people, press, etc think. Granted there will always be demand for the big datacenter stuff and Nvidia won’t be hurting anytime soon but expect to see demand for Nvidia hardware and cloud GPU usage to drop more and more as this approach eats more and more.