4 ms·
The focus on TOPS seems a bit out of line with reality for LLMs. TOPs doesn't matter for LLMs if your memory bandwidth can't keep up. Since it doesn't have quad
by bearjaws 3y ago
The focus on TOPS seems a bit out of line with reality for LLMs. TOPs doesn't matter for LLMs if your memory bandwidth can't keep up. Since it doesn't have quad channel memory mentioned I guess it's still dual channel?
Even top of the line DDR5 is around 128GB/s vs a M1 at 400GB/s.
At the end of the day, it still seems like AI in consumer chips is chasing a buzzword, what is the killer feature?
On mobile there are image processing benefits and voice to text, translation... but on desktop those are no where near common use cases.
- postalrat 3y agohttps://www.neatvideo.com/blog/post/m3 https://www.neatvideo.com/blog/post/m3 That says M1 is 68.25 GB/s
- givinguflac 3y agoThe op were obviously talking about M1 Max.
- postalrat 3y agoHow is it obvious? Anyone reading that could assume that any M1 gets that bandwidth.
- bearjaws 3y agoM1 Max sorry, I don't mean to compare a 4 year old tablet processor to the latest generation of laptop CPUs.
- deleted 3y ago[deleted]
- VHRanger 3y agoThe killer feature is presumably inference at the edge, but I don't see that being used on desktop much at all right now. Especially since most desktop applications people use are web apps. Of the native apps people use that leverage this sort of stuff, almost all are GPU accelerated already (eg. image and video editing AI tools)
- jzig 3y agoWhat does “at the edge” mean here?
- VHRanger 3y agoEdge is doing computing on the client (eg. browser, phone, laptop, etc.) instead of the server
- Dylan16807 3y agoHalf the definitions I see of edge include client devices, and half of them don't include client devices. I like the latter. Why even use a new word if it's just going to be the same as "client"?
- georgeecollins 3y agoNot using AI on the cloud. So if your connection is uncertain or you want use your bandwidth for something else-- like video conferencing or gaming. Probably the killer app is something that wants to use AI but doesn't involve paying a cloud provider. I was talking to a vendor about their chat bot built to put into MMOs or mobile games. It woudl be killer to have a character have life like conversation in those kinds of experiences. But the last thing you want to do is increase your server costs the way this AI would. Edge computing could solve that.
- PeterSmit 3y agoNot in the cloud.
- Zach_the_Lizard 3y agoI'm guessing "the edge" is doing inference work in the browser, etc. as opposed to somewhere in the backend of the web app. Maybe your local machine can run, I don't know, a model to make suggestions as you're editing a Google Doc, which frees up the Big Machine in the Sky to do other things. As this becomes more technically feasible, it reduces the effective cost of inference for a new service provider, since you, the client, are now running their code. The Jevons paradox might kick in, causing more and more uses of LLMs for use cases that were too expensive before.
- futureshock 3y agoUpscaling for gaming or video. Local context aware search Offline Speech to text and TTS Offline generation of clip art or stock images for document editing Offline LLM that can work with your documents as context and access application and OS APIs Improved enemy AI in gaming Webcam effects like background removal or filters. Audio upscaling and interpolation like for bad video call connections.
- bearjaws 3y ago> Upscaling for gaming or video. Already exists on all three major GPU manufacturers, and it definitely makes sense as a GPU workload. > Local context aware search You don't need an AI processor to do this, Windows search used to work better and had even less compute resources to work with. > Offline Speech to text and TTS See my point about not a very common use case for desktops & laptops vs cell phones. > Offline LLM that can work with your documents as context and access application and OS APIs Maybe for some sort of background task or only using really small models <13B parameters. Anything real time is going to run at 1-2t/s with a large model. Small models are pretty terrible though, I doubt people want even more incorrect information and hallucinations. > Improved enemy AI in gaming See Ageia PhysX > Webcam effects like background removal or filters. We already have this without NPUs. > Audio upscaling and interpolation like for bad video call connections. I could see this, or noise cancellation.
- bayindirh 3y agoIt's about power management, and doing more things with less power. These specialized IP blocks on CPUs allow these things to be done with less power and less latency. Intel's bottom of the barrel N95 & N100 CPUs have Gaussian & Neural accelerators for simple image processing and object detection tasks, plus a voice processor for low power voice based activation and command capture and process. You can always add more power hungry, general purpose components to add capabilities. Heck, video post processing entered hardware era with ATI Radeon 8500. But doing these things with negligible power costs is the new front. Apple is not adding coprocessors to their iPhones because it looks nice. All of these coprocessors reduce CPU wake-up cycles tremendously and allows the device to monitor tons of things out of bands with negligible power costs.