3 ms·
This makes me wonder whether on-device AI will ever have a real chance against cloud based AI except for extremely latency sensitive tasks. If the technology a
by fauigerzigerk 3y ago
This makes me wonder whether on-device AI will ever have a real chance against cloud based AI except for extremely latency sensitive tasks.
If the technology advances at such a pace that almost everyone is sitting on obsolete or far too weak hardware then the economics of on-device AI processing may not work out for app developers.
If it's just a matter of running a little bit slower or a little bit faster then it won't be a problem. But if people miss out on important capabilities altogether unless they spend thousands on hardware at least every other year then AI will be almost exclusively a cloud thing.
- brookst 3y agoI expect to see dynamic, hybrid AI frameworks that let the same application run anywhere, with differing amounts of local compute depending on device capabilities. Its the only way huge AI models can be latency and cost efficient across massively heterogeneous execution environments.
- fauigerzigerk 3y agoI'm not sure if it will pan out that way. It sounds awfully complex. I rather suspect that people will do some lowest common denominator inferencing on-device and shift everything else to the cloud.
- brookst 3y agoMulti-layer caches are also complex, but the benefits are worth it. We already see NPUs spanning 3 orders of magnitude in TOPs in consumer devices. In the next few years that will probably be eight or ten orders of magnitude. I don’t think a LCD approach is going to work. I would say that everything gets pushed to the cloud, but the economics are brutal. All of those companies spending billions a year on GPUs have very strong incentives to push as much inference to the edge as possible.