2 ms·
I expect to see dynamic, hybrid AI frameworks that let the same application run anywhere, with differing amounts of local compute depending on device capabiliti
by brookst 3y ago
I expect to see dynamic, hybrid AI frameworks that let the same application run anywhere, with differing amounts of local compute depending on device capabilities.
Its the only way huge AI models can be latency and cost efficient across massively heterogeneous execution environments.
- fauigerzigerk 3y agoI'm not sure if it will pan out that way. It sounds awfully complex. I rather suspect that people will do some lowest common denominator inferencing on-device and shift everything else to the cloud.
- brookst 3y agoMulti-layer caches are also complex, but the benefits are worth it. We already see NPUs spanning 3 orders of magnitude in TOPs in consumer devices. In the next few years that will probably be eight or ten orders of magnitude. I don’t think a LCD approach is going to work. I would say that everything gets pushed to the cloud, but the economics are brutal. All of those companies spending billions a year on GPUs have very strong incentives to push as much inference to the edge as possible.