Part of the narrative from both Google and Apple is that "on-device" is better aligned with preserving user privacy. That's true, and it also has performance ra

Part of the narrative from both Google and Apple is that "on-device" is better aligned with preserving user privacy. That's true, and it also has performance rationale when network latency makes online/cloud-inference non-viable for some use cases. Google and Apple have been promoting on-device inference/hardware accelerated AI/ML processing for several years, and with Google this goes back as early as Pixel 2. However, a big factor motivating these companies to promote "on-device" inference is the operating cost. While the compute and data requirements for training many models exceed the resources of almost everyone but these titans, it is still expected that when these models will be used for inference, and the total costs of inference over time, will far exceed the cost of training.