Aside from the runtime difference between training and inference, having on-device ML makes a lot of sense for other reasons. There can be more guarantees over

Aside from the runtime difference between training and inference, having on-device ML makes a lot of sense for other reasons. There can be more guarantees over data privacy, since your data can stay on-device. It also reduces bandwidth as there's no need to upload data for classification to the cloud. And that also may mean it's faster, potentially real time, since you don't have that round trip latency. This is not necessarily for phones. Lots of (virtually all?) low power IoT devices have ARM cores. There are plenty of environments where the cloud or compute power isn't available.