3 ms·

There's a bit of a difference between an on-device chip for some AI tasks (like all modern smartphones and some computers have), and high end server-class graph

There's a bit of a difference between an on-device chip for some AI tasks (like all modern smartphones and some computers have), and high end server-class graphics cards/GPUs/accelerators than take hundreds of watts to run for generative AI. State of the art for low end generative AI is still running on some sort of desktop class GPU with optimised models and getting not-so-great results. A phone can't do that. Lots of ML happens on device, more is moving to the device. Generative AI is not ready for that yet from what I can tell (based on external experience, not Google specific).

canucker2016 3y ago
from Feb 2023 blog post, https://www.qualcomm.com/news/onq/2023/02/worlds-first-on-device-demonstration-of-stable-diffusion-on-android https://www.qualcomm.com/news/onq/2023/02/worlds-first-on-de...: "...The result of this full-stack optimization is running Stable Diffusion on a smartphone under 15 seconds for 20 inference steps to generate a 512x512 pixel image — this is the fastest inference on a smartphone and comparable to cloud latency. User text input is completely unconstrained." Hasn't shown up in a shipping cellphone yet AFAIK.
- danpalmer 3y ago
  There's a big difference between tech demos and production software for a general audience. Generative AI is only barely production software at the best of times, so almost any compromises mean it's not shippable. This blog post is interesting, but there are so many caveats to it. It's a 1B parameter model which is tiny. Inference takes 20 seconds, which sucks for UX, but also that will be 20s of sustained extremely high load on the device, which means battery drain that would probably make this unshippable as well. It's also worth noting that the images in the blog post were not generated by this process as far as I can tell, they're just stable diffusion examples. It's good this research is being done of course, but I think we're a few years away from this being shipped in any real form on device.
  - canucker2016 3y ago
    You stated, "State of the art for low end generative AI is still running on some sort of desktop class GPU with optimised models and getting not-so-great results. A phone can't do that." The Qualcomm article states, "Read on to learn how Qualcomm AI Research performed full-stack AI optimizations using the Qualcomm AI Stack to deploy Stable Diffusion on an Android smartphone for the very first time. "
    - danpalmer 3y ago
      Ok I guess it depends what you think of state of the art as being. I was thinking in terms of the best option that is somewhat generally available, like the best thing I could do with tools and techniques available. Qualcomm's research is better, but at an earlier stage. Maybe that is state of the art! Not quite as useful as I was thinking about though.
- hypercube33 3y ago
  The problem is all of these mobiles have custom proprietary AI chips with custom SDKs and are usually not even market share (looking at you Qualcomm Microsoft and Google) so beyond the OEM is it worth any third party to invest time in development for them? Seems like general purpose gpu would be a better pick

canucker2016 3y ago

from Feb 2023 blog post, https://www.qualcomm.com/news/onq/2023/02/worlds-first-on-device-demonstration-of-stable-diffusion-on-android https://www.qualcomm.com/news/onq/2023/02/worlds-first-on-de...: "...The result of this full-stack optimization is running Stable Diffusion on a smartphone under 15 seconds for 20 inference steps to generate a 512x512 pixel image — this is the fastest inference on a smartphone and comparable to cloud latency. User text input is completely unconstrained." Hasn't shown up in a shipping cellphone yet AFAIK.

danpalmer 3y ago
There's a big difference between tech demos and production software for a general audience. Generative AI is only barely production software at the best of times, so almost any compromises mean it's not shippable. This blog post is interesting, but there are so many caveats to it. It's a 1B parameter model which is tiny. Inference takes 20 seconds, which sucks for UX, but also that will be 20s of sustained extremely high load on the device, which means battery drain that would probably make this unshippable as well. It's also worth noting that the images in the blog post were not generated by this process as far as I can tell, they're just stable diffusion examples. It's good this research is being done of course, but I think we're a few years away from this being shipped in any real form on device.
- canucker2016 3y ago
  You stated, "State of the art for low end generative AI is still running on some sort of desktop class GPU with optimised models and getting not-so-great results. A phone can't do that." The Qualcomm article states, "Read on to learn how Qualcomm AI Research performed full-stack AI optimizations using the Qualcomm AI Stack to deploy Stable Diffusion on an Android smartphone for the very first time. "
  - danpalmer 3y ago
    Ok I guess it depends what you think of state of the art as being. I was thinking in terms of the best option that is somewhat generally available, like the best thing I could do with tools and techniques available. Qualcomm's research is better, but at an earlier stage. Maybe that is state of the art! Not quite as useful as I was thinking about though.
hypercube33 3y ago
The problem is all of these mobiles have custom proprietary AI chips with custom SDKs and are usually not even market share (looking at you Qualcomm Microsoft and Google) so beyond the OEM is it worth any third party to invest time in development for them? Seems like general purpose gpu would be a better pick