4 ms·

Apple might have an advantage given they’ll have custom hardware to drive it, and the ability to combine data from outside the browser with data inside it. But

Apple might have an advantage given they’ll have custom hardware to drive it, and the ability to combine data from outside the browser with data inside it. But it’s an interesting idea.

jchw 2y ago
Apple may have a bit of a lead in getting it actually deployed end-to-end but given the number of times I've heard "AI accelerator" in reference to mobile processors I'm pretty sure that silicon with 'NPUs' are probably all over the place already, and if they're not, they certainly will be, for better or worse. I've got a laptop with a Ryzen 7040, which apparently has XDNA processors in it. I haven't a damn clue how to use them, but there is apparently a driver for it in Linux[1]. It's hard to think of a mobile chipset launch from any vendor that hasn't talked about AI performance in some regards, even the Rockchip ARM processors seem to have "AI engines". This is one of those places where Apple's vertical integration has a clear benefit, but even as a bit of a skeptic regarding "AI" technology, it does seem there's a good chance that accelerated ML inference is going to be one of the next battlegrounds for processor mobile performance and capability, if it hasn't started already. [1]: https://github.com/amd/xdna-driver https://github.com/amd/xdna-driver
- richardw 2y ago
  For sure many devices will have them, but the trick will be to build this local web model in a way that leverages all of the local chips. Apple’s advantage is in not having to worry about all that. It has a simpler problem and better access to real local data. Give my personal local data to a model running in the browser? Just feels a bit more risky.
  - rfoo 2y ago
    > in a way that leverages all of the local chips Which, in a way, is similar to building a browser leveraging all of the local GPUs to do render and HW-accelerated video decoding. Is Safari on Apple Silicon better than Chrome on random Windows laptop for playing YouTube in the last 5 years? Hardly.
    - cuu508 2y ago
      The video plays fine on both, yes, but the Windows laptop, generally speaking, gets hotter, and runs out of battery sooner.
      - ac29 2y ago
        If you compare a random older Windows laptop to a new or nearly new Mac, sure. But new Windows/x86 PCs have gotten pretty efficient. Here's a review showing a Meteor Lake laptop getting 15 hours of video playback on battery: https://www.anandtech.com/show/21282/intel-core-ultra-7-115h-review-meteor-lake-makes-makes-fresh-start-to-mobile-cpus/10 https://www.anandtech.com/show/21282/intel-core-ultra-7-115h...
        cuu508 2y ago
        I'm comparing laptops of similar age and in the same price range.
        smolder 2y ago
        No, you aren't. You compared windows laptops to Apple generally. Apple laptops are expensive.
      - rfoo 2y ago
        Exactly. So feature-wise it can work, and out of our "I proudly spent $2500 on my fancy Apple laptop"-tech bubble, people already learned to settle on something hotter.
        supermatt 2y ago
        What $2500 laptops that people buy for watching YouTube are you referring to?
    - richardw 2y ago
      And would still need to give Chrome access to your contacts, files etc to make it equivalent. It’s not useless, obviously it’ll be good, it’s just not the same. My original comment was replying to: “enable approximately what Apple is doing with their AI strategy”
  - jitl 2y ago
    I think they’ll package a variety of model formats and download the appropriate one for the user’s system. Apple and Microsoft both offer OS-level APIs that abstract over the specific compute architecture so within the platform you don’t need further specialization. On Apple hardware they’ll run their model via CoreML on whichever set of compute (CPU, GPU, NPU) makes sense for the task. On Windows it will use DirectML for the same. I’m not sure if there’s a similar OS abstraction layer on Linux, possibly there they’ll just use CPU inference or whatever stack they usually use on Android.

Apple might have an advantage given they’ll have custom hardware to drive it, and the ability to combine data from outside the browser with data inside it. But it’s an interesting idea.

jchw 2y ago

Apple may have a bit of a lead in getting it actually deployed end-to-end but given the number of times I've heard "AI accelerator" in reference to mobile processors I'm pretty sure that silicon with 'NPUs' are probably all over the place already, and if they're not, they certainly will be, for better or worse. I've got a laptop with a Ryzen 7040, which apparently has XDNA processors in it. I haven't a damn clue how to use them, but there is apparently a driver for it in Linux[1]. It's hard to think of a mobile chipset launch from any vendor that hasn't talked about AI performance in some regards, even the Rockchip ARM processors seem to have "AI engines". This is one of those places where Apple's vertical integration has a clear benefit, but even as a bit of a skeptic regarding "AI" technology, it does seem there's a good chance that accelerated ML inference is going to be one of the next battlegrounds for processor mobile performance and capability, if it hasn't started already. [1]: https://github.com/amd/xdna-driver https://github.com/amd/xdna-driver

richardw 2y ago
For sure many devices will have them, but the trick will be to build this local web model in a way that leverages all of the local chips. Apple’s advantage is in not having to worry about all that. It has a simpler problem and better access to real local data. Give my personal local data to a model running in the browser? Just feels a bit more risky.
- rfoo 2y ago
  > in a way that leverages all of the local chips Which, in a way, is similar to building a browser leveraging all of the local GPUs to do render and HW-accelerated video decoding. Is Safari on Apple Silicon better than Chrome on random Windows laptop for playing YouTube in the last 5 years? Hardly.
  - cuu508 2y ago
    The video plays fine on both, yes, but the Windows laptop, generally speaking, gets hotter, and runs out of battery sooner.
    - ac29 2y ago
      If you compare a random older Windows laptop to a new or nearly new Mac, sure. But new Windows/x86 PCs have gotten pretty efficient. Here's a review showing a Meteor Lake laptop getting 15 hours of video playback on battery: https://www.anandtech.com/show/21282/intel-core-ultra-7-115h-review-meteor-lake-makes-makes-fresh-start-to-mobile-cpus/10 https://www.anandtech.com/show/21282/intel-core-ultra-7-115h...
      - cuu508 2y ago
        I'm comparing laptops of similar age and in the same price range.
        smolder 2y ago
        No, you aren't. You compared windows laptops to Apple generally. Apple laptops are expensive.
    - rfoo 2y ago
      Exactly. So feature-wise it can work, and out of our "I proudly spent $2500 on my fancy Apple laptop"-tech bubble, people already learned to settle on something hotter.
      - supermatt 2y ago
        What $2500 laptops that people buy for watching YouTube are you referring to?
  - richardw 2y ago
    And would still need to give Chrome access to your contacts, files etc to make it equivalent. It’s not useless, obviously it’ll be good, it’s just not the same. My original comment was replying to: “enable approximately what Apple is doing with their AI strategy”
- jitl 2y ago
  I think they’ll package a variety of model formats and download the appropriate one for the user’s system. Apple and Microsoft both offer OS-level APIs that abstract over the specific compute architecture so within the platform you don’t need further specialization. On Apple hardware they’ll run their model via CoreML on whichever set of compute (CPU, GPU, NPU) makes sense for the task. On Windows it will use DirectML for the same. I’m not sure if there’s a similar OS abstraction layer on Linux, possibly there they’ll just use CPU inference or whatever stack they usually use on Android.