4 ms·

As a sibling poster has said, I don't know how much on-device AI is going to matter. I have pretty strong views on privacy, and I've generally thrown them all

by atonse 10mo ago

As a sibling poster has said, I don't know how much on-device AI is going to matter. I have pretty strong views on privacy, and I've generally thrown them all out in light of using AIs, because the value I get out of them is just so huge. If Apple actually had executed on their strategy (of running models in privacy-friendly sandboxes) I feel they would've hit it out of the park. But as it stands, these are all bleeding edge technologies and you have to have your best and brightest on them. And even with seemingly infinite money, Apple doesn't seem to have delivered yet. I hope the "yet" is important here. But judging by the various executives leaving (especially rumors of Johnny Srouji leaving), that's a huge red flag that their problem is that they're bleeding talent, and not a lack of money.

ebbi 10mo ago
Johnny Srouji sent out an email to his team confirming he is staying.
- atonse 10mo ago
  That’s huge. Hope they can continue to keep such people because it isn’t just about one person. It’s all the other smart people that want to work with them.
scrollop 10mo ago
You don't have to abandon privacy when using an eye - use a service that accesses enterprise APIs, which have good privacy policies. I use the service from the guys who create the This day in AI podcast called smithery.ai -we are access to all of the sota models so we can flip between any model including lots of open source ones within one chat or within multiple chats and compared the same query, using various MCPs and lots of other features. If you're interested have a look at the discord to simtheory.ai (I have no connection to the service or to the creators)
twoodfin 10mo ago
I’m much more optimistic on device-side matmul. There’s just so much of it in aggregate and the marginal cost is so low especially since you need to drive fancy graphics to the screen anyway. Somebody will figure out how to use it—complementing Cloud-side matmul, of course—and Apple will be one of the biggest suppliers.
ph4rsikal 10mo ago
On-device moves all compute cost (incl. electricity) to the consumer. I.e., as of 2025 that means much less battery life, a much warmer device, and much higher electricity costs. Unless the M-series can do substantially more with less this is a dead end.
- Marsymars 10mo ago
  Battery isn't relevant to plugged-in devices, and in the end, electricity costs roughly the same to generate and deliver to a data center as to a home. The real cost advantage that cloud has is better amortization of hardware since you can run powerful hardware at 100% 24/7 spread across multiple people. I wouldn't bet on that continuing indefinitely, consumer hardware tends to catch up to HPC-exclusive workloads eventually.
  - fn-mote 10mo ago
    You could have an AppleTV with 48 GB VRAM backing the local requests, but... the trend is "real computers" disappearing from homes, replaced by tablets and phones. The advantage the cloud has is Real Compute Power for the few seconds you need to process the interaction. That's not coming home any time soon.
    - gowld 10mo ago
      My Sun Ray is back in style! $30 on eBay!
    - 827a 10mo ago
      Interestingly, some of Apple’s devices do already serve a special purpose like this in their ecosystem. The HomePod, HomePod Mini, and Apple TV act as Home Hubs for your network, which proxy WAN Apple Home requests to your IoT devices. No other Apple devices can do this. They also already practice a concept of computational offloading with the Apple Watch and iPhone; more complicated fitness calculations, like VO2Max, rely on watch-collected data, but evidence suggests they’re calculated on the phone (new VO2Max algorithms are implemented when you update iOS, not watchOS) So yeah; I can imagine a future where Apple devices could offload substantial AI requests to other devices on your Apple account, to optimize for both power consumption (plugged in versus battery) and speed (if you have a more powerful Mac versus your iPhone). There’s good precedent in the Apple ecosystem for this. Then, of course, the highest tier of requests are processed in their private cloud.

by atonse 10mo ago

ebbi 10mo ago

Johnny Srouji sent out an email to his team confirming he is staying.

atonse 10mo ago
That’s huge. Hope they can continue to keep such people because it isn’t just about one person. It’s all the other smart people that want to work with them.

scrollop 10mo ago

You don't have to abandon privacy when using an eye - use a service that accesses enterprise APIs, which have good privacy policies. I use the service from the guys who create the This day in AI podcast called smithery.ai -we are access to all of the sota models so we can flip between any model including lots of open source ones within one chat or within multiple chats and compared the same query, using various MCPs and lots of other features. If you're interested have a look at the discord to simtheory.ai (I have no connection to the service or to the creators)

twoodfin 10mo ago

I’m much more optimistic on device-side matmul. There’s just so much of it in aggregate and the marginal cost is so low especially since you need to drive fancy graphics to the screen anyway. Somebody will figure out how to use it—complementing Cloud-side matmul, of course—and Apple will be one of the biggest suppliers.

ph4rsikal 10mo ago

On-device moves all compute cost (incl. electricity) to the consumer. I.e., as of 2025 that means much less battery life, a much warmer device, and much higher electricity costs. Unless the M-series can do substantially more with less this is a dead end.

Marsymars 10mo ago
Battery isn't relevant to plugged-in devices, and in the end, electricity costs roughly the same to generate and deliver to a data center as to a home. The real cost advantage that cloud has is better amortization of hardware since you can run powerful hardware at 100% 24/7 spread across multiple people. I wouldn't bet on that continuing indefinitely, consumer hardware tends to catch up to HPC-exclusive workloads eventually.
- fn-mote 10mo ago
  You could have an AppleTV with 48 GB VRAM backing the local requests, but... the trend is "real computers" disappearing from homes, replaced by tablets and phones. The advantage the cloud has is Real Compute Power for the few seconds you need to process the interaction. That's not coming home any time soon.
  - gowld 10mo ago
    My Sun Ray is back in style! $30 on eBay!
  - 827a 10mo ago
    Interestingly, some of Apple’s devices do already serve a special purpose like this in their ecosystem. The HomePod, HomePod Mini, and Apple TV act as Home Hubs for your network, which proxy WAN Apple Home requests to your IoT devices. No other Apple devices can do this. They also already practice a concept of computational offloading with the Apple Watch and iPhone; more complicated fitness calculations, like VO2Max, rely on watch-collected data, but evidence suggests they’re calculated on the phone (new VO2Max algorithms are implemented when you update iOS, not watchOS) So yeah; I can imagine a future where Apple devices could offload substantial AI requests to other devices on your Apple account, to optimize for both power consumption (plugged in versus battery) and speed (if you have a more powerful Mac versus your iPhone). There’s good precedent in the Apple ecosystem for this. Then, of course, the highest tier of requests are processed in their private cloud.