3 ms·

Looks more and more that Apple is pushing AI inference to the edges, with this and the Mac Mini and it's very suspiciously placed power button which says "No, y

by bentt 2y ago

Looks more and more that Apple is pushing AI inference to the edges, with this and the Mac Mini and it's very suspiciously placed power button which says "No, you don't need to shut this computer off. It has important work to do! (For everyone else)" Will be interesting to see how Apple and Nvidia's approaches to AI compare and contrast over the coming months/years.

deleted 2y ago
[deleted]
heywoods 2y ago
I am hoping Apple at some point in the future will allow iphone owners who also own a Mac with an M series chip to run Apple Intelligence from the Mac to take advantage of faster compute and larger parameter models while still operating at the 'edge'. If the apple private cloud tech allows for a Tailscale like connectivity between your mac and all iphone/ipads on an icloud account I could see that as an additional hook to stay inside the apple ecosystem in addition to the benefits brought to bear by having a much more capable LLM to offload compute to without involving 3rd party AI companies. Perhaps bundle the feature with iCloud+ subscription?
- bentt 2y ago
  Right??! Others are downvoting me for supposedly inventing a conspiracy theory but to me it lines up pretty well with their business and could offer some compelling value to customers. Most people haven't read up on their private cloud stuff. But I really did think that about the Mac Mini power button!
deleted 2y ago
[deleted]
dylan604 2y ago
> It has important work to do! (For everyone else) Where do you get that macOS is running code like this? I've never heard of shared compute on my computer for anything other than software I deliberately installed. I haven't paid that close attention to the past couple of OS propaganda films at the start of WWDC. Did I miss something?
- bentt 2y ago
  I'm not saying they currently are. They wouldn't until they knew that whatever they were doing was utterly secure. I'm just positing that they might be going there.
nobankai 2y ago
Apple and Nvidia have arguably already diverged at the important forks in the road. Nvidia believes in complex GPU/streaming multiprocessor stacks - Apple relies on specialized hardware. Apple used to support a more complex software stack that could enable them to compete with Nvidia, but abandoned it shortly before the crypto craze to focus on NPU hardware. ...and then NPUs sorta did nothing. They run a few tiny models, maybe, but for any "serious" inference tasks Apple will automatically prioritize your 10x more powerful GPU hardware. Oftentimes the GPU is more efficient too, depending on the task. So now Apple has a choice to make. They can either attempt to scale-up the NPU hardware and leave it on-device as dark silicon 99% of the time, or they can renovate their GPU hardware to support complex GPGPU operations and axe the NPU altogether. Right now it seems like Nvidia has the right idea, Apple just needs to find out how to scale it down as well as they can.
- bentt 2y ago
  Their private cloud for AI reads on the surface like the answer. If they can establish privacy then they can push the work to the edges. Again, I'm not saying the are. They're the only ones that could, though.
  - nobankai 2y ago
    If you just read my comment and arrived at the conclusion that private compute is the solution, you misunderstand the issue at hand. Networking compute isn't an issue. Compute architecture is.
    - bentt 2y ago
      Sure so then ask yourself what advantages could an NPU have over a general purpose GPU. Your assumption is that Apple has erred. However if security was top priority, then that could make NPU make sense in the long run. And then ask why security is so important… it becomes vastly more important when your device is operating on other people’s workloads.
crazygringo 2y ago
Are you promoting some kind of conspiracy theory that Apple is using some people's devices to do inference for other people's devices? There's absolutely zero evidence of that, and it would be easy to observe it happening. So why are you pushing some kind of totally false narrative?
- bentt 2y ago
  Maybe there's zero evidence now, but think about the lengths to which Apple has gone to build a private cloud infrastructure and why they would do so. It's one thing for them to offer that as a do-gooder service to their customers, which it certainly is. But once it's established as private and secure, why wouldn't they use spare compute at the edge for inference? It's smart. It's not a conspiracy.
  - crazygringo 2y ago
    > think about the lengths to which Apple has gone to build a private cloud infrastructure None? I don't know what you're referring to. Apple's not running any kind of cloud on my private device. Their Private Cloud Compute is about running inference on Apple's cloud but with privacy guarantees -- not on consumer devices. > It's one thing for them to offer that as a do-gooder service to their customers, which it certainly is. Again, no idea what you're talking about. Who's doing good to who? Apple's providing a service to get you to buy their hardware. Consumers aren't doing good for other consumers. > why wouldn't they use spare compute at the edge for inference? It's smart. You're talking about on people's devices? Because it would use up their battery and heat up their devices and connectivity can be spotty which leads to terrible latency. I mean, that would be as bad as secretly mining Bitcoin on people's devices. How would that ever be smart? And Apple's in the business of selling hardware. Not helping people save money by sharing hardware. You can't even share an iPad with different logins for family members.

by bentt 2y ago

deleted 2y ago

[deleted]

heywoods 2y ago

I am hoping Apple at some point in the future will allow iphone owners who also own a Mac with an M series chip to run Apple Intelligence from the Mac to take advantage of faster compute and larger parameter models while still operating at the 'edge'. If the apple private cloud tech allows for a Tailscale like connectivity between your mac and all iphone/ipads on an icloud account I could see that as an additional hook to stay inside the apple ecosystem in addition to the benefits brought to bear by having a much more capable LLM to offload compute to without involving 3rd party AI companies. Perhaps bundle the feature with iCloud+ subscription?

bentt 2y ago
Right??! Others are downvoting me for supposedly inventing a conspiracy theory but to me it lines up pretty well with their business and could offer some compelling value to customers. Most people haven't read up on their private cloud stuff. But I really did think that about the Mac Mini power button!

deleted 2y ago

[deleted]

dylan604 2y ago

> It has important work to do! (For everyone else) Where do you get that macOS is running code like this? I've never heard of shared compute on my computer for anything other than software I deliberately installed. I haven't paid that close attention to the past couple of OS propaganda films at the start of WWDC. Did I miss something?

bentt 2y ago
I'm not saying they currently are. They wouldn't until they knew that whatever they were doing was utterly secure. I'm just positing that they might be going there.

nobankai 2y ago

Apple and Nvidia have arguably already diverged at the important forks in the road. Nvidia believes in complex GPU/streaming multiprocessor stacks - Apple relies on specialized hardware. Apple used to support a more complex software stack that could enable them to compete with Nvidia, but abandoned it shortly before the crypto craze to focus on NPU hardware. ...and then NPUs sorta did nothing. They run a few tiny models, maybe, but for any "serious" inference tasks Apple will automatically prioritize your 10x more powerful GPU hardware. Oftentimes the GPU is more efficient too, depending on the task. So now Apple has a choice to make. They can either attempt to scale-up the NPU hardware and leave it on-device as dark silicon 99% of the time, or they can renovate their GPU hardware to support complex GPGPU operations and axe the NPU altogether. Right now it seems like Nvidia has the right idea, Apple just needs to find out how to scale it down as well as they can.

bentt 2y ago
Their private cloud for AI reads on the surface like the answer. If they can establish privacy then they can push the work to the edges. Again, I'm not saying the are. They're the only ones that could, though.
- nobankai 2y ago
  If you just read my comment and arrived at the conclusion that private compute is the solution, you misunderstand the issue at hand. Networking compute isn't an issue. Compute architecture is.
  - bentt 2y ago
    Sure so then ask yourself what advantages could an NPU have over a general purpose GPU. Your assumption is that Apple has erred. However if security was top priority, then that could make NPU make sense in the long run. And then ask why security is so important… it becomes vastly more important when your device is operating on other people’s workloads.

crazygringo 2y ago

Are you promoting some kind of conspiracy theory that Apple is using some people's devices to do inference for other people's devices? There's absolutely zero evidence of that, and it would be easy to observe it happening. So why are you pushing some kind of totally false narrative?

bentt 2y ago
Maybe there's zero evidence now, but think about the lengths to which Apple has gone to build a private cloud infrastructure and why they would do so. It's one thing for them to offer that as a do-gooder service to their customers, which it certainly is. But once it's established as private and secure, why wouldn't they use spare compute at the edge for inference? It's smart. It's not a conspiracy.
- crazygringo 2y ago
  > think about the lengths to which Apple has gone to build a private cloud infrastructure None? I don't know what you're referring to. Apple's not running any kind of cloud on my private device. Their Private Cloud Compute is about running inference on Apple's cloud but with privacy guarantees -- not on consumer devices. > It's one thing for them to offer that as a do-gooder service to their customers, which it certainly is. Again, no idea what you're talking about. Who's doing good to who? Apple's providing a service to get you to buy their hardware. Consumers aren't doing good for other consumers. > why wouldn't they use spare compute at the edge for inference? It's smart. You're talking about on people's devices? Because it would use up their battery and heat up their devices and connectivity can be spotty which leads to terrible latency. I mean, that would be as bad as secretly mining Bitcoin on people's devices. How would that ever be smart? And Apple's in the business of selling hardware. Not helping people save money by sharing hardware. You can't even share an iPad with different logins for family members.