3 ms·

> Apple also plans to add a much improved Neural Engine that has an increased number of cores for AI tasks. This makes total sense but does it? Does Apple real

> Apple also plans to add a much improved Neural Engine that has an increased number of cores for AI tasks. This makes total sense but does it? Does Apple really have an application roadmap to ultimately utilize these tensor cores? For all the M1, M2, M3s out there, Siri still sucks. We keep seeing latest arXiv papers [1] coming out of their research efforts hinting at potential improvements but there is not much in those papers that needs to wait for an M4 or M5... What gives? [1] https://arxiv.org/abs/2403.20329 https://arxiv.org/abs/2403.20329

apetresc 3y ago
I mean, the rumours at this point are deafening that Siri is getting a complete rehaul (or an outright replacement, which probably amounts to the same thing) at WWDC in just two months, so I'd reserve judgment on Apple's AI deployment capabilities until then.
Rinzler89 3y ago
I think their main AI case will be photo and video editing or maybe some LLM bots to help your organize and search through files, but Siri feels really abandoned.
- knodi123 3y ago
  If they just gave Siri an off-the-shelf LLM with a few extra capabilities that plugged into their ecosystem, it would be amazing.
  - Art9681 3y ago
    They recently published a paper where an on device model has UI awareness. https://arxiv.org/abs/2404.05719 https://arxiv.org/abs/2404.05719 Siri is about to get swole.
    - Rinzler89 3y ago
      Damn, it seems then Google is really behind here. IIRC their Pixel 8 chips are so underpowered that they can't even run Google's smallest LLM models locally on device, so despite their whole AI hype sales pitch, everything on the Pixel 8 is still run in the cloud. How is Google so slow at this? At least the likes of Intel and Microsoft have an excuse, they're old and crusty corporations that move slow.
      - r00fus 3y ago
        It's instructive to know where corporations make their money. Because that's often the only place they innovate. For Google it's web/ads. For Apple it's hardware.
- sroussey 3y ago
  Siri moved on device, which was a huge engineering effort. But capabilities stagnated, like because of the move on device.
- alwillis 3y ago
  > I think their main AI case will be photo and video editing Sure, AI features will come to all of their content creation apps at some point, most of it being processed on-device. But I suspect the most impact will be in other apps and areas of the operating systems that typically haven't gotten a lot of attention traditionally. I'd much rather have generative AI or an intelligent agent to deal with email than filling in backgrounds in images. Because the pertinent information is in my calendar, address book, etc., I should be able to say "email Tom I can't make it to the party" and that email is created for me.
- lamontcg 3y ago
  > Siri feels really abandoned. Apple has a fairly severe case of Apple Maps-related PTSD. They aren't likely to introduce a half-baked product in a rush to market like everyone else is doing. Apple is unlikely to release a fundamental modern-ML update to Siri without it both breaking new ground in features and being considerably more "safe" to use than other products ("safe" in the sense of keeping Apple out of the headlines).
digitcatphd 3y ago
Seems like marketing hype? Surely running local LLMs isn’t that big a market.
- binkHN 3y ago
  > Surely running local LLMs isn’t that big a market. Agreed. That said, if you do AI/LLMs, few, if any, portable non-Mac systems have the ability to pull this off. The Macs just crush it when it comes to GPU and memory bandwidth performance, and they do this while sipping battery life.
- fragmede 3y ago
  Right now there's not, but the market for smartphones was pretty small until the iPhone came out. If anybody could create a market for something, I'd believe Apple could do it.
- lxgr 3y ago
  An LLM able to e.g. summarize the gist of a long text conversation in a group chat that I missed, able to understand queries like "what time did X say we'd meet up, and how long would it take me to get there" etc. would be incredibly useful to me. Sure, most of that is possible server-side – but the appeal would be that local processing could preserve a lot more privacy.
- chaostheory 3y ago
  You underestimate how much people hate the guardrails for the cloud AIs.
  - weikju 3y ago
    It’s not like Apple won’t have guardrails in their local llms…
aiauthoritydev 3y ago
Generally if there are enough people, there will be an ecosystem building software for it.
highwaylights 3y ago
> What gives? You give. More money. These new models aren’t going to sell themselves.
_boffin_ 3y ago
I think that Apple's biggest sell with this stuff is going to be utilizing all the SQLite databases on OSX, IOS, IPadOS along with all the on-device analytics that are constantly running. I have a feeling that we're going to see Journal become more of a center focus over the coming years. Question: Is it illegal for apple to do something on-device that runs queries against all the databases for the user's applications?
- meepmorp 3y ago
  > Is it illegal for apple to do something on-device that runs queries against all the databases for the user's applications? Why would it be illegal?
  - appplication 3y ago
    I assume illegal in the EU antitrust sense where if Apple can access then so should everyone else
LorenDB 3y ago
I have a hunch that Apple hasn't released a folding iPhone simply because they don't want to sell a phone with a crease in the screen. Similarly, they might be sticking with the current Siri design until some LLM flaws can be ironed out. I am not an Apple user though, so I have very little experience with Siri.
alwillis 3y ago
> This makes total sense but does it? Does Apple really have an application roadmap to ultimately utilize these tensor cores? Apple always does things in a holistic way. They wouldn't put these AI cores in their devices if they didn't also have a software strategy to go along with it. Unlike with most companies, AI won't be bolted on; it'll well-integrated into the operating system. For example, they've been working on LLMs[1]: > We demonstrate large improvements over an existing system with similar functionality across different types of references, with our smallest model obtaining absolute gains of over 5% for on-screen references. We also benchmark against GPT-3.5 and GPT-4, with our smallest model achieving performance comparable to that of GPT-4, and our larger models substantially outperforming it. And it's not going to require a terabyte of RAM either. You can read the details: "ReALM: Reference Resolution As Language Modeling" [2] There's also evidence of an AI browsing assistant for Safari. [3] [1]: "Apple AI researchers boast useful on-device model that 'substantially outperforms' GPT-4"--https://9to5mac.com/2024/04/01/apple-ai-gpt-4/ https://9to5mac.com/2024/04/01/apple-ai-gpt-4/ [2]: https://arxiv.org/pdf/2403.20329.pdf https://arxiv.org/pdf/2403.20329.pdf [3]: https://www.macrumors.com/2024/04/10/ios-18-safari-browsing-assistant-rumor/ https://www.macrumors.com/2024/04/10/ios-18-safari-browsing-...
kingkongjaffa 3y ago
Is the lack of LLM siri already because the on-device compute would totally kill battery life? And battery life is a sacred criteria that is unacceptable to regress on for mobile?

apetresc 3y ago

I mean, the rumours at this point are deafening that Siri is getting a complete rehaul (or an outright replacement, which probably amounts to the same thing) at WWDC in just two months, so I'd reserve judgment on Apple's AI deployment capabilities until then.

Rinzler89 3y ago

I think their main AI case will be photo and video editing or maybe some LLM bots to help your organize and search through files, but Siri feels really abandoned.

knodi123 3y ago
If they just gave Siri an off-the-shelf LLM with a few extra capabilities that plugged into their ecosystem, it would be amazing.
- Art9681 3y ago
  They recently published a paper where an on device model has UI awareness. https://arxiv.org/abs/2404.05719 https://arxiv.org/abs/2404.05719 Siri is about to get swole.
  - Rinzler89 3y ago
    Damn, it seems then Google is really behind here. IIRC their Pixel 8 chips are so underpowered that they can't even run Google's smallest LLM models locally on device, so despite their whole AI hype sales pitch, everything on the Pixel 8 is still run in the cloud. How is Google so slow at this? At least the likes of Intel and Microsoft have an excuse, they're old and crusty corporations that move slow.
    - r00fus 3y ago
      It's instructive to know where corporations make their money. Because that's often the only place they innovate. For Google it's web/ads. For Apple it's hardware.
sroussey 3y ago
Siri moved on device, which was a huge engineering effort. But capabilities stagnated, like because of the move on device.
alwillis 3y ago
> I think their main AI case will be photo and video editing Sure, AI features will come to all of their content creation apps at some point, most of it being processed on-device. But I suspect the most impact will be in other apps and areas of the operating systems that typically haven't gotten a lot of attention traditionally. I'd much rather have generative AI or an intelligent agent to deal with email than filling in backgrounds in images. Because the pertinent information is in my calendar, address book, etc., I should be able to say "email Tom I can't make it to the party" and that email is created for me.
lamontcg 3y ago
> Siri feels really abandoned. Apple has a fairly severe case of Apple Maps-related PTSD. They aren't likely to introduce a half-baked product in a rush to market like everyone else is doing. Apple is unlikely to release a fundamental modern-ML update to Siri without it both breaking new ground in features and being considerably more "safe" to use than other products ("safe" in the sense of keeping Apple out of the headlines).

digitcatphd 3y ago

Seems like marketing hype? Surely running local LLMs isn’t that big a market.

binkHN 3y ago
> Surely running local LLMs isn’t that big a market. Agreed. That said, if you do AI/LLMs, few, if any, portable non-Mac systems have the ability to pull this off. The Macs just crush it when it comes to GPU and memory bandwidth performance, and they do this while sipping battery life.
fragmede 3y ago
Right now there's not, but the market for smartphones was pretty small until the iPhone came out. If anybody could create a market for something, I'd believe Apple could do it.
lxgr 3y ago
An LLM able to e.g. summarize the gist of a long text conversation in a group chat that I missed, able to understand queries like "what time did X say we'd meet up, and how long would it take me to get there" etc. would be incredibly useful to me. Sure, most of that is possible server-side – but the appeal would be that local processing could preserve a lot more privacy.
chaostheory 3y ago
You underestimate how much people hate the guardrails for the cloud AIs.
- weikju 3y ago
  It’s not like Apple won’t have guardrails in their local llms…

aiauthoritydev 3y ago

Generally if there are enough people, there will be an ecosystem building software for it.

highwaylights 3y ago

> What gives? You give. More money. These new models aren’t going to sell themselves.

_boffin_ 3y ago

I think that Apple's biggest sell with this stuff is going to be utilizing all the SQLite databases on OSX, IOS, IPadOS along with all the on-device analytics that are constantly running. I have a feeling that we're going to see Journal become more of a center focus over the coming years. Question: Is it illegal for apple to do something on-device that runs queries against all the databases for the user's applications?

meepmorp 3y ago
> Is it illegal for apple to do something on-device that runs queries against all the databases for the user's applications? Why would it be illegal?
- appplication 3y ago
  I assume illegal in the EU antitrust sense where if Apple can access then so should everyone else

LorenDB 3y ago

I have a hunch that Apple hasn't released a folding iPhone simply because they don't want to sell a phone with a crease in the screen. Similarly, they might be sticking with the current Siri design until some LLM flaws can be ironed out. I am not an Apple user though, so I have very little experience with Siri.

alwillis 3y ago

> This makes total sense but does it? Does Apple really have an application roadmap to ultimately utilize these tensor cores? Apple always does things in a holistic way. They wouldn't put these AI cores in their devices if they didn't also have a software strategy to go along with it. Unlike with most companies, AI won't be bolted on; it'll well-integrated into the operating system. For example, they've been working on LLMs[1]: > We demonstrate large improvements over an existing system with similar functionality across different types of references, with our smallest model obtaining absolute gains of over 5% for on-screen references. We also benchmark against GPT-3.5 and GPT-4, with our smallest model achieving performance comparable to that of GPT-4, and our larger models substantially outperforming it. And it's not going to require a terabyte of RAM either. You can read the details: "ReALM: Reference Resolution As Language Modeling" [2] There's also evidence of an AI browsing assistant for Safari. [3] [1]: "Apple AI researchers boast useful on-device model that 'substantially outperforms' GPT-4"--https://9to5mac.com/2024/04/01/apple-ai-gpt-4/ https://9to5mac.com/2024/04/01/apple-ai-gpt-4/ [2]: https://arxiv.org/pdf/2403.20329.pdf https://arxiv.org/pdf/2403.20329.pdf [3]: https://www.macrumors.com/2024/04/10/ios-18-safari-browsing-assistant-rumor/ https://www.macrumors.com/2024/04/10/ios-18-safari-browsing-...

kingkongjaffa 3y ago

Is the lack of LLM siri already because the on-device compute would totally kill battery life? And battery life is a sacred criteria that is unacceptable to regress on for mobile?