6 ms·

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Appl

> Apple has a ridiculous, almost unfathomably deep moat for training and running personalised, customised LLMs and other AI models on the 'edge' with these Apple Silicon chips in all their devices. Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. I haven't built anything against Siri APIs, but I've used Siri and various integrations and every single time I give her a shot, she disappoints me. It can do cookie-cutter, super well-traveled code paths that were engineered together for demos, basically just one off tricks without cohesion, but the whole infrastructure around Siri is not open and has not been improved in any way in many OS versions in any way that I've discerned... But I am not an expert with these systems or APIs. The Neural Engine architecture especially considering M1/M2 hardware leaps (which are absolutely mind-bogglingly impressive) are both really superb technologies from my layman view as a mere software engineer, with improved battery life as well as performance, a real improvement over Intel x86 architectures - I just don't see Apple as a serious AI player right now, despite head start in AI w/ Siri acq, despite these hardware leaps that may make it easier to do cool stuff in the future like this post. The OCR stuff in Photos is cool and useful and I use it every day, usually to look up my Known Traveler Number. Ramblings over, just wanted to ask you to opine as to Apple LLM related information if you have any additional context!

feifan 4y ago
The part that Siri is bad at will be commoditized — someone will open-source a GPT-4-level language model. And Apple's moat will be being able to run that on-device with all the attendant benefits (privacy, zero marginal cost to the company, availability in more scenarios, etc)
- arthurcolle 4y ago
  yeah, makes sense. At least these integrations at a lower level are happening now. I just can't help but feel disappointed that we didn't see earlier cohesion at least a few years ago, maybe around the early or pre-M1 era when there were all those A7, A8 whatever SoCs. But in retrospect, that is basically 10 minutes ago in the history of the universe so yeah, I'm just overly excitable!
- jocaal 4y ago
  You realize that these language models are like 100's of GBs in size and consumes 10's of GB's of memory. Last time I checked, apple still ships their products with less than the market average in both of these specs. If you want a local running LLM on an iphone, get ready to sell a kidney.
  - ithkuil 4y ago
    (assuming we're not talking about the near future) I think this can be a scenario of converging incentives: on one side large models will incentivized hardware manufacturers to increase the memory available on the devices, while on the other sides model developers will be incentivized to trim the fat on the models and devise compression mechanisms that don't compromise quality too much. It's not unthinkable to imagine a hand held device able to run full inference locally a few device generations in the future.
  - l33tman 4y ago
    You can today run an LLM vastly better than Siri on a few GB of RAM using Llama 7B at 4-bit quantization and alpaca.cpp. This is moving so fast, every day there is something new coming. There won't be any moat in LLMs soon or even in dedicated HW as it turns out you don't need that much for "basic intelligence". Note I'm not suggesting you can pack the full knowledgebase of humanity into those 2GB of RAM, but the key feature of an edge AI is simply to understand instructions, something Siri and Ok Google struggle with at best..
parker_mountain 4y ago
> how garbage Siri quality is today a few notes on this - siri is incredibly underinvested. They have another team that's building some sort of search and natural language processing engine, that has slowly sapped away some key headcount from the siri team. - apple doesn't get the full advantage of tons of user data from the wild. this is both a bug and a feature - the siri api model is clearly generations old, and i suspect that apple has been marshaling its resources into a big leap forward - alongside their hand tracking, ar, and hardware - apple has shipped everything required for you to point at a light /in your house/ to turn it on or off (and optionally flick). this includes software - the individual components are built and ready - the only thing missing is gluing it together There's something happening, and I think that the rumored glasses are the hardware totem.
- afavour 4y ago
  “Yes Siri is garbage but it doesn’t matter because Apple are probably working on a top secret new thing that will blow everyone else out of the water” You might be right but there’s no real evidence for it yet.
  - mikepurvis 4y ago
    Lots of things with Apple come out of nowhere. In this case there's actually quite a bit more evidence than usual because of all the fielded, underutilized hardware that's ready to go— when it launches, it'll likely be available on millions of devices, rather than as something you have to get the latest hardware to have, as is the case when the camera or screen undergoes a drastic improvement, or there's a new feature like wireless charging or NFC payments.
  - LoganDark 4y ago
    Hasn't Siri remained relatively unchanged for over a decade? I think the most recent innovation was just bringing it to Intel Macs.
    - bigDinosaur 4y ago
      It has had features removed.
BonoboIO 4y ago
Same experience. Alexa is DIMENSIONS better than Siri. Siri can create a timer ... okay even an alarm. That’s it. It is comically bad. Their text to speech is excellent, but the rest is unbelievable bad. If Apple has some kind of silver bullet, it’s time to put it out or be left behind.
- bradDonniger 4y ago
  I don’t have all the details because NDAs but I ask questions and get nods and grins from colleagues and friends at chip companies; Apple (and nVidia and Intels) silver bullet to cloud hosted, software based AI is AI chips. We’re circling back around to local compute being the default as hardware performance of next gen phones and tablets reaches a “good enough” point for most users. There will be scientific problems that will require modern server clusters but most consumer facing AI needs will be done on hardware within a decade. I’m not saying AGI in a decade, I’m saying cutting edge logic embedded in software now will be the basis for logic in chips in the years to come.
- fbdab103 4y ago
  Can we take a minute to comment on how bad the text correction is on the iPhone?
- Tiktaalik 4y ago
  Amazon has publicly walked back from Alexa because turns out people don't use it for much beyond creating a timer and it doesn't make money. I think it's quite likely that Apple realized the same quite a bit earlier and eased up on Siri versus focusing on other things.
  - vbezhenar 4y ago
    Honestly I observe zero progress in the last 10 years. My first iPhone was 4S. My current iPhone is 8. My use-case is absolutely the same. Calls, SMS, Whatsapp, navigation, books, browser. The same with macs. I boot, open IDE, Terminal and that's about it. Yes, hardware's getting faster (which is compensated by software getting slower). But nothing changed dramatically. I think that the only thing that changed is adoption of fingerprint reader. Both in iPhone 8 and in mac. AI might be that thing that could tremendously change my usage pattern. But it should be as smart as human assistant. GPT4-level intelligence might have the necessary power for that.
gumby 4y ago
> > Apple has a ridiculous, almost unfathomably deep moat for training and running [LLMs]... > Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. Siri is a dead end. When Jobs bought Siri (what, 10 years ago?) he explicitly junked almost all the AI back end, mainly buying the speech recognition engine. I didn't understand why and still don't (but strangely he didn't ask me :-). John Giannandrea has run Apple's AI effort for the past five or six years. He is the reason Google has a big AI effort (he consolidated a bunch of AI projects and bought Deep Mind, etc) before he decamped for Apple. For all I know the Siri team isn't even part of his remit. You can never look into Apple (even if you work there) so one can only speculate based on what visible signs appear. But Siri isn't one of them.
- duped 4y ago
  Jobs has been dead for over ten years
  - gumby 4y ago
    And it feels like nobody has worked on it since then
  - abudabi123 4y ago
    The cinema industry know how to squeeze a billion dollar a year out of Apple. Do the Apple or Sony ”social engineers“ have total view? When Amazon was sky rocketing and Blue Origin had not reached orbit a niche producer from the cinema industry had Jeff B paying a lot although he might have backed out of that deal like giving up on the Android phone with a lot of cameras.
    - dev_tty01 4y ago
      >When Amazon was sky rocketing and Blue Origin had not reached orbit... Blue Origin still has not reached orbit.
- sidibe 4y ago
  > He is the reason Google has a big AI effort (he consolidated a bunch of AI projects and bought Deep Mind, etc) before he decamped for Apple I'm sure he's great but this made me laugh
richardw 4y ago
If you’re the exec in charge of this at Apple, what are you doing right now. It’s not watching OpenAI become what Siri could be unless you have something to do about it. Your motivation levels are extremely high.
deleted 4y ago
[deleted]
theptip 4y ago
I don’t think Apple necessarily needs to lead the pack on LLM research. They just need to take the SOTA FOSS model and bake it into silicon, using their chip design expertise. If you do that, you can do inference on mobile devices, which is a huge privacy win; it plus into their general privacy positioning in a big way. If you open up that SDK to developers it would be the iOS app gold rush all over again. Note, Google is also moving in the direction of on-device inference with Coral, but they obviously will want to transmit the personalized model weights back to the mothership.
mannyv 4y ago
Siri is garbage because it's trying to be a real AI, as it was understood to be at the time. Alexa works because it's not an AI but it seems as if it's an AI.
agentcoops 4y ago
Honestly, it speaks mountains to Apple as an organisation that they did not invest all of their resources into Siri, especially when you consider how Amazon invested in Alexa etc. Everyone has long understood that the "personal assistant" represents really the pinnacle of not just what AI but the personal computer should be able to offer -- the mistake this past decade was thinking that had to do with then current ML paradigms or with these completely useless voice agents. That Apple could realise it was a dead-end and not sink much more investment/engineering resources into the tool any further than what anyone actually uses these tools for (changing song, lowering volume, etc) is outstanding. Ultimately, I'm very reassured that it's Apple and Microsoft leading the way for personal and business AI assistants respectively. As Stratechery has emphasised, the latter not only already has all the data from most firms outside Silicon Valley, but more importantly has always adhered to and designed their software according to a computer-as-a-tool philosophy versus Google's "ML/AI will do everything for you behind the scenes"; the former has not only made a show of data security but invested massively in the hardware architectures necessary to making on-device AI a possibility. Without at all being a "fanboy", Apple is quite literally the only company I would trust my personal data to for use in a GPT-based assistant.
- mark212 4y ago
  Amazon's investment in Alexa is a perfect example of why Apple didn't (and shouldn't have) invested all of their resources into Siri. The Alexa team is getting seriously gutted in these rounds of downsizing [1], or as CNBC puts it "the team behind the technology was a prime target of the largest layoffs in the company’s history." What they have been invested heavily in is the Apple Neural Engine ("ANE"), special silicon right on the SoC to handle ML / AI code. Optimize on a server, then run the model on your iPhone or probably soon, your Apple Watch. WWDC this year is going to be very, very important. [1] https://www.cnbc.com/2023/01/06/amazon-fully-committed-to-alexa-despite-layoffs-hardware-chief-says.html https://www.cnbc.com/2023/01/06/amazon-fully-committed-to-al...

feifan 4y ago

The part that Siri is bad at will be commoditized — someone will open-source a GPT-4-level language model. And Apple's moat will be being able to run that on-device with all the attendant benefits (privacy, zero marginal cost to the company, availability in more scenarios, etc)

arthurcolle 4y ago
yeah, makes sense. At least these integrations at a lower level are happening now. I just can't help but feel disappointed that we didn't see earlier cohesion at least a few years ago, maybe around the early or pre-M1 era when there were all those A7, A8 whatever SoCs. But in retrospect, that is basically 10 minutes ago in the history of the universe so yeah, I'm just overly excitable!
jocaal 4y ago
You realize that these language models are like 100's of GBs in size and consumes 10's of GB's of memory. Last time I checked, apple still ships their products with less than the market average in both of these specs. If you want a local running LLM on an iphone, get ready to sell a kidney.
- ithkuil 4y ago
  (assuming we're not talking about the near future) I think this can be a scenario of converging incentives: on one side large models will incentivized hardware manufacturers to increase the memory available on the devices, while on the other sides model developers will be incentivized to trim the fat on the models and devise compression mechanisms that don't compromise quality too much. It's not unthinkable to imagine a hand held device able to run full inference locally a few device generations in the future.
- l33tman 4y ago
  You can today run an LLM vastly better than Siri on a few GB of RAM using Llama 7B at 4-bit quantization and alpaca.cpp. This is moving so fast, every day there is something new coming. There won't be any moat in LLMs soon or even in dedicated HW as it turns out you don't need that much for "basic intelligence". Note I'm not suggesting you can pack the full knowledgebase of humanity into those 2GB of RAM, but the key feature of an edge AI is simply to understand instructions, something Siri and Ok Google struggle with at best..

parker_mountain 4y ago

> how garbage Siri quality is today a few notes on this - siri is incredibly underinvested. They have another team that's building some sort of search and natural language processing engine, that has slowly sapped away some key headcount from the siri team. - apple doesn't get the full advantage of tons of user data from the wild. this is both a bug and a feature - the siri api model is clearly generations old, and i suspect that apple has been marshaling its resources into a big leap forward - alongside their hand tracking, ar, and hardware - apple has shipped everything required for you to point at a light /in your house/ to turn it on or off (and optionally flick). this includes software - the individual components are built and ready - the only thing missing is gluing it together There's something happening, and I think that the rumored glasses are the hardware totem.

afavour 4y ago
“Yes Siri is garbage but it doesn’t matter because Apple are probably working on a top secret new thing that will blow everyone else out of the water” You might be right but there’s no real evidence for it yet.
- mikepurvis 4y ago
  Lots of things with Apple come out of nowhere. In this case there's actually quite a bit more evidence than usual because of all the fielded, underutilized hardware that's ready to go— when it launches, it'll likely be available on millions of devices, rather than as something you have to get the latest hardware to have, as is the case when the camera or screen undergoes a drastic improvement, or there's a new feature like wireless charging or NFC payments.
- LoganDark 4y ago
  Hasn't Siri remained relatively unchanged for over a decade? I think the most recent innovation was just bringing it to Intel Macs.
  - bigDinosaur 4y ago
    It has had features removed.

BonoboIO 4y ago

Same experience. Alexa is DIMENSIONS better than Siri. Siri can create a timer ... okay even an alarm. That’s it. It is comically bad. Their text to speech is excellent, but the rest is unbelievable bad. If Apple has some kind of silver bullet, it’s time to put it out or be left behind.

bradDonniger 4y ago
I don’t have all the details because NDAs but I ask questions and get nods and grins from colleagues and friends at chip companies; Apple (and nVidia and Intels) silver bullet to cloud hosted, software based AI is AI chips. We’re circling back around to local compute being the default as hardware performance of next gen phones and tablets reaches a “good enough” point for most users. There will be scientific problems that will require modern server clusters but most consumer facing AI needs will be done on hardware within a decade. I’m not saying AGI in a decade, I’m saying cutting edge logic embedded in software now will be the basis for logic in chips in the years to come.
fbdab103 4y ago
Can we take a minute to comment on how bad the text correction is on the iPhone?
Tiktaalik 4y ago
Amazon has publicly walked back from Alexa because turns out people don't use it for much beyond creating a timer and it doesn't make money. I think it's quite likely that Apple realized the same quite a bit earlier and eased up on Siri versus focusing on other things.
- vbezhenar 4y ago
  Honestly I observe zero progress in the last 10 years. My first iPhone was 4S. My current iPhone is 8. My use-case is absolutely the same. Calls, SMS, Whatsapp, navigation, books, browser. The same with macs. I boot, open IDE, Terminal and that's about it. Yes, hardware's getting faster (which is compensated by software getting slower). But nothing changed dramatically. I think that the only thing that changed is adoption of fingerprint reader. Both in iPhone 8 and in mac. AI might be that thing that could tremendously change my usage pattern. But it should be as smart as human assistant. GPT4-level intelligence might have the necessary power for that.

gumby 4y ago

> > Apple has a ridiculous, almost unfathomably deep moat for training and running [LLMs]... > Do they? I can completely fathom, given my own anecdotal experiences, how garbage Siri quality is today. Siri is a dead end. When Jobs bought Siri (what, 10 years ago?) he explicitly junked almost all the AI back end, mainly buying the speech recognition engine. I didn't understand why and still don't (but strangely he didn't ask me :-). John Giannandrea has run Apple's AI effort for the past five or six years. He is the reason Google has a big AI effort (he consolidated a bunch of AI projects and bought Deep Mind, etc) before he decamped for Apple. For all I know the Siri team isn't even part of his remit. You can never look into Apple (even if you work there) so one can only speculate based on what visible signs appear. But Siri isn't one of them.

duped 4y ago
Jobs has been dead for over ten years
- gumby 4y ago
  And it feels like nobody has worked on it since then
- abudabi123 4y ago
  The cinema industry know how to squeeze a billion dollar a year out of Apple. Do the Apple or Sony ”social engineers“ have total view? When Amazon was sky rocketing and Blue Origin had not reached orbit a niche producer from the cinema industry had Jeff B paying a lot although he might have backed out of that deal like giving up on the Android phone with a lot of cameras.
  - dev_tty01 4y ago
    >When Amazon was sky rocketing and Blue Origin had not reached orbit... Blue Origin still has not reached orbit.
sidibe 4y ago
> He is the reason Google has a big AI effort (he consolidated a bunch of AI projects and bought Deep Mind, etc) before he decamped for Apple I'm sure he's great but this made me laugh

richardw 4y ago

If you’re the exec in charge of this at Apple, what are you doing right now. It’s not watching OpenAI become what Siri could be unless you have something to do about it. Your motivation levels are extremely high.

deleted 4y ago

[deleted]

theptip 4y ago

I don’t think Apple necessarily needs to lead the pack on LLM research. They just need to take the SOTA FOSS model and bake it into silicon, using their chip design expertise. If you do that, you can do inference on mobile devices, which is a huge privacy win; it plus into their general privacy positioning in a big way. If you open up that SDK to developers it would be the iOS app gold rush all over again. Note, Google is also moving in the direction of on-device inference with Coral, but they obviously will want to transmit the personalized model weights back to the mothership.

mannyv 4y ago

Siri is garbage because it's trying to be a real AI, as it was understood to be at the time. Alexa works because it's not an AI but it seems as if it's an AI.

agentcoops 4y ago

Honestly, it speaks mountains to Apple as an organisation that they did not invest all of their resources into Siri, especially when you consider how Amazon invested in Alexa etc. Everyone has long understood that the "personal assistant" represents really the pinnacle of not just what AI but the personal computer should be able to offer -- the mistake this past decade was thinking that had to do with then current ML paradigms or with these completely useless voice agents. That Apple could realise it was a dead-end and not sink much more investment/engineering resources into the tool any further than what anyone actually uses these tools for (changing song, lowering volume, etc) is outstanding. Ultimately, I'm very reassured that it's Apple and Microsoft leading the way for personal and business AI assistants respectively. As Stratechery has emphasised, the latter not only already has all the data from most firms outside Silicon Valley, but more importantly has always adhered to and designed their software according to a computer-as-a-tool philosophy versus Google's "ML/AI will do everything for you behind the scenes"; the former has not only made a show of data security but invested massively in the hardware architectures necessary to making on-device AI a possibility. Without at all being a "fanboy", Apple is quite literally the only company I would trust my personal data to for use in a GPT-based assistant.

mark212 4y ago
Amazon's investment in Alexa is a perfect example of why Apple didn't (and shouldn't have) invested all of their resources into Siri. The Alexa team is getting seriously gutted in these rounds of downsizing [1], or as CNBC puts it "the team behind the technology was a prime target of the largest layoffs in the company’s history." What they have been invested heavily in is the Apple Neural Engine ("ANE"), special silicon right on the SoC to handle ML / AI code. Optimize on a server, then run the model on your iPhone or probably soon, your Apple Watch. WWDC this year is going to be very, very important. [1] https://www.cnbc.com/2023/01/06/amazon-fully-committed-to-alexa-despite-layoffs-hardware-chief-says.html https://www.cnbc.com/2023/01/06/amazon-fully-committed-to-al...