7 ms·

Build your own Siri locally and on-device

185 points by andreeamiclaus 1y ago

rrr_oh_man 1y ago
Why an LLM-written article, though?
- norskeld 1y ago
  This summary-like style — with heavy formatting and every (!) paragraph as a bulleted list — drives me nuts tbqh. Especially in lengthy texts, it just looks... noisy, bland, and sometimes confusing.
  - andreeamiclaus 1y ago
    What's the format you would prefer? We're not using ChatGPT to write and we've experimented with this format. The other articles may have a better format?
    - rrr_oh_man 1y ago
      > We're not using ChatGPT to write We can go into semantics, but that listicle has all the hallmarks of LLM-produced stuff, down the the "Generated Image"-tagged image.
      - andreeamiclaus 1y ago
        The image yes, it was generated and that's pretty clear. We used to rely more on ChatGPT but that did not work out (figures!) so now it's a lot of authentic human speech. So I'm curious when you say it has the hallmarks of LLMs since I also recognise a lot of them but not so much here
caust1c 1y ago
So build your own crappy agent-assistant? In earnest though, I'm certain we'll see a community replacement of Siri by end-of-year if the iPhone permissions model allows it or there's some workaround. IDK what the limitations are here but I'm eagerly awaiting the community to step in where Siri has failed.
- Ancapistani 1y ago
  The assistant is only half the story here. This looks like a great, well-defined tutorial project to learn how to put this stuff together locally.
- andrewmcwatters 1y ago
  Crappy? Dude, Siri at one point couldn't even tell you what today's date was. The bar is on the ground.
  - 4ndrewl 1y ago
    Think Different
  - 0cf8612b2e1e 1y ago
    I do not know if it is because I have been trained to make simple requests, but there are only a half dozen things I would verbally ask a robot. - time of day - calendar date - weather - set a timer - simple math calculation That’s 90% of the functionality right there.
    - andrewmcwatters 1y ago
      Funny you mention this. My usage is the same. I suspect we’ve all been trained to expect little-to-nothing from these assistants.
    - jihadjihad 1y ago
      And incredibly it manages to get those wrong a non-trivial amount of the time.
thedeep_mind 1y ago
This is great, thanks for putting this together. Haven't followed it through yet, but does this model run successfully on an iPhone? My 9 year old ran a Qwen 0.6B model using ollama quite well, anything else was too slow to offer a good UX.
- SparkyMcUnicorn 1y ago
  MLC[0] indicates that it can run models in the 8B range on iOS, but 1-3B sounds more reasonable to me. [0] https://llm.mlc.ai/docs/deploy/ios.html#bring-your-own-model https://llm.mlc.ai/docs/deploy/ios.html#bring-your-own-model
- parpfish 1y ago
  Oh, a nine year old PHONE. I was thinking there was a fourth grader out there deploying models when at that age I was still learning multiplication tables.
  - NetOpWibby 1y ago
    My son just turned 9 today so I was like, "Wow! I wonder if my kid would be interested in doing this?"
mrcwinn 1y ago
Cool project and nice write-up!
mystified5016 1y ago
Does Apple even allow you to replace Siri with another assistant? For the longest time on android, all non-Google assistants were crippled by not being able to listen in the background or use the assistant hardkey, gestures, or shortcuts. I'm not sure if the Google assistant still has privileges others don't, but I wouldn't be surprised in the least.
- jedisct1 1y ago
  More or less. This is what Perplexity does.
  - bronco21016 1y ago
    I saw an article about this and downloaded the Perplexity app but I was unable to figure out if this was true? Do I need a paid tier? I just quickly worked through the free sign up and couldn't sort it out. The demo looked really slick. Is it worth pursuing?
- matthewfcarlson 1y ago
  Part of the problem is the wake word “hey siri” is actually handed by a separate coprocessor (AOP) with the model compiled into the firmware. While anything is technically possible, it isn’t as simple as just letting the google app run in the background since the AP is asleep when any of these gesture happen. You could probably setup the action button on the side to open an assistant, but that’s going to be a less pleasant experience (app might not be open, etc). Details are listed below https://machinelearning.apple.com/research/hey-siri https://machinelearning.apple.com/research/hey-siri
  - kimixa 1y ago
    Same with android phones - a super-specific hardcoded phrase is much easier to work in the power budgets required for an "always on" part of the device. It's why a manufacturer (like Samsung) can change that sort of thing on their devices, but it's not realistically something an end user (or even an app) can customize in software. It's not some "arbitrary" limitation.
catapart 1y ago
Man, I'd really love it if this were just a product/app I could download and use a UI to configure/teach. But this guide gives me what I need to make that, I think, so a big thank you for this!
worldsayshi 1y ago
I love the idea and I would like to build something like this. But the few attempts i have made using whisper locally has so far been underwhelming. Has anyone gotten results with small whisper models that are good enough for a use case like this? Maybe I've just had a bad microphone.
- codethief 1y ago
  > Maybe I've just had a bad microphone. Yeah, I would definitely double-check your setup. At work we use Whisper to live-transcribe-and-translate all-hands meetings and it works exceptionally well.
  - s3p 1y ago
    +1 this. Whisper works insanely well. I've been using the medium model as it has yet to mis transcribe anything noticeable, and it's very lightweight. I even converted it to a coreML model so it runs accelerated on apple silicon. It doesn't run *that* much faster than before.. but it ran really fast to begin with. For anyone tinkering, ive had much success with whisper.cpp.
    - azinman2 1y ago
      What was the process of converting it like? I assume you then had to write all of the inference code as well?
      - tough 1y ago
        not the gp but found this https://github.com/ggml-org/whisper.cpp/blob/master/models/convert-whisper-to-coreml.py https://github.com/ggml-org/whisper.cpp/blob/master/models/c...
  - Grimblewald 1y ago
    I'd agree with your experience. I simply sit my phone (~200 dollar motorola, cheap phone) in centre of room, split voice file into chunks using voice prints/ID's I get from a voice embedding model I trained, then feed labelled chunks through whisper, and get a nice transcript of everything said. I combine that with my handwritten notes (as image, get a VLM to transcribe) and the agenda, and I get out really nice meeting minutes as a LaTex document. Works a charm and has turned an hour or two of work per meeting into maybe 30 minutes (proofing what was written).
carmelion 1y ago
[flagged]
jtr1 1y ago
I’ve noticed recently (maybe I missed an announcement) that Siri now functions locally for at least some commands. Try putting an Apple watch in airplane mode and asking it to set a timer or reminder
- swiftcoder 1y ago
  Siri has had limited offline functionality since at least iOS 15? Although I don't think most users noticed at the time, since most of Siri's command vocabulary is for things that require a network connection...
cadamsdotcom 1y ago
Why haven’t Apple taken a look at the data then hardcoded handlers for the top ~1000 usages???
- s3p 1y ago
  They are doing this, just at a mind-numbingly slow pace. They seem to add controls for brightness and power but don't make it clear what works when offline. It's not even worth trying because there's no guide or documentation on what commands would be available. You just have to go into airplane mode and try asking stuff. Awful UX
paul7986 1y ago
Faithful year and half user of chatGPT on my iPhone which has made me loathe Siri for how dumb she is in every sense of the way! When will OpenAI (with the help of Microsoft) release a GPT phone to compete with the iPhone? Im so tired of the boring iPhone! Give me a GPT phone where from my lock screen GPT does everything for me. Fingers crossed :) it's secretively in the works!
Lirael 1y ago
[dead]
kristel100 1y ago
[dead]

Why an LLM-written article, though?

norskeld 1y ago
This summary-like style — with heavy formatting and every (!) paragraph as a bulleted list — drives me nuts tbqh. Especially in lengthy texts, it just looks... noisy, bland, and sometimes confusing.
- andreeamiclaus 1y ago
  What's the format you would prefer? We're not using ChatGPT to write and we've experimented with this format. The other articles may have a better format?
  - rrr_oh_man 1y ago
    > We're not using ChatGPT to write We can go into semantics, but that listicle has all the hallmarks of LLM-produced stuff, down the the "Generated Image"-tagged image.
    - andreeamiclaus 1y ago
      The image yes, it was generated and that's pretty clear. We used to rely more on ChatGPT but that did not work out (figures!) so now it's a lot of authentic human speech. So I'm curious when you say it has the hallmarks of LLMs since I also recognise a lot of them but not so much here

So build your own crappy agent-assistant? In earnest though, I'm certain we'll see a community replacement of Siri by end-of-year if the iPhone permissions model allows it or there's some workaround. IDK what the limitations are here but I'm eagerly awaiting the community to step in where Siri has failed.

Ancapistani 1y ago
The assistant is only half the story here. This looks like a great, well-defined tutorial project to learn how to put this stuff together locally.
andrewmcwatters 1y ago
Crappy? Dude, Siri at one point couldn't even tell you what today's date was. The bar is on the ground.
- 4ndrewl 1y ago
  Think Different
- 0cf8612b2e1e 1y ago
  I do not know if it is because I have been trained to make simple requests, but there are only a half dozen things I would verbally ask a robot. - time of day - calendar date - weather - set a timer - simple math calculation That’s 90% of the functionality right there.
  - andrewmcwatters 1y ago
    Funny you mention this. My usage is the same. I suspect we’ve all been trained to expect little-to-nothing from these assistants.
  - jihadjihad 1y ago
    And incredibly it manages to get those wrong a non-trivial amount of the time.

thedeep_mind 1y ago

This is great, thanks for putting this together. Haven't followed it through yet, but does this model run successfully on an iPhone? My 9 year old ran a Qwen 0.6B model using ollama quite well, anything else was too slow to offer a good UX.

SparkyMcUnicorn 1y ago
MLC[0] indicates that it can run models in the 8B range on iOS, but 1-3B sounds more reasonable to me. [0] https://llm.mlc.ai/docs/deploy/ios.html#bring-your-own-model https://llm.mlc.ai/docs/deploy/ios.html#bring-your-own-model
parpfish 1y ago
Oh, a nine year old PHONE. I was thinking there was a fourth grader out there deploying models when at that age I was still learning multiplication tables.
- NetOpWibby 1y ago
  My son just turned 9 today so I was like, "Wow! I wonder if my kid would be interested in doing this?"

mrcwinn 1y ago

Cool project and nice write-up!

mystified5016 1y ago

Does Apple even allow you to replace Siri with another assistant? For the longest time on android, all non-Google assistants were crippled by not being able to listen in the background or use the assistant hardkey, gestures, or shortcuts. I'm not sure if the Google assistant still has privileges others don't, but I wouldn't be surprised in the least.

jedisct1 1y ago
More or less. This is what Perplexity does.
- bronco21016 1y ago
  I saw an article about this and downloaded the Perplexity app but I was unable to figure out if this was true? Do I need a paid tier? I just quickly worked through the free sign up and couldn't sort it out. The demo looked really slick. Is it worth pursuing?
matthewfcarlson 1y ago
Part of the problem is the wake word “hey siri” is actually handed by a separate coprocessor (AOP) with the model compiled into the firmware. While anything is technically possible, it isn’t as simple as just letting the google app run in the background since the AP is asleep when any of these gesture happen. You could probably setup the action button on the side to open an assistant, but that’s going to be a less pleasant experience (app might not be open, etc). Details are listed below https://machinelearning.apple.com/research/hey-siri https://machinelearning.apple.com/research/hey-siri
- kimixa 1y ago
  Same with android phones - a super-specific hardcoded phrase is much easier to work in the power budgets required for an "always on" part of the device. It's why a manufacturer (like Samsung) can change that sort of thing on their devices, but it's not realistically something an end user (or even an app) can customize in software. It's not some "arbitrary" limitation.

catapart 1y ago

Man, I'd really love it if this were just a product/app I could download and use a UI to configure/teach. But this guide gives me what I need to make that, I think, so a big thank you for this!

worldsayshi 1y ago

I love the idea and I would like to build something like this. But the few attempts i have made using whisper locally has so far been underwhelming. Has anyone gotten results with small whisper models that are good enough for a use case like this? Maybe I've just had a bad microphone.

codethief 1y ago
> Maybe I've just had a bad microphone. Yeah, I would definitely double-check your setup. At work we use Whisper to live-transcribe-and-translate all-hands meetings and it works exceptionally well.
- s3p 1y ago
  +1 this. Whisper works insanely well. I've been using the medium model as it has yet to mis transcribe anything noticeable, and it's very lightweight. I even converted it to a coreML model so it runs accelerated on apple silicon. It doesn't run *that* much faster than before.. but it ran really fast to begin with. For anyone tinkering, ive had much success with whisper.cpp.
  - azinman2 1y ago
    What was the process of converting it like? I assume you then had to write all of the inference code as well?
    - tough 1y ago
      not the gp but found this https://github.com/ggml-org/whisper.cpp/blob/master/models/convert-whisper-to-coreml.py https://github.com/ggml-org/whisper.cpp/blob/master/models/c...
- Grimblewald 1y ago
  I'd agree with your experience. I simply sit my phone (~200 dollar motorola, cheap phone) in centre of room, split voice file into chunks using voice prints/ID's I get from a voice embedding model I trained, then feed labelled chunks through whisper, and get a nice transcript of everything said. I combine that with my handwritten notes (as image, get a VLM to transcribe) and the agenda, and I get out really nice meeting minutes as a LaTex document. Works a charm and has turned an hour or two of work per meeting into maybe 30 minutes (proofing what was written).

carmelion 1y ago

[flagged]

jtr1 1y ago

I’ve noticed recently (maybe I missed an announcement) that Siri now functions locally for at least some commands. Try putting an Apple watch in airplane mode and asking it to set a timer or reminder

swiftcoder 1y ago
Siri has had limited offline functionality since at least iOS 15? Although I don't think most users noticed at the time, since most of Siri's command vocabulary is for things that require a network connection...

cadamsdotcom 1y ago

Why haven’t Apple taken a look at the data then hardcoded handlers for the top ~1000 usages???

s3p 1y ago
They are doing this, just at a mind-numbingly slow pace. They seem to add controls for brightness and power but don't make it clear what works when offline. It's not even worth trying because there's no guide or documentation on what commands would be available. You just have to go into airplane mode and try asking stuff. Awful UX

paul7986 1y ago

Faithful year and half user of chatGPT on my iPhone which has made me loathe Siri for how dumb she is in every sense of the way! When will OpenAI (with the help of Microsoft) release a GPT phone to compete with the iPhone? Im so tired of the boring iPhone! Give me a GPT phone where from my lock screen GPT does everything for me. Fingers crossed :) it's secretively in the works!

Lirael 1y ago

[dead]

kristel100 1y ago

[dead]