4 ms·

All Apple needs to do is allow 3rd party apps to integrate with Siri, then use an LLM as a way to convert natural language into the set of allowed API calls. Ba

All Apple needs to do is allow 3rd party apps to integrate with Siri, then use an LLM as a way to convert natural language into the set of allowed API calls. Basically similar what chatGPT does with function calling. And on osx, they should have already had a head start due to applescript integration everywhere in most apps. I have no idea why they're trying to reinvent the wheel with "Shortcuts" which is severely limited. They already have everything waiting there for their taking, and they're squandering it for no reason. Siri on osx should have been built on top of AppleScript from the get-go, then the switch to LLM would have been easy. For that matter, I wonder why on osx I haven't seen any 3rd party apps be a siri replacement using applescript to drive applications directly. So much effort is spent on screen scraping and trying to get "agents" to use computers like humans do, but for decades osx has already had a parallel set of APIs built into every application specifically for machine consumption. And most good 3rd party apps even make use of it.

justonceokay 2y ago
What you are describing is definitely how Alexa used to work, no idea about now. My only point being that I’m sure this has been thought of before
tsujamin 2y ago
They already have a compostable automation api with 3rd party integrations: Shortcuts! It’s not perfect, but surely you could natural language -> llm -> temporary shortcut script and that gets you a decent part of the way to a smarter Siri
- krackers 2y ago
  I'm not as familiar with Shortcuts API but from a quick glance it seems less rich than apple events/apple script. With LLM + apple script you could achieve computer use agents on easy mode. Not just one-off "what's the weather" queries but complex multi-step interactions like "send the latest photo I took to John". To start with, Automator on mac would be the perfect place for LLM integration. And Script Editor too. Being perhaps one of the few read-only languages, people would probably _prefer_ an LLM spit out applescript for them. And Apple probably has the highest quality data set internally. Combined with the fact that there there is a uniform way to specify the API surface (sdef), this is a task that is possible by most LLMs today. Just apply a little marketing spin to change the angle from "niche power user feature" to "apple uses computer-use agent AI to allow average joe to automate their entire workflow" and it's a smash hit. From there it's not much of a stretch to slap some speech recognition (just take Whisper, it already beats whatever iOS uses), add some guardrails, and have it be orders of magnitude better than what Siri currently is capable of. And this is all possible with today's LLMs, and thanks to deepseek without paying a cent to anyone else. When interactive computer-use does get mostly solved, that can be added in as a fallback for anything where applescript doesn't cut it (e.g. web navigation, electron apps, etc.). But there's a clear logical progression that allows you to easily evolve the user experience as the technology matures without having to throw out your stack the entire time. But to me I think their fate was sealed was Shortcuts was shipped on mac when Automator already existed. And it's clear apple events has been a languishing feature, with integration in native apps already breaking.
  - walterbell 2y ago
    They have since added App Intents to the LLM automation stew.
seec 2y ago
Shortcuts takes precedence because it was originally created on iOS and TC want absolute convergence between the OS. They don't realize (or don't care) how restricting it is to go this route. To be honest that may be the desired result. I have tried to use Shortcuts a few times, and if Automator could be frustrating, Shortcuts is even worse. It seems to have improved with th recent scripting additions, but in my opinion it doesn't a lot of sense. On the phone it seems only useful to make up for lacking software with mediocre voice assistant and on the desktop, it is very lacking and limited. Applescript is an obtuse language but at least it's a programming language still, with the Shortcut toolkit it seems like you need to approach everything in a roundabout way. In any case, it shows the failing of Apple ideology of "easy" computing for everyone. At some point you need to admit that some things are going to need some learning and competency and the best thing you can do is provide tools to enable the competent user. Not create some Frankenstein thing and the incompetent won't understand better anyway. Apple has really failed, chasing "simplicity" at the expense of actual usefulness. And this has permeated even the desktop OS which is just dumb.

justonceokay 2y ago

What you are describing is definitely how Alexa used to work, no idea about now. My only point being that I’m sure this has been thought of before

tsujamin 2y ago

They already have a compostable automation api with 3rd party integrations: Shortcuts! It’s not perfect, but surely you could natural language -> llm -> temporary shortcut script and that gets you a decent part of the way to a smarter Siri

krackers 2y ago
I'm not as familiar with Shortcuts API but from a quick glance it seems less rich than apple events/apple script. With LLM + apple script you could achieve computer use agents on easy mode. Not just one-off "what's the weather" queries but complex multi-step interactions like "send the latest photo I took to John". To start with, Automator on mac would be the perfect place for LLM integration. And Script Editor too. Being perhaps one of the few read-only languages, people would probably _prefer_ an LLM spit out applescript for them. And Apple probably has the highest quality data set internally. Combined with the fact that there there is a uniform way to specify the API surface (sdef), this is a task that is possible by most LLMs today. Just apply a little marketing spin to change the angle from "niche power user feature" to "apple uses computer-use agent AI to allow average joe to automate their entire workflow" and it's a smash hit. From there it's not much of a stretch to slap some speech recognition (just take Whisper, it already beats whatever iOS uses), add some guardrails, and have it be orders of magnitude better than what Siri currently is capable of. And this is all possible with today's LLMs, and thanks to deepseek without paying a cent to anyone else. When interactive computer-use does get mostly solved, that can be added in as a fallback for anything where applescript doesn't cut it (e.g. web navigation, electron apps, etc.). But there's a clear logical progression that allows you to easily evolve the user experience as the technology matures without having to throw out your stack the entire time. But to me I think their fate was sealed was Shortcuts was shipped on mac when Automator already existed. And it's clear apple events has been a languishing feature, with integration in native apps already breaking.
- walterbell 2y ago
  They have since added App Intents to the LLM automation stew.

seec 2y ago

Shortcuts takes precedence because it was originally created on iOS and TC want absolute convergence between the OS. They don't realize (or don't care) how restricting it is to go this route. To be honest that may be the desired result. I have tried to use Shortcuts a few times, and if Automator could be frustrating, Shortcuts is even worse. It seems to have improved with th recent scripting additions, but in my opinion it doesn't a lot of sense. On the phone it seems only useful to make up for lacking software with mediocre voice assistant and on the desktop, it is very lacking and limited. Applescript is an obtuse language but at least it's a programming language still, with the Shortcut toolkit it seems like you need to approach everything in a roundabout way. In any case, it shows the failing of Apple ideology of "easy" computing for everyone. At some point you need to admit that some things are going to need some learning and competency and the best thing you can do is provide tools to enable the competent user. Not create some Frankenstein thing and the incompetent won't understand better anyway. Apple has really failed, chasing "simplicity" at the expense of actual usefulness. And this has permeated even the desktop OS which is just dumb.