3 ms·

I don't understand why Alexa/Siri etc don't just keep their hardcoded rules for things like "set an alarm" and only ship things to a cloud LLM if they don't mat

by lukasb 2y ago

I don't understand why Alexa/Siri etc don't just keep their hardcoded rules for things like "set an alarm" and only ship things to a cloud LLM if they don't match a rule.

mulderc 2y ago
Because that is harder and gives a less consistent experience.
- lukasb 2y ago
  It is more complexity than just "ship everything to an LLM and use tool calls", but the payoff - perfect behavior, along with offline support, for your most common inputs - is worth it I think. I disagree about things being less consistent. Let's imagine a 100% LLM world - in this world, you use a bunch of training to try to get the LLM to match your hardcoded responses for common inputs. If you get your training really right, you get 100% accuracy for these inputs. In this world, no one is complaining about consistency! So why not just hardcode that behavior?
  - TeMPOraL 2y ago
    The whole benefit of LLMs is that humans are not consistent enough. Or at least Apple, Amazon, Google and Microsoft all believe normies don't want to be consistent enough to speak the most common input the same way, allowing to use much simpler and efficient approaches to voice input - like the ones that worked off-line 15+ years ago on a regular PC. LLMs are actually the only reason I'd consider processing voice in the cloud to be a good idea. Alas, knowing how the aforementioned companies designed their assistants in the past, I'm certain they'll find a way to degrade the experience and strip most of the benefits of having LLMs in the loop. After all, as past experience shows, you can't have an assistant letting you operate commercial products and services without speaking the brand names out loud. That's unthinkable.
neuralspark 2y ago
Siri does this for some simpler requests.
jedberg 2y ago
With the Alexa devices that have local processing hardware, that is exactly what happened (prior to this change, if you had the local only option set). But now, in the age of LLMs, there are no "simple rules". An example. Say you live in San Francisco: "Alexa, I'm thinking of going to New York, how many flights are there each day?" This is a hard question, and one that will go to the cloud. "Alexa, what's the weather" This seems like an easy question. But with local only processing, you'd get the weather in San Francisco. But with the LLM, it will probably give you the weather in New York, which is most likely what you wanted if you asked these things just a few seconds apart.
happyopossum 2y ago
Put your iPhone in airplane mode and disable WiFi - most of the basic stuff like 'set an alarm', 'start a timer', etc. will still work. This has been the case for several years - offline Siri was one of the big things they added in iOS 15. [0]https://www.macworld.com/article/678307/how-to-use-siri-offline-on-iphone-and-ipad.html https://www.macworld.com/article/678307/how-to-use-siri-offl...

by lukasb 2y ago

I don't understand why Alexa/Siri etc don't just keep their hardcoded rules for things like "set an alarm" and only ship things to a cloud LLM if they don't match a rule.

mulderc 2y ago

Because that is harder and gives a less consistent experience.

lukasb 2y ago
It is more complexity than just "ship everything to an LLM and use tool calls", but the payoff - perfect behavior, along with offline support, for your most common inputs - is worth it I think. I disagree about things being less consistent. Let's imagine a 100% LLM world - in this world, you use a bunch of training to try to get the LLM to match your hardcoded responses for common inputs. If you get your training really right, you get 100% accuracy for these inputs. In this world, no one is complaining about consistency! So why not just hardcode that behavior?
- TeMPOraL 2y ago
  The whole benefit of LLMs is that humans are not consistent enough. Or at least Apple, Amazon, Google and Microsoft all believe normies don't want to be consistent enough to speak the most common input the same way, allowing to use much simpler and efficient approaches to voice input - like the ones that worked off-line 15+ years ago on a regular PC. LLMs are actually the only reason I'd consider processing voice in the cloud to be a good idea. Alas, knowing how the aforementioned companies designed their assistants in the past, I'm certain they'll find a way to degrade the experience and strip most of the benefits of having LLMs in the loop. After all, as past experience shows, you can't have an assistant letting you operate commercial products and services without speaking the brand names out loud. That's unthinkable.

neuralspark 2y ago

Siri does this for some simpler requests.

jedberg 2y ago

With the Alexa devices that have local processing hardware, that is exactly what happened (prior to this change, if you had the local only option set). But now, in the age of LLMs, there are no "simple rules". An example. Say you live in San Francisco: "Alexa, I'm thinking of going to New York, how many flights are there each day?" This is a hard question, and one that will go to the cloud. "Alexa, what's the weather" This seems like an easy question. But with local only processing, you'd get the weather in San Francisco. But with the LLM, it will probably give you the weather in New York, which is most likely what you wanted if you asked these things just a few seconds apart.

happyopossum 2y ago

Put your iPhone in airplane mode and disable WiFi - most of the basic stuff like 'set an alarm', 'start a timer', etc. will still work. This has been the case for several years - offline Siri was one of the big things they added in iOS 15. [0]https://www.macworld.com/article/678307/how-to-use-siri-offline-on-iphone-and-ipad.html https://www.macworld.com/article/678307/how-to-use-siri-offl...