8 ms·

Question for those familiar with the backend of things like Alexa, Google Home, Siri: At what point can we say things like “turn off the bedroom light in 5 min

Question for those familiar with the backend of things like Alexa, Google Home, Siri: At what point can we say things like “turn off the bedroom light in 5 minutes” or “stop music and turn off all the lights”? Even something like “keep the lights on” in a motion sensor system is impossible it seems. Because to me they feel like low hanging fruit, and yet despite all the advances in machine learning and these systems being around for the better part of a decade, anything but the simplest single-task no-modifier commands result in a “sorry… I didn’t understand that” or completely unpredictable results. Is there something inherently difficult about these types of queries?

runnerup 4y ago
Also why doesn’t Siri work at all in Honda Civics and Honda HR-V’s when connected to car play and driving on the highway with no radio playing? Google assistant works fine, for the most part anyways.
- jeffbee 4y ago
  Weird. Works fine in an Insight, which is in all respects a Civic Hybrid.
Closi 4y ago
These are all subtly different problems I think, but in general most of these architectures currently assume there is a single intent. > stop music and turn off all the lights This is probably the easiest of the bunch because you are asking it to perform two distinct actions. > turn off the bedroom light in 5 minutes This is much more complex, because you are asking the application to setup some sort of workflow - after it understands what you want it to do, it then has to work out how to execute that, which will be utilising the device API's / services. This is a simple example, but there are lots of permutations of different actions here, for example you might want to say "turn off the sound system once this song finishes playing" which assumes that the assistant has the capability to then understand you want it to create a task specifically waiting for the trigger of a particular song finishing playing, and that it has the ability to setup that trigger. > "keep the lights on” in a motion sensor system Now this is where the orchestration gets tricky - The assistant has to: * Work out that the lights are being affected by a motion sensor system, which is likely outside it's own platform. * Work out that your intent is that you want the assistant to override that. * Understand how to connect to the platform in order to control it. * Work out what parameter it is supposed to alter to achieve this task. * Override the existing users settings, and presumably reinstate the settings after some portion of time.
- visarga 4y ago
  Language models can generate code from text instructions. It just needs a training set to learn the target APIs. I expect in the next couple of years to see automated desktop operation (RPA) from text commands, generalising access over human interfaces. It's really a shame the good language models are not deployed as voice assistants. It would probably be expensive to offer and they don't have the scale necessary. Just to load one of these models you need a $100K computer.
  - Closi 4y ago
    It also depends what's the biggest priority - I would assume there is a bigger 'quick win' from becoming more reliable at single-intent actions from a market/customer experience perspective rather than pursuing highly complex multi-intent statements. >99% of commands will be single intent, and they probably work 80% of the time at the moment, so getting those to 99% will have a much bigger short-term impact than focussing on solving the 1% (with the added benfit that once you have solved the first case of getting single-intent right all the time, solving the second more complex queries will be easier as you will have built a more robust base).
  - petra 4y ago
    I'm curious: what's the leading edge in commercial chatbots?
    - speed_spread 4y ago
      Hello, I'm Gretta! I'm here to help. It seems you're looking for information on enterprise chatbots geometry. Answer "yes" to access your account's balance or press # for more options.
- omega3 4y ago
  > This is much more complex, because you are asking the application to setup some sort of workflow - after it understands what you want it to do, it then has to work out how to execute that, which will be utilising the device API's / services. I don't see conceptually the difference between the first and the second example. You're still executing two distinct actions, first being the waiting for x amount of time?
liquidwax 4y ago
My guess is that it's a very different (and difficult) problem to generalize that way. Interpreting intent and taking action are different aspects. Someone needs to write code to call a vendor's API to execute those actions and that's a super specialized action. Next step is probably instruct a CoPilot-like tool to do it.
alphabetting 4y ago
I have google home and the first one worked. The 2nd did not but I don't think I'd ever say that bc I just have routines where I say "i'm leaving" or "i'm going to bed" and everything shuts off at once.
jcoder 4y ago
It’s sad that the only option is a voice assistant that must learn how to interpret my words through this slow, error-prone process. I would much rather have a pure speech-to-text option where I must learn the exact words to say to get a reliable result.
- kupopuffs 4y ago
  Yeah. I wouldn't even mind learning weird syntax or grammar, like Thread.sleep(5 * 60 * 1000); light.off();
  - Ros2 4y ago
    This is basically what iOS shortcuts is hoping to be. You can do some simple coding with widgets like that and then you can invoke your shortcut like `hey siri, <shortcut>`. Avoids reciting the steps you want and gives you reliability, but I suppose requires any 'variables' to be hardcoded (or for you to create multiple instances)
- saalweachter 4y ago
  There was a scifi story where people on a particular space station used a whistle-based UI, where you would whistle a pattern of tones at the computer and it would beep back. I'd go for one of those.
- trention 4y ago
  So, in essence you want a language that will tell the AI what to do? A programming language, if I may say so? It's almost if that natural language thing is highly ambiguous which is why we needed to create a new grammar to give precise instructions for machine-driven processes.
  - jcoder 4y ago
    Nope, no AI, no NLP. Being so in love with that stuff is why everything is so disappointing now. I’d like a pure speech to text mode where I must get the syntax correct, or the system rejects the command. Eg I must say “set a four minute timer”, not “timer for four minutes”.
jeffbee 4y ago
My benchmark for Siri will be when it learns to do "Siri, wake me up an hour earlier tomorrow". What it currently does it set a new alarm for 1 hour in the future.
csnweb 4y ago
> turn off the bedroom light in 5 minutes This actually works already with Siri (and as mentioned in a sibling comment with google Home as well). I just tried that for fun a few days ago and was surprised that it actually worked.
- isatty 4y ago
  It didn’t work for me. “Turn off the lights in 5 minutes did but “turn off the floorstanding lamp in 5 minutes” did not Honestly that’s more frustrating when it’s not uniform and now I’ve to remember this weird behavior.
  - bee_rider 4y ago
    I wonder if the problem is that it doesn't understand what "the lights" is? It must depend on how you've tagged these things, right? In the Phillips* Hue app at least, they have the idea of lights independently, but you can also group them into rooms and stuff like that. So the multiple bulbs in my floor lamp are in a "lamp" grouping but also a "living room" grouping. It all seems quite flexible. What about something like "all lights" "living room lights" etc etc? * Or rather, whoever bought that brand from Phillips
  - csnweb 4y ago
    Does home have some label for that? Can you say turn of the floorstanding lamp and it works? If so that would indeed be very weird. I have a sleeping room in home and said it should turn of the light in the sleeping room in 5 minutes and it worked. May be it only works with rooms.
Rebelgecko 4y ago
>turn off the bedroom light in 5 minutes This works for me, except when it mishears what I'm saying as "turn off the light for 5 minutes", which gets a bit annoying
wyldfire 4y ago
> At what point can we say things like “turn off the bedroom light in 5 minutes” I often get ready to leave the house and say "start the robovac in five minutes" and it just works. The Google home gadget confirms and states that it will start robovac "at $time-of-day". I've not got too many things connected but I assumed that it's generic among those things that can be turned on/off.
wetpaws 4y ago
Anecdotal story, if you have a group of lights, you can't just say to alexa "turn lights off", she will keep asking which one you want. I ended up hardcoding "turn lights off" as a routine trigger. You had one job, Amazon!
shepherdjerred 4y ago
I swear there was a golden age in 2016 where voice assistants were magical and worked very well, and then around 2017/2018 it started to: * Not recognize my voice/command * Load infinitely when I give a command * Do the wrong thing entirely
- function_seven 4y ago
  Or my new favorite (Alexa): “It’s going to be 43° today with clear skies. … By the way, did you know I can tell you about great deals on gifts or everyday items? Just say ‘Alexa, tell me about Prime Savings’. Would you like to hear today’s?” Me: already down the hall cursing my smart speaker.
amitport 4y ago
These systems have important complexity layers that may not be immediately apparent: latency and edge hardware limits. Yes, maybe, gpu cluster server can understand you quickly, but taking whatever model you have and getting it to work quickly enough for people not to be upset is a giant problem.

runnerup 4y ago

Also why doesn’t Siri work at all in Honda Civics and Honda HR-V’s when connected to car play and driving on the highway with no radio playing? Google assistant works fine, for the most part anyways.

jeffbee 4y ago
Weird. Works fine in an Insight, which is in all respects a Civic Hybrid.

Closi 4y ago

These are all subtly different problems I think, but in general most of these architectures currently assume there is a single intent. > stop music and turn off all the lights This is probably the easiest of the bunch because you are asking it to perform two distinct actions. > turn off the bedroom light in 5 minutes This is much more complex, because you are asking the application to setup some sort of workflow - after it understands what you want it to do, it then has to work out how to execute that, which will be utilising the device API's / services. This is a simple example, but there are lots of permutations of different actions here, for example you might want to say "turn off the sound system once this song finishes playing" which assumes that the assistant has the capability to then understand you want it to create a task specifically waiting for the trigger of a particular song finishing playing, and that it has the ability to setup that trigger. > "keep the lights on” in a motion sensor system Now this is where the orchestration gets tricky - The assistant has to: * Work out that the lights are being affected by a motion sensor system, which is likely outside it's own platform. * Work out that your intent is that you want the assistant to override that. * Understand how to connect to the platform in order to control it. * Work out what parameter it is supposed to alter to achieve this task. * Override the existing users settings, and presumably reinstate the settings after some portion of time.

visarga 4y ago
Language models can generate code from text instructions. It just needs a training set to learn the target APIs. I expect in the next couple of years to see automated desktop operation (RPA) from text commands, generalising access over human interfaces. It's really a shame the good language models are not deployed as voice assistants. It would probably be expensive to offer and they don't have the scale necessary. Just to load one of these models you need a $100K computer.
- Closi 4y ago
  It also depends what's the biggest priority - I would assume there is a bigger 'quick win' from becoming more reliable at single-intent actions from a market/customer experience perspective rather than pursuing highly complex multi-intent statements. >99% of commands will be single intent, and they probably work 80% of the time at the moment, so getting those to 99% will have a much bigger short-term impact than focussing on solving the 1% (with the added benfit that once you have solved the first case of getting single-intent right all the time, solving the second more complex queries will be easier as you will have built a more robust base).
- petra 4y ago
  I'm curious: what's the leading edge in commercial chatbots?
  - speed_spread 4y ago
    Hello, I'm Gretta! I'm here to help. It seems you're looking for information on enterprise chatbots geometry. Answer "yes" to access your account's balance or press # for more options.
omega3 4y ago
> This is much more complex, because you are asking the application to setup some sort of workflow - after it understands what you want it to do, it then has to work out how to execute that, which will be utilising the device API's / services. I don't see conceptually the difference between the first and the second example. You're still executing two distinct actions, first being the waiting for x amount of time?

liquidwax 4y ago

My guess is that it's a very different (and difficult) problem to generalize that way. Interpreting intent and taking action are different aspects. Someone needs to write code to call a vendor's API to execute those actions and that's a super specialized action. Next step is probably instruct a CoPilot-like tool to do it.

alphabetting 4y ago

I have google home and the first one worked. The 2nd did not but I don't think I'd ever say that bc I just have routines where I say "i'm leaving" or "i'm going to bed" and everything shuts off at once.

jcoder 4y ago

It’s sad that the only option is a voice assistant that must learn how to interpret my words through this slow, error-prone process. I would much rather have a pure speech-to-text option where I must learn the exact words to say to get a reliable result.

kupopuffs 4y ago
Yeah. I wouldn't even mind learning weird syntax or grammar, like Thread.sleep(5 * 60 * 1000); light.off();
- Ros2 4y ago
  This is basically what iOS shortcuts is hoping to be. You can do some simple coding with widgets like that and then you can invoke your shortcut like `hey siri, <shortcut>`. Avoids reciting the steps you want and gives you reliability, but I suppose requires any 'variables' to be hardcoded (or for you to create multiple instances)
saalweachter 4y ago
There was a scifi story where people on a particular space station used a whistle-based UI, where you would whistle a pattern of tones at the computer and it would beep back. I'd go for one of those.
trention 4y ago
So, in essence you want a language that will tell the AI what to do? A programming language, if I may say so? It's almost if that natural language thing is highly ambiguous which is why we needed to create a new grammar to give precise instructions for machine-driven processes.
- jcoder 4y ago
  Nope, no AI, no NLP. Being so in love with that stuff is why everything is so disappointing now. I’d like a pure speech to text mode where I must get the syntax correct, or the system rejects the command. Eg I must say “set a four minute timer”, not “timer for four minutes”.

jeffbee 4y ago

My benchmark for Siri will be when it learns to do "Siri, wake me up an hour earlier tomorrow". What it currently does it set a new alarm for 1 hour in the future.

csnweb 4y ago

> turn off the bedroom light in 5 minutes This actually works already with Siri (and as mentioned in a sibling comment with google Home as well). I just tried that for fun a few days ago and was surprised that it actually worked.

isatty 4y ago
It didn’t work for me. “Turn off the lights in 5 minutes did but “turn off the floorstanding lamp in 5 minutes” did not Honestly that’s more frustrating when it’s not uniform and now I’ve to remember this weird behavior.
- bee_rider 4y ago
  I wonder if the problem is that it doesn't understand what "the lights" is? It must depend on how you've tagged these things, right? In the Phillips* Hue app at least, they have the idea of lights independently, but you can also group them into rooms and stuff like that. So the multiple bulbs in my floor lamp are in a "lamp" grouping but also a "living room" grouping. It all seems quite flexible. What about something like "all lights" "living room lights" etc etc? * Or rather, whoever bought that brand from Phillips
- csnweb 4y ago
  Does home have some label for that? Can you say turn of the floorstanding lamp and it works? If so that would indeed be very weird. I have a sleeping room in home and said it should turn of the light in the sleeping room in 5 minutes and it worked. May be it only works with rooms.

Rebelgecko 4y ago

>turn off the bedroom light in 5 minutes This works for me, except when it mishears what I'm saying as "turn off the light for 5 minutes", which gets a bit annoying

wyldfire 4y ago

> At what point can we say things like “turn off the bedroom light in 5 minutes” I often get ready to leave the house and say "start the robovac in five minutes" and it just works. The Google home gadget confirms and states that it will start robovac "at $time-of-day". I've not got too many things connected but I assumed that it's generic among those things that can be turned on/off.

wetpaws 4y ago

Anecdotal story, if you have a group of lights, you can't just say to alexa "turn lights off", she will keep asking which one you want. I ended up hardcoding "turn lights off" as a routine trigger. You had one job, Amazon!

shepherdjerred 4y ago

I swear there was a golden age in 2016 where voice assistants were magical and worked very well, and then around 2017/2018 it started to: * Not recognize my voice/command * Load infinitely when I give a command * Do the wrong thing entirely

function_seven 4y ago
Or my new favorite (Alexa): “It’s going to be 43° today with clear skies. … By the way, did you know I can tell you about great deals on gifts or everyday items? Just say ‘Alexa, tell me about Prime Savings’. Would you like to hear today’s?” Me: already down the hall cursing my smart speaker.

amitport 4y ago

These systems have important complexity layers that may not be immediately apparent: latency and edge hardware limits. Yes, maybe, gpu cluster server can understand you quickly, but taking whatever model you have and getting it to work quickly enough for people not to be upset is a giant problem.