9 ms·

AI Assistants Have Poor Usability: Study of Alexa, Google Assistant, Siri (2018)

56 points by spideymans 6y ago

meheszjeno 6y ago
These solutions don't mean to be help/assist their users but to help their vendors to collect information about the users.
- trevyn 6y ago
  Correct; incentives are not aligned. Even for Apple.
- briga 6y ago
  Can't they do both? I personally find Google Assistant very helpful. I'm sure Google finds my personal data helpful. Seems like a fairly reasonable trade-off for me
  - imglorp 6y ago
    I've had the opposite experience: completely useless due to poor integrations. Example: "Send a slack to Mary". Fail. "Read my most recent text message", fail. Even Hangouts is badly integrated, if at all. I think the problem is, as always, a combination of business goals. First, I suspect they don't want to help apps they haven't made a cross marketing deal with, like Slack. They'll install the app but won't work with it. Eg, "Play Tool on Spotify" works fine. Second, Google's internal novelty-chasing merit system doesn't reward a coherent customer experience. You'd think Hangouts (or whatever chat is called this week) would be seamless after 5 years, but it's like they never met.
- drdaeman 6y ago
  I disagree. It's not about collecting information - at least, not directly. Assistants don't seem to be designed to gather neither any substantial amounts of information, nor any unique information. Due to their limited communication capabilities they're extremely poor at collecting meaningful data - there's simply no point in talking to them as their utility fails with anything but the simplest and most direct commands. There's not much data in "turn kitchen lights on", "set alarm at 10am", "play $album by $artist", "remind me to buy milk" and "convert 4.7 miles to kilometers" - at best, this exposes a few minor behavioral patterns that can be trivially determined even without a dedicated device to talk to. And, hopefully, outright spying wasn't one of the design goals, even though they can be of dual use. Google, Apple and Amazon may be cooperative but they aren't NSA, after all. My opinion, it's about use of particular services. Alexa will push you towards Amazon purchases, use of Amazon Music, etc. Similarly, Homepod requires investment into Apple ecosystem. And Google Assistant will push towards all things Google. In my opinion, voice assistants were made to be helpful within their particular ecosystems and thus add more ties to their vendors' services. (And, sure, those services collect data.)
- IshKebab 6y ago
  Why does HN have so many conspiracy theorists? They're clearly designed to make money by keeping you in the Amazon / Google / Apple ecosystem. I don't think any of them make much money from data collection because they don't collect that much data. I have Google Homes and you can go and look at all the data it collects - for me it is 90% "hey Google play some music" or "hey Google what's the weather" or "hey Google set a timer for 10 minutes". Useful for improving they speech recognition but not much else.
  - hyperman1 6y ago
    There is 1 more important piece of information: Your voiceprint. With it, google can identify you by voice, put a mic in a public infopanel, and gues which commercial has most impact. If the NSA has access, they can classify public recordings by people at that place, so I assume they are interested. HN has so many conspiracy theorists because we know what is possible with infotech on an industrial scale, the spooks have the money for it, and snowden basically provided the proof they did it.
    - viraptor 6y ago
      > put a mic in a public infopanel, and gues which commercial has most impact There are infopanels (from smaller dodgy companies) with cameras which try to identify viewers behaviours. But actually expecting Google to both identify you and measure response is a bit out there for many reasons. 1. They actually do care about access to data they store - creating a fingerprint database for identification would be beyond what they do. 2. Who actually reacts vocally to an ad in a way that can be captured without environment noise - the assistant can't even set the timer correctly sometimes when I talk to it in perfect conditions. 3. If they did want to do super-shady targeting, your location data crossed with ad location + search/browser history is both better and cheaper than a voice database.
      - TeMPOraL 6y ago
        > 1. They actually do care about access to data they store - creating a fingerprint database for identification would be beyond what they do. Would it? What if it was structured the way they handle other personal data: customers send in "voice telemetry" from their "info"panels, and Google gives them bulk analytics ("#x different people talked in your venue, with following age/demographic split"), as well as using that telemetry to handle ad attribution and ad targeting in the background. This way - just like with regular web telemetry - neither the advertiser not the "info"panel owner get to see the voiceprint data, but it still gets used to uniquely identify individuals and target ads at them.
- TulliusCicero 6y ago
  They're pretty obviously designed to do both.
sdenton4 6y ago
This is a bit inevitable. The UI of your usual screen based app has buttons for the things you can do. This has the effect of limiting your expectations to those features which have been implemented. A general voice query is open ended, though: there's no guidance around what exists, so it's very easy to invent a query that's 'off book' in one way or another... Things which haven't been implemented at all, or which have, but are requested in a way the machine does not expect.
- chrisin2d 6y ago
  Indeed, an open-ended interface like a VUI invite unimplemented interactions. I worked on a conversational agent and now realise that there are fundamental usability barriers that are very difficult to overcome: - Limited information bandwidth. Adults can visually read between 250–600 words per minute, plus they have peripheral vision that can help in scanning for shapes, colors, and images. Plus information in a GUI is persistent on a screen for easy reference. For voice, adults can only comfortably listen at 150–160 WPM and would need to hold information in their memory for reference. This makes voice ecommerce impractical for anything beyond familiar essentials. - Lack of editing layer. It’s simple and straightforward to correct text box input, but it’s difficult to correct a voice command and doing so is ambiguous and adds extra chaotic information. A lot of people think aloud, so they frequently self-correct themselves mid-sentence or add on information in an ad-hoc manner: “I’d like a cappuci—er, make it a latte. (pause) And oh with soy milk.” - High context switching cost. GUI’s keep contexts within frames, tabs, and windows — and the user can easily (low cost) switch between them. This is an Amazon shopping cart context, this is a Reminders context, and so on. A button press is unambiguously contained within its context. In voice, contexts are not quite parallel but sequential and have to be built up over time.
marta_morena_25 6y ago
I always hated voice detection in call centers. Happened about 20 years ago or so? And since then, nothing much has changed. I still have to control myself whenever I am greeted with "What can I do for you today" and the inevitable "I did not understand. What can I do for you today?". I think the main problem here is that they are trying to build devices for which science is still decades away. What AI assistants would need is the capability to fully parse any normal utterance and then interpret what it really means in computer terms and execute on it and report back in a manner that the user understands. This is essentially general AI. However, we have to start somewhere, otherwise there will be no funding... Kudos to those people who use these devices and fund the technological advance :).
gundmc 6y ago
What's the point of posting this analysis that is over 2 years old in a young and rapidly iterating space?
- outtatime 6y ago
  Have human voices changed that much in two years?
  - leafboi 6y ago
    No But recognition and analysis has...
raobit 6y ago
Who genuinely uses Assistants from their respective phone and for what tasks in general,just wanted to know is someone really making most use of it?
- cik2e 6y ago
  I use mine all the time to send texts, make phone calls, set alarms and timers, and add calendar events.
- xapata 6y ago
  Weather, news, music, and random questions while I'm not at my computer, or I don't have my hands on the keyboard.
- endanke 6y ago
  I use them almost on a daily basis to control the lights in my apartment, it's surprisingly convenient and helps with multitasking when I'm in a hurry. The same thing goes for weather and reminders. The best use case when you're in the middle of something, eg. cooking and just want to make a simple note or set a timer.
  - gambiting 6y ago
    Does it actually work for you though? I've quickly abandoned Google assistant because it just misunderstands what I say about 2/3s of the time. A friend of mine who has all his lights controllable by voice complains about the same issue - a lot of the time, Google just doesn't understand the command so by the time you get it "right" you could have walked up to the switch 3 times.
    - TheOtherHobbes 6y ago
      I had Alexa controlling the lights for a while, but I found talking requires much more cognitive effort than picking up a remote, finding the right button by touch, and pushing it. I can literally do the latter three quarters asleep, but not so much the former. It would be far more useful to have the process almost completely automated. Lights go on when someone enters a room and go off when everyone leaves, with optional manual override. This turns out to be a hard(ish) problem that needs better sensors and/or some form of personal ID.
29athrowaway 6y ago
These days, a useful command has been "Alexa, what is the air quality index?". It's a good way of knowing if I can open the window.
- brokenmachine 6y ago
  That's /r/aboringdystopia material right there.
mlang23 6y ago
My main criticism about Alexa at least boils down to bad usability. To me, it feels like learning a bunch of magic spells by heart. It feels so awkward. I am used to exact commands as a long-time CLI user. However, when it comes to speech, I somehow don't want to accept having to remember the other end of a pattern matcher. All the skills I have tried so far had a similar feel to it. I feel overwhelmed when I realize I have to remember how to launch the skill. I should probably give GA a try one day. But for now, I give the voice assistant thing 5 more years, hopefully it will be something I want to use then.
- lolinder 6y ago
  I took a class in college on voice interface design, and the professor named this as the biggest hurdle that's yet to be overcome. Screens and keyboards are the computer's domain, and we're comfortable adapting to them. But speech is our domain, and we expect the computer to adapt to us. The problem is that the tech isn't quite there for real understanding of language.
Catsandkites 6y ago
Half on-topic: Is there an API on Android that allows you to write your own assistant with your own keyword set without sending everything over the wire? An app with microphone permissions would obviously work-ish, but is it possible to listen in the background like the assistants do?
thelazydogsback 6y ago
We've had better technology in NLP and problem-solving since the mid-1970's -- and many methods that were deemed intractable are trivial now given that compute is on the order of 1 million times more powerful. Systems that use partial-order hierarchical planning, abductive logic programming, bi-directional search, constraint satisfaction, multi-modal interaction, multi-agent conversational modeling based on planning applicable speech-acts, etc., have been replaced with simple state-machines, and in "advanced" cases, simple slot-filling -- and more recently relatively shallow black-box stochastic methods based on deep learning models. Having worked at two companies that offered such systems, I can say that this is due to both not understanding/appreciating what has been done in the past, but also because companies treat these systems like car companies treat features -- they know that an deliver rich features in the future, so the "logic" is to wait to do so, in the hope they can make more money in the long-term, and be able to dribble out new features. To be fair, there is also the issue of reproducibility and predictability -- companies expect everybody's agent to respond the same way given the same input, rather than have variability in agents due to non-determinism or context and learning -- but we can't ask for for flexible "human-like" AI on one hand and also not expect the variation (and occasional misunderstandings) that humans would also suffer.
jbarrs 6y ago
>Even though it goes against the basic premise of human-centered design, users have to train themselves to understand when an intelligent assistant will be useful and when it’s better to avoid using it. I find this is one of the truest things about this article. It's very easy when you first get a new voice assistant to get carried away with the features it has available. Most are either very basic, struggling to take much additional context, or simply just gimmicks/novelties. The majority of tasks can be performed faster through a GUI when you account for the time it takes to brute force the right "natural language" expression to do the thing you need.
- aaomidi 6y ago
  Well the idea is that you get hands free control of the device. I'm laying in bed and I suddenly realize I need an alarm for tomorrow morning. I don't want to turn my phone on cause that can mess with my sleep. I just tell siri or Google to set an alarm and sleep. Or I'm in the middle of cooking and I want to listen to music but don't want to have to wash my hands first. That's really where the benefit of the assistants lays.
dafoex 6y ago
This would potentially destroy any hands free benefit a virtual assistant would have, back in the day of the secretary (the roll these talking Pringles tubes are trying to emulate) the bossman CEO or whatever would press a button on the office intercom to talk with his secretary. I think virtual assistants would benefit from a simple push-to-talk style input so they don't misinterpret a pause as a stop.

56 points by spideymans 6y ago

These solutions don't mean to be help/assist their users but to help their vendors to collect information about the users.

trevyn 6y ago
Correct; incentives are not aligned. Even for Apple.
briga 6y ago
Can't they do both? I personally find Google Assistant very helpful. I'm sure Google finds my personal data helpful. Seems like a fairly reasonable trade-off for me
- imglorp 6y ago
  I've had the opposite experience: completely useless due to poor integrations. Example: "Send a slack to Mary". Fail. "Read my most recent text message", fail. Even Hangouts is badly integrated, if at all. I think the problem is, as always, a combination of business goals. First, I suspect they don't want to help apps they haven't made a cross marketing deal with, like Slack. They'll install the app but won't work with it. Eg, "Play Tool on Spotify" works fine. Second, Google's internal novelty-chasing merit system doesn't reward a coherent customer experience. You'd think Hangouts (or whatever chat is called this week) would be seamless after 5 years, but it's like they never met.
drdaeman 6y ago
I disagree. It's not about collecting information - at least, not directly. Assistants don't seem to be designed to gather neither any substantial amounts of information, nor any unique information. Due to their limited communication capabilities they're extremely poor at collecting meaningful data - there's simply no point in talking to them as their utility fails with anything but the simplest and most direct commands. There's not much data in "turn kitchen lights on", "set alarm at 10am", "play $album by $artist", "remind me to buy milk" and "convert 4.7 miles to kilometers" - at best, this exposes a few minor behavioral patterns that can be trivially determined even without a dedicated device to talk to. And, hopefully, outright spying wasn't one of the design goals, even though they can be of dual use. Google, Apple and Amazon may be cooperative but they aren't NSA, after all. My opinion, it's about use of particular services. Alexa will push you towards Amazon purchases, use of Amazon Music, etc. Similarly, Homepod requires investment into Apple ecosystem. And Google Assistant will push towards all things Google. In my opinion, voice assistants were made to be helpful within their particular ecosystems and thus add more ties to their vendors' services. (And, sure, those services collect data.)
IshKebab 6y ago
Why does HN have so many conspiracy theorists? They're clearly designed to make money by keeping you in the Amazon / Google / Apple ecosystem. I don't think any of them make much money from data collection because they don't collect that much data. I have Google Homes and you can go and look at all the data it collects - for me it is 90% "hey Google play some music" or "hey Google what's the weather" or "hey Google set a timer for 10 minutes". Useful for improving they speech recognition but not much else.
- hyperman1 6y ago
  There is 1 more important piece of information: Your voiceprint. With it, google can identify you by voice, put a mic in a public infopanel, and gues which commercial has most impact. If the NSA has access, they can classify public recordings by people at that place, so I assume they are interested. HN has so many conspiracy theorists because we know what is possible with infotech on an industrial scale, the spooks have the money for it, and snowden basically provided the proof they did it.
  - viraptor 6y ago
    > put a mic in a public infopanel, and gues which commercial has most impact There are infopanels (from smaller dodgy companies) with cameras which try to identify viewers behaviours. But actually expecting Google to both identify you and measure response is a bit out there for many reasons. 1. They actually do care about access to data they store - creating a fingerprint database for identification would be beyond what they do. 2. Who actually reacts vocally to an ad in a way that can be captured without environment noise - the assistant can't even set the timer correctly sometimes when I talk to it in perfect conditions. 3. If they did want to do super-shady targeting, your location data crossed with ad location + search/browser history is both better and cheaper than a voice database.
    - TeMPOraL 6y ago
      > 1. They actually do care about access to data they store - creating a fingerprint database for identification would be beyond what they do. Would it? What if it was structured the way they handle other personal data: customers send in "voice telemetry" from their "info"panels, and Google gives them bulk analytics ("#x different people talked in your venue, with following age/demographic split"), as well as using that telemetry to handle ad attribution and ad targeting in the background. This way - just like with regular web telemetry - neither the advertiser not the "info"panel owner get to see the voiceprint data, but it still gets used to uniquely identify individuals and target ads at them.
TulliusCicero 6y ago
They're pretty obviously designed to do both.

sdenton4 6y ago

This is a bit inevitable. The UI of your usual screen based app has buttons for the things you can do. This has the effect of limiting your expectations to those features which have been implemented. A general voice query is open ended, though: there's no guidance around what exists, so it's very easy to invent a query that's 'off book' in one way or another... Things which haven't been implemented at all, or which have, but are requested in a way the machine does not expect.

chrisin2d 6y ago
Indeed, an open-ended interface like a VUI invite unimplemented interactions. I worked on a conversational agent and now realise that there are fundamental usability barriers that are very difficult to overcome: - Limited information bandwidth. Adults can visually read between 250–600 words per minute, plus they have peripheral vision that can help in scanning for shapes, colors, and images. Plus information in a GUI is persistent on a screen for easy reference. For voice, adults can only comfortably listen at 150–160 WPM and would need to hold information in their memory for reference. This makes voice ecommerce impractical for anything beyond familiar essentials. - Lack of editing layer. It’s simple and straightforward to correct text box input, but it’s difficult to correct a voice command and doing so is ambiguous and adds extra chaotic information. A lot of people think aloud, so they frequently self-correct themselves mid-sentence or add on information in an ad-hoc manner: “I’d like a cappuci—er, make it a latte. (pause) And oh with soy milk.” - High context switching cost. GUI’s keep contexts within frames, tabs, and windows — and the user can easily (low cost) switch between them. This is an Amazon shopping cart context, this is a Reminders context, and so on. A button press is unambiguously contained within its context. In voice, contexts are not quite parallel but sequential and have to be built up over time.

marta_morena_25 6y ago

I always hated voice detection in call centers. Happened about 20 years ago or so? And since then, nothing much has changed. I still have to control myself whenever I am greeted with "What can I do for you today" and the inevitable "I did not understand. What can I do for you today?". I think the main problem here is that they are trying to build devices for which science is still decades away. What AI assistants would need is the capability to fully parse any normal utterance and then interpret what it really means in computer terms and execute on it and report back in a manner that the user understands. This is essentially general AI. However, we have to start somewhere, otherwise there will be no funding... Kudos to those people who use these devices and fund the technological advance :).

gundmc 6y ago

What's the point of posting this analysis that is over 2 years old in a young and rapidly iterating space?

outtatime 6y ago
Have human voices changed that much in two years?
- leafboi 6y ago
  No But recognition and analysis has...

raobit 6y ago

Who genuinely uses Assistants from their respective phone and for what tasks in general,just wanted to know is someone really making most use of it?

cik2e 6y ago
I use mine all the time to send texts, make phone calls, set alarms and timers, and add calendar events.
xapata 6y ago
Weather, news, music, and random questions while I'm not at my computer, or I don't have my hands on the keyboard.
endanke 6y ago
I use them almost on a daily basis to control the lights in my apartment, it's surprisingly convenient and helps with multitasking when I'm in a hurry. The same thing goes for weather and reminders. The best use case when you're in the middle of something, eg. cooking and just want to make a simple note or set a timer.
- gambiting 6y ago
  Does it actually work for you though? I've quickly abandoned Google assistant because it just misunderstands what I say about 2/3s of the time. A friend of mine who has all his lights controllable by voice complains about the same issue - a lot of the time, Google just doesn't understand the command so by the time you get it "right" you could have walked up to the switch 3 times.
  - TheOtherHobbes 6y ago
    I had Alexa controlling the lights for a while, but I found talking requires much more cognitive effort than picking up a remote, finding the right button by touch, and pushing it. I can literally do the latter three quarters asleep, but not so much the former. It would be far more useful to have the process almost completely automated. Lights go on when someone enters a room and go off when everyone leaves, with optional manual override. This turns out to be a hard(ish) problem that needs better sensors and/or some form of personal ID.

29athrowaway 6y ago

These days, a useful command has been "Alexa, what is the air quality index?". It's a good way of knowing if I can open the window.

brokenmachine 6y ago
That's /r/aboringdystopia material right there.

mlang23 6y ago

My main criticism about Alexa at least boils down to bad usability. To me, it feels like learning a bunch of magic spells by heart. It feels so awkward. I am used to exact commands as a long-time CLI user. However, when it comes to speech, I somehow don't want to accept having to remember the other end of a pattern matcher. All the skills I have tried so far had a similar feel to it. I feel overwhelmed when I realize I have to remember how to launch the skill. I should probably give GA a try one day. But for now, I give the voice assistant thing 5 more years, hopefully it will be something I want to use then.

lolinder 6y ago
I took a class in college on voice interface design, and the professor named this as the biggest hurdle that's yet to be overcome. Screens and keyboards are the computer's domain, and we're comfortable adapting to them. But speech is our domain, and we expect the computer to adapt to us. The problem is that the tech isn't quite there for real understanding of language.

Catsandkites 6y ago

Half on-topic: Is there an API on Android that allows you to write your own assistant with your own keyword set without sending everything over the wire? An app with microphone permissions would obviously work-ish, but is it possible to listen in the background like the assistants do?

thelazydogsback 6y ago

We've had better technology in NLP and problem-solving since the mid-1970's -- and many methods that were deemed intractable are trivial now given that compute is on the order of 1 million times more powerful. Systems that use partial-order hierarchical planning, abductive logic programming, bi-directional search, constraint satisfaction, multi-modal interaction, multi-agent conversational modeling based on planning applicable speech-acts, etc., have been replaced with simple state-machines, and in "advanced" cases, simple slot-filling -- and more recently relatively shallow black-box stochastic methods based on deep learning models. Having worked at two companies that offered such systems, I can say that this is due to both not understanding/appreciating what has been done in the past, but also because companies treat these systems like car companies treat features -- they know that an deliver rich features in the future, so the "logic" is to wait to do so, in the hope they can make more money in the long-term, and be able to dribble out new features. To be fair, there is also the issue of reproducibility and predictability -- companies expect everybody's agent to respond the same way given the same input, rather than have variability in agents due to non-determinism or context and learning -- but we can't ask for for flexible "human-like" AI on one hand and also not expect the variation (and occasional misunderstandings) that humans would also suffer.

jbarrs 6y ago

>Even though it goes against the basic premise of human-centered design, users have to train themselves to understand when an intelligent assistant will be useful and when it’s better to avoid using it. I find this is one of the truest things about this article. It's very easy when you first get a new voice assistant to get carried away with the features it has available. Most are either very basic, struggling to take much additional context, or simply just gimmicks/novelties. The majority of tasks can be performed faster through a GUI when you account for the time it takes to brute force the right "natural language" expression to do the thing you need.

aaomidi 6y ago
Well the idea is that you get hands free control of the device. I'm laying in bed and I suddenly realize I need an alarm for tomorrow morning. I don't want to turn my phone on cause that can mess with my sleep. I just tell siri or Google to set an alarm and sleep. Or I'm in the middle of cooking and I want to listen to music but don't want to have to wash my hands first. That's really where the benefit of the assistants lays.

dafoex 6y ago

This would potentially destroy any hands free benefit a virtual assistant would have, back in the day of the secretary (the roll these talking Pringles tubes are trying to emulate) the bossman CEO or whatever would press a button on the office intercom to talk with his secretary. I think virtual assistants would benefit from a simple push-to-talk style input so they don't misinterpret a pause as a stop.