28 ms·

Voice assistants are basically just mainstream non-visual command-lines, and it's unsurprising to me that something that relies heavily on memorization and extr

by Shank 4y ago

Voice assistants are basically just mainstream non-visual command-lines, and it's unsurprising to me that something that relies heavily on memorization and extremely specialized "skills" isn't quite taking off in the way it was imagined. A voice system that can do literally everything one can do with a keyboard and a mouse would be magical, but no system offers that. Instead, it's a guessing game about syntax and semantics, and frequently a source of frustration. There are many failure points: it can "hear" you wrong, it can miss the wake word, it can hear correctly but interpret wrong, miss context clues, or simply be unable to process whatever the request is. In my experience, most normal people either relegate voice commands to ultra-specific tasks, like timers, weather, and music, and that's that. Google and Alexa are relatively good at "trivia" questions, but Siri is a complete failure. All systems have edge cases that make them brittle. I think there's potential here. Cortana was the most promising: an assistant that's integrated into the OS and can change any setting or perform anything on-screen would, again, be really awesome. We just don't have that. I think maybe OS-wide + GPT 4 (or later) might get closer to what we expect, but it's just not great right now. I really want to be able to say something as unstructured as "hey siri, create alarms every 5 minutes starting at 6am tomorrow" or "hey siri, when I get home every day, turn on all of the lights, change my focus to personal, and turn on the news". There /is/ power to-be-had, but nobody has really tapped it.

PhasmaFelis 4y ago
Yeah, the whole idea has a lot of potential that seems like it should be within reach, but somehow it's 2022 and my phone still can't handle "hey Google, play my driving playlist on Spotify."
antupis 4y ago
If I would be in this space I would just build voice assistant to very specific situations where you cannot type like driving, cooking, doing some sport etc. There is lots of potential but big players are kinda trying build generic tool for every situation which is super hard problem.
- dmitriid 4y ago
  You want utility. The big players want a product that can be monetized and milked for revenue.
  - Mistletoe 4y ago
    My Alexa asked me today if I wanted an Avatar theme. No I really do not, Alexa. I was reminded of the article a few days ago how they can’t monetize this well and are somehow losing $10 billion. :)
Eleison23 4y ago
Me and voice assistants are like me on the ballroom dance floor. I loved to take the lessons and learn all sorts of moves and chain them all together and look impressive, but when I got onto the floor with a partner, I just wouldn't know what to do or where to start. I kept to the "basic" steps and maybe a timid little turn once in a while. Maybe it's possible to learn a working vocabulary and know how to command a voice assistant. I know my way around several command lines, but I have no idea what to say to Hey Google.
- Avicebron 4y ago
  it almost sounds like you are describing how it feels to learn a new language. And if that's the case and people need to learn "voice assistant" to communicate with their device effectively, hasn't it utterly failed as a natural language processor? Also I know this is true in other domains as well, obviously there is a common "google-ese" that people learn to narrow down their searches.
sublinear 4y ago
I don't think this is actually reliably possible due to the fact that while grammar does tend to follow patterns sometimes, we're fundamentally dealing with an exponential amount of ways to say things to a voice assistant. In the spirit of the title of this post, someone else also has to say something. If your argument is that this is a "non-visual command line" there's slim hope of the layperson learning a whole secret grammar without even a goddamn man page just to do their menial tasks.
- ianai 4y ago
  I really doubt *nix would have made it so far if the cli were audio based, too. It's a fundamentally slower and lower bandwidth communication channel.
  - zozbot234 4y ago
    *nix was optimized for low-bandwith channels. That's why the command names and options are extremely terse and typically return trivial output on success. OTOH it was assumed that input would be reliable, so there's no confirmation required for potentially dangerous commands. A "*nix for voice" would need to address that, at the very least.
    - ianai 4y ago
      I’d sure be lost if I had to listen to the entirety of a manpage or dmesg output or /var/log/messages read out by voice. Some of those could take hours to read out. Nothing actually trivial about *nix command output. Just sometimes terse.
_dain_ 4y ago
>Voice assistants are basically just mainstream non-visual command-lines, and it's unsurprising to me that something that relies heavily on memorization and extremely specialized "skills" isn't quite taking off in the way it was imagined. This got me thinking. Voice recognition is basically a commodity now .. there are open source AI engines that can do it offline really well. So the recognition part is solved, you can just grab it from your distro's package manager. Now there's just the language part. Thing is, I don't want to speak to my computer using English. Aside from the enormous practical problems in natural language processing you've outlined, I just find the idea creepy[1]. What I want is to unambiguously tell it to do arbitrary things. I.e. use it as an actual computer, not a toy that can do a few tricks. I.e. actually program it. In some kind of Turing complete shell language that is optimized for being spoken aloud. You would speak words into the open source voice recognizer, it writes those to stdout, then an interpreter reads from stdin and executes the instructions. Is there any language like this? What should it look like? And yeah that would take effort to learn to use it right, just like any other programming language; so be it. This would be a hobbyist thing. [1] https://i.kym-cdn.com/photos/images/original/002/054/961/748.jpg https://i.kym-cdn.com/photos/images/original/002/054/961/748...
- viraptor 4y ago
  > So the recognition part is solved If you're using an averaged American voice - maybe. But it's really not solved for everyone. Google assistant can't set the right timer for me 1/10 times. And that's before we get to heavy accent Scots and others.
  - bambax 4y ago
    Obligatory reference: https://www.youtube.com/watch?v=NMS2VnDveP8 https://www.youtube.com/watch?v=NMS2VnDveP8
  - arethuza 4y ago
    Even my "affected Edinburgh accent", as someone once described it, causes no end of trouble with voice recognition.
- 4y ago
qsort 4y ago
The problem isn't voice, it's natural language. Natural language is a fundamentally wrong vehicle to convey information to a computer. It can be useful for some specific tasks, automated Q/A, simple interfaces to databases, stuff where I can't be properly f_ed to remember the syntax or the shortcut like IDE commands. But the idea it can replace formal language is fundamentally and dangerously incorrect. I agree with Dijkstra's quip, we shouldn't regard formal language as a burden, but rather as a privilege.
- darkerside 4y ago
  The problem is both
- ClumsyPilot 4y ago
  > we shouldn't regard formal language as a burden, but rather as a privilege What the hell? Is riding public transport or riding a bike either a burdain or a privilidge? Is Driving a car? I am trying to control shit in my home, it should be neither.
  - duggan 4y ago
    Dijkstra's full essay[1] is a bit more illuminating, but essentially it's about how, for example, developing a system of symbols and formal language around mathematics has allowed "school children [to] learn to do what in earlier days only genius could achieve". 1: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/EWD667.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/E...
    - em500 4y ago
      I think his argument even generalizes to literacy in general. Remember that reading and writing skills don't develop naturally (as opposed to spoken language). They require a large educational investment, and used to be reserved for the wealthy and the privileged.
- 4b11b4 4y ago
  An example backing this is voice assistants that DO work, e.g. Talon voice. But these require defining a language, and then they are very accurate and powerful. I don't see why a voice assistant for the masses couldn't "train it's own users", for example suggesting the language it does expect. But even then, most times people are talking in noisy environments or talk to fast or don't have an understand of how the machine might work. Regardless, who cares. They ruin the audio environment of a home. They're good for setting timers while you're cooking, that's about it.
7952 4y ago
I think the biggest potential is with Microsoft Teams in business. It is ubiquitousness in people's work life, has access to data and has integrated with everything. And adding cortana to calls would be an easy step for people to understand and learn. People would say "cortana share my screen". People would learn phrases from each other.
- happymellon 4y ago
  But teams hasn't figured out how to send text in a coherent way. It's used because companies can cheap out on buying a license for other communication applications, it is fundamentally worse than anything else in any other metric. If voice lets me respond to a message without hunting for the hidden reply because Teams shoves it below the bottom of the screen then it could be a win. Considering UX is so low for Teams I doubt it will.
bambax 4y ago
> I think there's potential here. But how? Even if those interfaces were actually working, it's still extremely inconvenient to talk when you can click. You have to be somewhere where talking out loud doesn't disturb the people around you. That excludes most situations: open space offices, restaurants, coffee shops, public transport, cars with passengers, and most places in the home except maybe the bathroom. And even if you're all alone in a silent place, giving instructions out loud takes more time than configuring a screen, and will always be error prone, because the feedback will always be ambiguous and imprecise. Except maybe if the feedback is on a screen, but then if there's already a screen, why not use it.
- papito 4y ago
  Well, you are not trying to operate heavy machinery with Amazon Echo - hopefully. Voice as a common interface - I agree with all of that, but to me the everyday utility of being able to add something to my shopping list or my TODO list without having to fire up an APP greatly increases my quality of life. That part is magical, but I don't expect a lot more from it.
  - ghaff 4y ago
    I used to use Alexa for my shopping list. I guess over time I came to the conclusion that adding something to a steno pad or my whiteboard was even easier.
- t-sauer 4y ago
  I think the best use cases for voice assistants are when you don’t have free hands. I have two scenarios where I use voice assistants: setting a timer while cooking and changing the music while showering. Both could be done by other means as well but they wouldn‘t be more convenient.
  - jfoster 4y ago
    Asking the time whilst getting ready.
brycehalley 4y ago
Voice assistants have reached the Unhelpful Valley stage. When they were a novelty I recall the excitement of trying new commands and layering in context, after many failures I've been conditioned to now only attempt and expect success with generic queries.
serial_dev 4y ago
> Instead, it's a guessing game about syntax and semantics, and frequently a source of frustration My biggest frustration with Alexa is getting it play the podcasts I want to listen to. Even popular podcasts with English names are hard to get just right for Alexa. The same goes for song titles and bands that are not popular, or they are in other languages. Usually when I want to take a shower, I try to get the podcasts/music to play for 2 minutes, then sigh, give up and just say "Alexa play Britney Spears".
- ghaff 4y ago
  And discoverability. For a long drive I probably want to pick out some specific podcast episodes rather than play whatever. I'm just not a whatever background sound sort of person. The interfaces aren't really good enough to present me with some options with voice control only. So I end up mostly pre-populating a "Car" playlist.
amelius 4y ago
> There /is/ power to-be-had This is not power. This is just first-world problems.
4b11b4 4y ago
Talon voice can do everything a keyboard and mouse offers, plus more (contextual awareness, higher level abstraction). Very powerful in combination with modal editing. I'm not affiliated, just a user. Granted, this is for a specific user base and yes, not in coffee shops.
SheinhardtWigCo 4y ago
> There /is/ power to-be-had, but nobody has really tapped it. This kind of thing can't be built for modern mainstream operating systems because they generally prevent subjugation of the OS components and other programs, even if the user wants that, ostensibly for security reasons. Unlike a human operator, an assistant "app" can only operate within the bounds of APIs defined by the OS vendors and third-party developers. Gone are the days of third-party software that extends the operating system in ways that the overlords couldn't (or wouldn't) dream of.
- sdf4j 4y ago
  That's not entirely true. Accessibility APIs on macOS, for example, would let you control so many aspects of the OS from user land apps given that permissions are granted. But voice assistants are not up to the task.
matthewmacleod 4y ago
"hey siri, when I get home every day, turn on all of the lights, change my focus to personal, and turn on the news" I think the problem with that is that even I, as a human, struggle to know for sure what you want. You want to turn all the lights on in the house? Does that include the lamps in the bedroom? How about new lights that you add later? Or the ones in the garden? It's full of ambiguity. What device do you want to watch the news on? Or did you mean the radio? Do you want this to apply when you get back at 2am one night, meaning your family gets woken up when you turn on all the lights and start playing the news in their bedrooms? I think that's probably why voice interfaces aren't likely to work well for anything beyond direct, specific, well-scoped requests: turn on the lights in the bedroom; turn off the heating at home; roll up the blinds; what's the weather like today; what's the remaining range on my car. They really struggle to deal with anything more complex – not so bad in theory, but really incredibly irritating when they make the wrong decision. If you had some kind of 24-hour live-in assistant (a butler, maybe?), then they probably have the knowledge and intuition to make sensible decisions in response to fairly unstructured requests. But I think we're miles off getting a voice assistant to do it – not because they can't, necessarily, but because if they mess it up at all it's infuriating.
- bombcar 4y ago
  You can do some of this with shortcuts, and then use Siri to trigger the shortcut. But that involves thinking; the magic of Jeeves is that he knows what you want even before you do.
  - bluGill 4y ago
    The problem is there are more different combinations I might want as a shortcut then I have time to program/remember. I can remember something like a dozen commonly used shortcuts. However when 5 years from now I arrive home at 2am (for the first time in several decades, but it will probably happen at some point again in my life) will I remember the correct shortcut - and assuming I do, is it up to date with whatever changes have been made to my house? What about the shortcut for when I need to leave at 3am for some reason. then a different shortcut for when it isn't just me, but my whole family leaving at 3am. An still another for my son having to leave that early. Jeeves can figure it out when I arrive at 2am so I don't need to program it.
Sakos 4y ago
The big issue is that there's no clearly defined interface for users. What commands are possible? Nobody knows. So people default to the most obvious things like setting a timer. Is it possible to setup your own commands and build your own work flows? AFAIK, no. So the tech is essentially dead in the water until companies fundamentally rethink what they're trying to do with voice assistants.
- jasmer 4y ago
  Yup. At the risk of being glib I would say this is 90% of the issue. Or more like 'the big blocking issue' at the moment. Voice can do way more than we know, but we have no idea what it does or how to use it. Standardizing the interface and providing tutorials would possibly change things dramatically. And this goes for the back-end protocols as well. The tech is way, way ahead of the UI and integration. Imagine getting the power of 'git' with no tutorial and not really an understanding of what it does? Good luck with that. 90% of us would be using it in the car to do a lot of things if we really knew how to do it: You: "Siri: Command. Open. Mail. Prompt. Recipients starting with S" Siri: "Sarah, Sue, Sundar" You: "Stop. Command. Message. To: Sunar. Thanks for the note. Stop. Send without Review" Some of this already exists, but it's product specific etc. there needs to be some kind of natural universal interface - or we have to wait until the AI is really, really that good.
eternityforest 4y ago
Timers and reminders alone are enough to make them a pretty nice thing to have though. I don't really want them to be all that much more powerful, because natural language can be imprecise, and... there's just not much I that I want to automate in a home setting beyond some real simple timers for lights and stuff. What if I had a bad day and didn't want to see depressing news? Or what if I came home and was talking on the phone when it turned the news on? True automation as opposed to just telemetry and remote control can easily be annoying more than helpful. I like the idea of automation... but I don't actually... automate anything aside from timers and reminders.
- ghaff 4y ago
  I think that's generally true though playing music is a little more freeform. (And, guess what? Voice assistants tend to be worse at that.) The problem is that you have many many billions of dollars have been sunk into making these devices about more than setting alarms and timers. There's actually been a lot of pretty amazing progress. But it's yet another one of those things that getting to 90% to anyone but techies who want to fiddle with their smarthome stuff or otherwise play with the technology.
  - eternityforest 4y ago
    They might have a sudden increase in usefulness when smarthome stuff is more common, although smart bulbs are a bit of a hassle in most switched outlets, because the switch is usually more convenient. Maybe they'll add an app that lets you browse possible commands so it's more discoverable.
    - ghaff 4y ago
      It's probably true that a well-integrated smarthome would benefit from voice control. But I'd observe that I'm going up to my brother's tomorrow and he has all manner of timers and other WiFi-connected stuff and none of it has any sort of centralized control and that's pretty normal even for people who have a lot of that sort of thing. And, yeah, the only smart light thing I have at home is one thing that doesn't have a controlling light switch and I used X10 for it for years before I got an Alexa.
phkahler 4y ago
>> A voice system that can do literally everything one can do with a keyboard and a mouse would be magical, but no system offers that. And even then, a voice assistant is essentially a user interface, not a product or service. It could be a service if you could reliably say "Alexa, plan my trip to customer X the week of the 30th and send me my itinerary". But for now they are an alternative to a phone UI.
- ghaff 4y ago
  The reality is that even a human personal assistant can rapidly devolve to being more of a hindrance than a help if they're not very good once you get beyond simple mechanical tasks. Even with all the knowledge about the world that most adults carry around in their heads. Yes, a poor human assistant can fall down in other ways such as forgetting to do something--but they have a lot of context. This seems a really high bar for voice assistants aspiring to do much more than set alarms or turn the odd light etc. on or off.
  - bluGill 4y ago
    These days few people have personal secretaries, but back when they were common they really were personal - once you got a personal secretary she (nearly always she, I feel like we should acknowledge sexism even though it is irrelevant my point) would follow you (nearly always male), as you moved job to job and up the ladder. She went with the you because once you spent a few years training her to how you worked, a new secretary would greatly limit your effectiveness. These days a large part of what people relied on secretaries for a computer can do faster, so only at the highest levels do you see them. There are still secretaries at the low levels, but not nearly as many, and they are not doing the same tasks.
    - ghaff 4y ago
      That's pretty much it. We call them executive admins these days where they exist. And, yeah, assistants shared with a bunch of other people--as with travel agents in general--aren't really all that useful. If I'm mostly just giving fairly mechanical instructions to execute, it's probably easier for me to go online and figure out the options myself. A secretary made a lot more sense when you dictated memos for inter-office mail and retrieving information often involved making multiple phone calls.
PurpleRamen 4y ago
The potential would be there, if they would focus on the assistant-part, and take the voice just as a mean to interact with the assistant, besides other means like clicking, typing, showing complex information on a screen, etc. Voice alone sucks, it's just too limited to be useful on a grand scale. Similarly, command lines suck too. The shell in general has the same problems that Voice assistants have, just that they have more value and had decades to mature into something actually useful. And toady we have unix-shells which reduce the problematic parts by many levels, and still receive constant improvements. This is missing for voice assistants, because unix-shells are growing and improving in an open space, where everyone can add their own things. This is not happening in big tech.
mc32 4y ago
To me what’s interesting is that MS smelled that it was a problem a while ago and pulled the plug before it ate a hole in their wallet but Amazon and Google keep plugging along ploughing money into a bottomless pit. Apple has a different play and looks like they are controlling their losses there quite well and may act as a slight loss leader for other products.
- foobarian 4y ago
  I can't fathom how they managed to spend so much on it, though. The product has been around for quite a while, as well, so it's not some initial ramp-up cost. $3B/quarter $10B/year? Wow. Edit: Maybe things like this happen because there are various nerds who lead these products and are good at talking the businesspeople into funding it. Maybe this was only possible at the big tech growth stage while business wasn't that good at telling the value proposition. So end result, lots more engineers get paid which is great in my book :-)
iquerno 4y ago
I would think that a good command-line is one that responds to me within milliseconds on a crapbox i386 machine, and I can COMMAND it what to do. A good command-line is not a binary blob that cannot parse simple instructions correctly. At the same time, siri seems to be getting slower and fatter every iteration so perhaps it is becoming more human ;)
sokoloff 4y ago
> "hey siri, create alarms every 5 minutes starting at 6am tomorrow" “OK, I’ve created an infinite number of alarms, every five minutes, starting at 6 AM tomorrow!” (As a native English speaker, I'm not sure what specific outcome you want to happen from that request. That's the one that makes the most sense.)
- ghaff 4y ago
  As a native English speaker, that seems a profoundly odd request but that is what you asked for. And you now have me wondering how open-ended calendar requests are actually implemented given that they can't literally have entries out to infinity. (I assume they go out some finite period and some background process periodically re-populates future entries.)
  - mercutio2 4y ago
    A recurrence rule is added to a start event, then an occurrence cache is either generated on the fly for periods of interest, or, yes, a rolling cache a year or two in the future is maintained and updated daily.
    - ghaff 4y ago
      Perhaps trivial, but actually seems like an interesting question given you have to potentially tradeoff RPCs for routine queries (and the number of database records) vs. being wrong for the random "Am I free on this day three years from now?" query. Of course, the answer may be that, in general, the differences don't really matter.
spookthesunset 4y ago
To me the hardest problem is simply remembering what every light on my network is named. Did I call the light next to my desk “desk light” or did I call it “office light”? If I don’t get the name exactly right, I cannot control the light. Multiply that by every other light in the house and it becomes a lot to remember. I have probably 15 lights controlled by Alexa and I can only remember the name of like three of them. Thus most of the time it is just “Alexa turn on the lights” so it can turn everything on in a room. If these voice assistants were smarter about “alternative” names for every device it might be easier to use. But as it stands, it’s kind of a pain because the way you phrase each request is so unforgiving… Oh yeah, and god help you if your device name is similar to your room name. If your room is “office” (or did I name it “the office”?) and your light is “office light” Alexa is gonna have a bad time figuring the two apart. I have no clue how to fix this… PS: this is why I question steering wheel free self driving cars. How will we tell these things exactly where to go when we cannot even reliably tell our voice assistants exactly what light to turn on?
bistable 4y ago
I think you're identifying some of the right problems here. All voice assistants are based on turn-taking, and when the VoiceAI hits one of those failure points and just comes back with "I didn't get that" it leaves the user in a frustrating state trying to debug what's wrong. I work at SoundHound where we've been worried about these issues. (I'm going to plug our recent work...) Our new approach is to do natural language understanding in real-time instead of at the utterance (turn) taking level. That way we can give the user constant feedback in real-time. In the case of a screen that means the user sees right away that they are understood, and if not, a better hint of what went wrong. For example a likely mistake is an ASR mistranscription for a word or two. We still need to prove this is a better paradigm for VoiceAI in products that people can try for themselves, and are working towards that goal. I hope that voice interfaces that were clunky with turn-taking will finally be more naturally usable with real-time NLU. https://www.youtube.com/watch?v=5WLYH1qHfq8 https://www.youtube.com/watch?v=5WLYH1qHfq8
bogdanstanciu 4y ago
I think these assistants just need to give the user a way to edit interpretations. A 'debug' area that lets you ask a command, see what was interpreted - and immediately edit or click "that's not what I wanted". But not an afterthought and not a cumbersome process like setting up an automation that is triggered by specific commands. Imagine telling your voice assistant "You're wrong, as usual" and instead of it giving you the boiler plate "I'm sorry ", it actually offered a way to improve itself.
freeone3000 4y ago
Your queries continue to be money-sinks -- even in your ideal case, you aren't buying anything! This query costs them money but earns them nothing. This is useless.
gernb 4y ago
> an assistant that's integrated into the OS and can change any setting That sounds like a security nightmare. Someone walks by and starts changing your system settings? No thank you
gspencley 4y ago
I might be in the minority, but I also don't want to add things to my life that make my environment noisier or that require me or others living with me to speak more. As much of a Star Trek fan as I am, I never found "The Computer" to be appealing, and always thought of it more as an artistic device. It's a lot easier to communicate a character's intent / action if they are vocalizing it for performance. Even in scenes where they are "typing" something into the computer, they will inevitably be communicating to the captain or another character what they are doing. In practical reality these interfaces feel, to me, as extremely inefficient. As someone who doesn't particularly like to speak, and prefers silent environments, these interfaces require more energy from me to use. Unless they are serving someone who has a physical impairment then I don't see what problems exist that these solve, but I can identify lots of problems that they introduce (not only noise but privacy / security vulnerabilities etc.) Personal preference.
1MachineElf 4y ago
Another pitfall of most voice assistants is that they are really designed first with the corporation in mind rather than the user. Most are proxies for surveillance, advertising, or are just steering consumers back to a preferred set of walled-garden services.
sliken 4y ago
I tried Amazon's Alexa, the top end model with a display. Often it would taunt you about new/interesting things on the screen, but I could never get them to work. I'd had to memorize things to get even the basics working. Ended up unplugging it. However Google's Assistant in comparison worked great, no memorization, and very useful. Sure time, weather, set timers, and alarms worked great with a very flexible set of natural language queries. Even more complex things like what will be the temperature tomorrow at 10pm, simple calculations and unit conversions. But also things like IMDB like queries about directors, actors, which movies someone was in, etc generally worked well. It seemed to really understand things, not just "A web search returned ...". Even more complex things like the wheelbase of a 2004 WRX would return an answer, not a search result. With all that said I'm looking for a non-cloud/on site solution, even if it requires more work, most recently noticed https://github.com/rhasspy/rhasspy https://github.com/rhasspy/rhasspy
Razengan 4y ago
This timeline is such a mishmash of mediocrity. Voice assistants could have been a vibrant ecosystem of different personalities, like say buying a Darth Vader voice pack or having your computer sound like a snooty English butler.. There's a great little game series called Megaman Battle Network (Rockman.exe in Japan) which diverges from the mainline by showing an alternate universe where scientists focused on AI instead of robotics, resulting in a world where "Navis" are ubiquitous. I wonder, what if our early software engineers focused on bringing natural voice control to CLIs, before perfecting GUIs first?

by Shank 4y ago

PhasmaFelis 4y ago

Yeah, the whole idea has a lot of potential that seems like it should be within reach, but somehow it's 2022 and my phone still can't handle "hey Google, play my driving playlist on Spotify."

antupis 4y ago

If I would be in this space I would just build voice assistant to very specific situations where you cannot type like driving, cooking, doing some sport etc. There is lots of potential but big players are kinda trying build generic tool for every situation which is super hard problem.

dmitriid 4y ago
You want utility. The big players want a product that can be monetized and milked for revenue.
- Mistletoe 4y ago
  My Alexa asked me today if I wanted an Avatar theme. No I really do not, Alexa. I was reminded of the article a few days ago how they can’t monetize this well and are somehow losing $10 billion. :)

Eleison23 4y ago

Me and voice assistants are like me on the ballroom dance floor. I loved to take the lessons and learn all sorts of moves and chain them all together and look impressive, but when I got onto the floor with a partner, I just wouldn't know what to do or where to start. I kept to the "basic" steps and maybe a timid little turn once in a while. Maybe it's possible to learn a working vocabulary and know how to command a voice assistant. I know my way around several command lines, but I have no idea what to say to Hey Google.

Avicebron 4y ago
it almost sounds like you are describing how it feels to learn a new language. And if that's the case and people need to learn "voice assistant" to communicate with their device effectively, hasn't it utterly failed as a natural language processor? Also I know this is true in other domains as well, obviously there is a common "google-ese" that people learn to narrow down their searches.

sublinear 4y ago

I don't think this is actually reliably possible due to the fact that while grammar does tend to follow patterns sometimes, we're fundamentally dealing with an exponential amount of ways to say things to a voice assistant. In the spirit of the title of this post, someone else also has to say something. If your argument is that this is a "non-visual command line" there's slim hope of the layperson learning a whole secret grammar without even a goddamn man page just to do their menial tasks.

ianai 4y ago
I really doubt *nix would have made it so far if the cli were audio based, too. It's a fundamentally slower and lower bandwidth communication channel.
- zozbot234 4y ago
  *nix was optimized for low-bandwith channels. That's why the command names and options are extremely terse and typically return trivial output on success. OTOH it was assumed that input would be reliable, so there's no confirmation required for potentially dangerous commands. A "*nix for voice" would need to address that, at the very least.
  - ianai 4y ago
    I’d sure be lost if I had to listen to the entirety of a manpage or dmesg output or /var/log/messages read out by voice. Some of those could take hours to read out. Nothing actually trivial about *nix command output. Just sometimes terse.

_dain_ 4y ago

>Voice assistants are basically just mainstream non-visual command-lines, and it's unsurprising to me that something that relies heavily on memorization and extremely specialized "skills" isn't quite taking off in the way it was imagined. This got me thinking. Voice recognition is basically a commodity now .. there are open source AI engines that can do it offline really well. So the recognition part is solved, you can just grab it from your distro's package manager. Now there's just the language part. Thing is, I don't want to speak to my computer using English. Aside from the enormous practical problems in natural language processing you've outlined, I just find the idea creepy[1]. What I want is to unambiguously tell it to do arbitrary things. I.e. use it as an actual computer, not a toy that can do a few tricks. I.e. actually program it. In some kind of Turing complete shell language that is optimized for being spoken aloud. You would speak words into the open source voice recognizer, it writes those to stdout, then an interpreter reads from stdin and executes the instructions. Is there any language like this? What should it look like? And yeah that would take effort to learn to use it right, just like any other programming language; so be it. This would be a hobbyist thing. [1] https://i.kym-cdn.com/photos/images/original/002/054/961/748.jpg https://i.kym-cdn.com/photos/images/original/002/054/961/748...

viraptor 4y ago
> So the recognition part is solved If you're using an averaged American voice - maybe. But it's really not solved for everyone. Google assistant can't set the right timer for me 1/10 times. And that's before we get to heavy accent Scots and others.
- bambax 4y ago
  Obligatory reference: https://www.youtube.com/watch?v=NMS2VnDveP8 https://www.youtube.com/watch?v=NMS2VnDveP8
- arethuza 4y ago
  Even my "affected Edinburgh accent", as someone once described it, causes no end of trouble with voice recognition.
4y ago

qsort 4y ago

The problem isn't voice, it's natural language. Natural language is a fundamentally wrong vehicle to convey information to a computer. It can be useful for some specific tasks, automated Q/A, simple interfaces to databases, stuff where I can't be properly f_ed to remember the syntax or the shortcut like IDE commands. But the idea it can replace formal language is fundamentally and dangerously incorrect. I agree with Dijkstra's quip, we shouldn't regard formal language as a burden, but rather as a privilege.

darkerside 4y ago
The problem is both
ClumsyPilot 4y ago
> we shouldn't regard formal language as a burden, but rather as a privilege What the hell? Is riding public transport or riding a bike either a burdain or a privilidge? Is Driving a car? I am trying to control shit in my home, it should be neither.
- duggan 4y ago
  Dijkstra's full essay[1] is a bit more illuminating, but essentially it's about how, for example, developing a system of symbols and formal language around mathematics has allowed "school children [to] learn to do what in earlier days only genius could achieve". 1: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/EWD667.html https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/E...
  - em500 4y ago
    I think his argument even generalizes to literacy in general. Remember that reading and writing skills don't develop naturally (as opposed to spoken language). They require a large educational investment, and used to be reserved for the wealthy and the privileged.
4b11b4 4y ago
An example backing this is voice assistants that DO work, e.g. Talon voice. But these require defining a language, and then they are very accurate and powerful. I don't see why a voice assistant for the masses couldn't "train it's own users", for example suggesting the language it does expect. But even then, most times people are talking in noisy environments or talk to fast or don't have an understand of how the machine might work. Regardless, who cares. They ruin the audio environment of a home. They're good for setting timers while you're cooking, that's about it.

7952 4y ago

I think the biggest potential is with Microsoft Teams in business. It is ubiquitousness in people's work life, has access to data and has integrated with everything. And adding cortana to calls would be an easy step for people to understand and learn. People would say "cortana share my screen". People would learn phrases from each other.

happymellon 4y ago
But teams hasn't figured out how to send text in a coherent way. It's used because companies can cheap out on buying a license for other communication applications, it is fundamentally worse than anything else in any other metric. If voice lets me respond to a message without hunting for the hidden reply because Teams shoves it below the bottom of the screen then it could be a win. Considering UX is so low for Teams I doubt it will.

bambax 4y ago

> I think there's potential here. But how? Even if those interfaces were actually working, it's still extremely inconvenient to talk when you can click. You have to be somewhere where talking out loud doesn't disturb the people around you. That excludes most situations: open space offices, restaurants, coffee shops, public transport, cars with passengers, and most places in the home except maybe the bathroom. And even if you're all alone in a silent place, giving instructions out loud takes more time than configuring a screen, and will always be error prone, because the feedback will always be ambiguous and imprecise. Except maybe if the feedback is on a screen, but then if there's already a screen, why not use it.

papito 4y ago
Well, you are not trying to operate heavy machinery with Amazon Echo - hopefully. Voice as a common interface - I agree with all of that, but to me the everyday utility of being able to add something to my shopping list or my TODO list without having to fire up an APP greatly increases my quality of life. That part is magical, but I don't expect a lot more from it.
- ghaff 4y ago
  I used to use Alexa for my shopping list. I guess over time I came to the conclusion that adding something to a steno pad or my whiteboard was even easier.
t-sauer 4y ago
I think the best use cases for voice assistants are when you don’t have free hands. I have two scenarios where I use voice assistants: setting a timer while cooking and changing the music while showering. Both could be done by other means as well but they wouldn‘t be more convenient.
- jfoster 4y ago
  Asking the time whilst getting ready.

brycehalley 4y ago

Voice assistants have reached the Unhelpful Valley stage. When they were a novelty I recall the excitement of trying new commands and layering in context, after many failures I've been conditioned to now only attempt and expect success with generic queries.

serial_dev 4y ago

> Instead, it's a guessing game about syntax and semantics, and frequently a source of frustration My biggest frustration with Alexa is getting it play the podcasts I want to listen to. Even popular podcasts with English names are hard to get just right for Alexa. The same goes for song titles and bands that are not popular, or they are in other languages. Usually when I want to take a shower, I try to get the podcasts/music to play for 2 minutes, then sigh, give up and just say "Alexa play Britney Spears".

ghaff 4y ago
And discoverability. For a long drive I probably want to pick out some specific podcast episodes rather than play whatever. I'm just not a whatever background sound sort of person. The interfaces aren't really good enough to present me with some options with voice control only. So I end up mostly pre-populating a "Car" playlist.

amelius 4y ago

> There /is/ power to-be-had This is not power. This is just first-world problems.

4b11b4 4y ago

Talon voice can do everything a keyboard and mouse offers, plus more (contextual awareness, higher level abstraction). Very powerful in combination with modal editing. I'm not affiliated, just a user. Granted, this is for a specific user base and yes, not in coffee shops.

SheinhardtWigCo 4y ago

> There /is/ power to-be-had, but nobody has really tapped it. This kind of thing can't be built for modern mainstream operating systems because they generally prevent subjugation of the OS components and other programs, even if the user wants that, ostensibly for security reasons. Unlike a human operator, an assistant "app" can only operate within the bounds of APIs defined by the OS vendors and third-party developers. Gone are the days of third-party software that extends the operating system in ways that the overlords couldn't (or wouldn't) dream of.

sdf4j 4y ago
That's not entirely true. Accessibility APIs on macOS, for example, would let you control so many aspects of the OS from user land apps given that permissions are granted. But voice assistants are not up to the task.

matthewmacleod 4y ago

"hey siri, when I get home every day, turn on all of the lights, change my focus to personal, and turn on the news" I think the problem with that is that even I, as a human, struggle to know for sure what you want. You want to turn all the lights on in the house? Does that include the lamps in the bedroom? How about new lights that you add later? Or the ones in the garden? It's full of ambiguity. What device do you want to watch the news on? Or did you mean the radio? Do you want this to apply when you get back at 2am one night, meaning your family gets woken up when you turn on all the lights and start playing the news in their bedrooms? I think that's probably why voice interfaces aren't likely to work well for anything beyond direct, specific, well-scoped requests: turn on the lights in the bedroom; turn off the heating at home; roll up the blinds; what's the weather like today; what's the remaining range on my car. They really struggle to deal with anything more complex – not so bad in theory, but really incredibly irritating when they make the wrong decision. If you had some kind of 24-hour live-in assistant (a butler, maybe?), then they probably have the knowledge and intuition to make sensible decisions in response to fairly unstructured requests. But I think we're miles off getting a voice assistant to do it – not because they can't, necessarily, but because if they mess it up at all it's infuriating.

bombcar 4y ago
You can do some of this with shortcuts, and then use Siri to trigger the shortcut. But that involves thinking; the magic of Jeeves is that he knows what you want even before you do.
- bluGill 4y ago
  The problem is there are more different combinations I might want as a shortcut then I have time to program/remember. I can remember something like a dozen commonly used shortcuts. However when 5 years from now I arrive home at 2am (for the first time in several decades, but it will probably happen at some point again in my life) will I remember the correct shortcut - and assuming I do, is it up to date with whatever changes have been made to my house? What about the shortcut for when I need to leave at 3am for some reason. then a different shortcut for when it isn't just me, but my whole family leaving at 3am. An still another for my son having to leave that early. Jeeves can figure it out when I arrive at 2am so I don't need to program it.

Sakos 4y ago

The big issue is that there's no clearly defined interface for users. What commands are possible? Nobody knows. So people default to the most obvious things like setting a timer. Is it possible to setup your own commands and build your own work flows? AFAIK, no. So the tech is essentially dead in the water until companies fundamentally rethink what they're trying to do with voice assistants.

jasmer 4y ago
Yup. At the risk of being glib I would say this is 90% of the issue. Or more like 'the big blocking issue' at the moment. Voice can do way more than we know, but we have no idea what it does or how to use it. Standardizing the interface and providing tutorials would possibly change things dramatically. And this goes for the back-end protocols as well. The tech is way, way ahead of the UI and integration. Imagine getting the power of 'git' with no tutorial and not really an understanding of what it does? Good luck with that. 90% of us would be using it in the car to do a lot of things if we really knew how to do it: You: "Siri: Command. Open. Mail. Prompt. Recipients starting with S" Siri: "Sarah, Sue, Sundar" You: "Stop. Command. Message. To: Sunar. Thanks for the note. Stop. Send without Review" Some of this already exists, but it's product specific etc. there needs to be some kind of natural universal interface - or we have to wait until the AI is really, really that good.

eternityforest 4y ago

Timers and reminders alone are enough to make them a pretty nice thing to have though. I don't really want them to be all that much more powerful, because natural language can be imprecise, and... there's just not much I that I want to automate in a home setting beyond some real simple timers for lights and stuff. What if I had a bad day and didn't want to see depressing news? Or what if I came home and was talking on the phone when it turned the news on? True automation as opposed to just telemetry and remote control can easily be annoying more than helpful. I like the idea of automation... but I don't actually... automate anything aside from timers and reminders.

ghaff 4y ago
I think that's generally true though playing music is a little more freeform. (And, guess what? Voice assistants tend to be worse at that.) The problem is that you have many many billions of dollars have been sunk into making these devices about more than setting alarms and timers. There's actually been a lot of pretty amazing progress. But it's yet another one of those things that getting to 90% to anyone but techies who want to fiddle with their smarthome stuff or otherwise play with the technology.
- eternityforest 4y ago
  They might have a sudden increase in usefulness when smarthome stuff is more common, although smart bulbs are a bit of a hassle in most switched outlets, because the switch is usually more convenient. Maybe they'll add an app that lets you browse possible commands so it's more discoverable.
  - ghaff 4y ago
    It's probably true that a well-integrated smarthome would benefit from voice control. But I'd observe that I'm going up to my brother's tomorrow and he has all manner of timers and other WiFi-connected stuff and none of it has any sort of centralized control and that's pretty normal even for people who have a lot of that sort of thing. And, yeah, the only smart light thing I have at home is one thing that doesn't have a controlling light switch and I used X10 for it for years before I got an Alexa.

phkahler 4y ago

>> A voice system that can do literally everything one can do with a keyboard and a mouse would be magical, but no system offers that. And even then, a voice assistant is essentially a user interface, not a product or service. It could be a service if you could reliably say "Alexa, plan my trip to customer X the week of the 30th and send me my itinerary". But for now they are an alternative to a phone UI.

ghaff 4y ago
The reality is that even a human personal assistant can rapidly devolve to being more of a hindrance than a help if they're not very good once you get beyond simple mechanical tasks. Even with all the knowledge about the world that most adults carry around in their heads. Yes, a poor human assistant can fall down in other ways such as forgetting to do something--but they have a lot of context. This seems a really high bar for voice assistants aspiring to do much more than set alarms or turn the odd light etc. on or off.
- bluGill 4y ago
  These days few people have personal secretaries, but back when they were common they really were personal - once you got a personal secretary she (nearly always she, I feel like we should acknowledge sexism even though it is irrelevant my point) would follow you (nearly always male), as you moved job to job and up the ladder. She went with the you because once you spent a few years training her to how you worked, a new secretary would greatly limit your effectiveness. These days a large part of what people relied on secretaries for a computer can do faster, so only at the highest levels do you see them. There are still secretaries at the low levels, but not nearly as many, and they are not doing the same tasks.
  - ghaff 4y ago
    That's pretty much it. We call them executive admins these days where they exist. And, yeah, assistants shared with a bunch of other people--as with travel agents in general--aren't really all that useful. If I'm mostly just giving fairly mechanical instructions to execute, it's probably easier for me to go online and figure out the options myself. A secretary made a lot more sense when you dictated memos for inter-office mail and retrieving information often involved making multiple phone calls.

PurpleRamen 4y ago

The potential would be there, if they would focus on the assistant-part, and take the voice just as a mean to interact with the assistant, besides other means like clicking, typing, showing complex information on a screen, etc. Voice alone sucks, it's just too limited to be useful on a grand scale. Similarly, command lines suck too. The shell in general has the same problems that Voice assistants have, just that they have more value and had decades to mature into something actually useful. And toady we have unix-shells which reduce the problematic parts by many levels, and still receive constant improvements. This is missing for voice assistants, because unix-shells are growing and improving in an open space, where everyone can add their own things. This is not happening in big tech.

mc32 4y ago

To me what’s interesting is that MS smelled that it was a problem a while ago and pulled the plug before it ate a hole in their wallet but Amazon and Google keep plugging along ploughing money into a bottomless pit. Apple has a different play and looks like they are controlling their losses there quite well and may act as a slight loss leader for other products.

foobarian 4y ago
I can't fathom how they managed to spend so much on it, though. The product has been around for quite a while, as well, so it's not some initial ramp-up cost. $3B/quarter $10B/year? Wow. Edit: Maybe things like this happen because there are various nerds who lead these products and are good at talking the businesspeople into funding it. Maybe this was only possible at the big tech growth stage while business wasn't that good at telling the value proposition. So end result, lots more engineers get paid which is great in my book :-)

iquerno 4y ago

I would think that a good command-line is one that responds to me within milliseconds on a crapbox i386 machine, and I can COMMAND it what to do. A good command-line is not a binary blob that cannot parse simple instructions correctly. At the same time, siri seems to be getting slower and fatter every iteration so perhaps it is becoming more human ;)

sokoloff 4y ago

> "hey siri, create alarms every 5 minutes starting at 6am tomorrow" “OK, I’ve created an infinite number of alarms, every five minutes, starting at 6 AM tomorrow!” (As a native English speaker, I'm not sure what specific outcome you want to happen from that request. That's the one that makes the most sense.)

ghaff 4y ago
As a native English speaker, that seems a profoundly odd request but that is what you asked for. And you now have me wondering how open-ended calendar requests are actually implemented given that they can't literally have entries out to infinity. (I assume they go out some finite period and some background process periodically re-populates future entries.)
- mercutio2 4y ago
  A recurrence rule is added to a start event, then an occurrence cache is either generated on the fly for periods of interest, or, yes, a rolling cache a year or two in the future is maintained and updated daily.
  - ghaff 4y ago
    Perhaps trivial, but actually seems like an interesting question given you have to potentially tradeoff RPCs for routine queries (and the number of database records) vs. being wrong for the random "Am I free on this day three years from now?" query. Of course, the answer may be that, in general, the differences don't really matter.

spookthesunset 4y ago

To me the hardest problem is simply remembering what every light on my network is named. Did I call the light next to my desk “desk light” or did I call it “office light”? If I don’t get the name exactly right, I cannot control the light. Multiply that by every other light in the house and it becomes a lot to remember. I have probably 15 lights controlled by Alexa and I can only remember the name of like three of them. Thus most of the time it is just “Alexa turn on the lights” so it can turn everything on in a room. If these voice assistants were smarter about “alternative” names for every device it might be easier to use. But as it stands, it’s kind of a pain because the way you phrase each request is so unforgiving… Oh yeah, and god help you if your device name is similar to your room name. If your room is “office” (or did I name it “the office”?) and your light is “office light” Alexa is gonna have a bad time figuring the two apart. I have no clue how to fix this… PS: this is why I question steering wheel free self driving cars. How will we tell these things exactly where to go when we cannot even reliably tell our voice assistants exactly what light to turn on?

bistable 4y ago

I think you're identifying some of the right problems here. All voice assistants are based on turn-taking, and when the VoiceAI hits one of those failure points and just comes back with "I didn't get that" it leaves the user in a frustrating state trying to debug what's wrong. I work at SoundHound where we've been worried about these issues. (I'm going to plug our recent work...) Our new approach is to do natural language understanding in real-time instead of at the utterance (turn) taking level. That way we can give the user constant feedback in real-time. In the case of a screen that means the user sees right away that they are understood, and if not, a better hint of what went wrong. For example a likely mistake is an ASR mistranscription for a word or two. We still need to prove this is a better paradigm for VoiceAI in products that people can try for themselves, and are working towards that goal. I hope that voice interfaces that were clunky with turn-taking will finally be more naturally usable with real-time NLU. https://www.youtube.com/watch?v=5WLYH1qHfq8 https://www.youtube.com/watch?v=5WLYH1qHfq8

bogdanstanciu 4y ago

I think these assistants just need to give the user a way to edit interpretations. A 'debug' area that lets you ask a command, see what was interpreted - and immediately edit or click "that's not what I wanted". But not an afterthought and not a cumbersome process like setting up an automation that is triggered by specific commands. Imagine telling your voice assistant "You're wrong, as usual" and instead of it giving you the boiler plate "I'm sorry ", it actually offered a way to improve itself.

freeone3000 4y ago

Your queries continue to be money-sinks -- even in your ideal case, you aren't buying anything! This query costs them money but earns them nothing. This is useless.

gernb 4y ago

> an assistant that's integrated into the OS and can change any setting That sounds like a security nightmare. Someone walks by and starts changing your system settings? No thank you

gspencley 4y ago

I might be in the minority, but I also don't want to add things to my life that make my environment noisier or that require me or others living with me to speak more. As much of a Star Trek fan as I am, I never found "The Computer" to be appealing, and always thought of it more as an artistic device. It's a lot easier to communicate a character's intent / action if they are vocalizing it for performance. Even in scenes where they are "typing" something into the computer, they will inevitably be communicating to the captain or another character what they are doing. In practical reality these interfaces feel, to me, as extremely inefficient. As someone who doesn't particularly like to speak, and prefers silent environments, these interfaces require more energy from me to use. Unless they are serving someone who has a physical impairment then I don't see what problems exist that these solve, but I can identify lots of problems that they introduce (not only noise but privacy / security vulnerabilities etc.) Personal preference.

1MachineElf 4y ago

Another pitfall of most voice assistants is that they are really designed first with the corporation in mind rather than the user. Most are proxies for surveillance, advertising, or are just steering consumers back to a preferred set of walled-garden services.

sliken 4y ago

I tried Amazon's Alexa, the top end model with a display. Often it would taunt you about new/interesting things on the screen, but I could never get them to work. I'd had to memorize things to get even the basics working. Ended up unplugging it. However Google's Assistant in comparison worked great, no memorization, and very useful. Sure time, weather, set timers, and alarms worked great with a very flexible set of natural language queries. Even more complex things like what will be the temperature tomorrow at 10pm, simple calculations and unit conversions. But also things like IMDB like queries about directors, actors, which movies someone was in, etc generally worked well. It seemed to really understand things, not just "A web search returned ...". Even more complex things like the wheelbase of a 2004 WRX would return an answer, not a search result. With all that said I'm looking for a non-cloud/on site solution, even if it requires more work, most recently noticed https://github.com/rhasspy/rhasspy https://github.com/rhasspy/rhasspy

Razengan 4y ago

This timeline is such a mishmash of mediocrity. Voice assistants could have been a vibrant ecosystem of different personalities, like say buying a Darth Vader voice pack or having your computer sound like a snooty English butler.. There's a great little game series called Megaman Battle Network (Rockman.exe in Japan) which diverges from the mainline by showing an alternate universe where scientists focused on AI instead of robotics, resulting in a world where "Navis" are ubiquitous. I wonder, what if our early software engineers focused on bringing natural voice control to CLIs, before perfecting GUIs first?