12 ms·

I ask 100 information questions to four digital assistants

298 points by forgot-my-pw 9y ago

elicash 9y ago
I got 22 out of the "40 verbose Assistant questions" correct. Not bad! I beat them all (as a percentage). Maybe not a bad idea for a gameshow.
- chris_overseas 9y ago
  That's similar to how The Chase[1] works, except contestants go head to head with a professional quiz master instead of a digital assistant. [1] https://en.wikipedia.org/wiki/The_Chase_(UK_game_show) https://en.wikipedia.org/wiki/The_Chase_(UK_game_show)
- forgot-my-pw 9y ago
  Makes me wonder if Watson will do a better contextual search and single result answer. IBM should release a voice assistant too.
51Cards 9y ago
I notice some different answers on my devices. For example on the "Are tomatoes vegetables?" question my Google Home states that they are definitely a fruit. (quoting Oxford Dictionary) Edit: And "What's the height in meters of the Empire State Building?" gets me "381 meters, 443 meters to tip"
IanCal 9y ago
I tried the conversational weather one on google assistant on my phone. "Should I take an umbrella tomorrow?" "No..." and shows me tomorrows forecast. "What about the day after?" "No..." and shows a forecast that when I look more closely at I notice is for today. Neither of these also spotted that I'm heading to another city tomorrow, which is in my calendar. If I changed it to "do I need to take an umbrella for my trip tomorrow" it just searches google and gives me a search result suggesting I take a small folding umbrella... for a trip to Thailand.
- visarga 9y ago
  It might be so, but remember that experts were predicting computer will beat top humans in Go in 10 years. Maybe next month there will be a breakthrough with NLP. The amount of research and compute going into this problem is amazing, and we don't know what's around the corner. Also, it might be possible to create a much better assistant today, but it would be too expensive to offer to the public for free. What if it requires 100 TPUs to run?
  - IanCal 9y ago
    The thing for me in that example is it's not a particularly complex set of options. It knows I'm asking for the weather, it knows that it should give me the weather in a particular location, and google knows where I'll be tomorrow. The other main mistake was (despite it getting the conversational part right) thinking "the day after" means today. I don't think these things require TPUs or new NLP.
    - visarga 9y ago
      Yes, they managed assistants badly - Google, Apple and Amazon. They could have added more APIs, recognized many more patterns of user requests, and it's just plain old programming, so what's keeping them? For example, Google Assistant doesn't play nice with YouTube. I guess it isn't in Google's interest to have a free agent software serve a free music site when the same service makes money for Amazon (Alexa). They wanted to make their agent profitable, so didn't let it be extended for free. Just my guess.
      - johnsmith21006 9y ago
        Google Home with YouTube is amazing. I say play Gwen Bottle and get Gwen Stefanni sing Message in a bottle. Or say play One Love Peas and get Blach Eye Peas ad One Love concert. What are you talking about?
  - marcosdumay 9y ago
    This one problem of NLP is kinda solved¹ for more than 10 years already. What you are seeing here is a much simpler CS problem, but much harder social problem. It is "why can't my applications talk to each other?". I really doubt it will be solved in 10 years. 1 - Humans don't have a perfect solution for it, and machines are still worse, but not that worse.
codekilla 9y ago
more people are starting to appreciate that AGIish stuff is actually really, really hard.
- snarf21 9y ago
  Agreed, determining human intent is really, really, really hard. There are some many context clues we use in everyday life that the voice interface will never have access to like where am I standing, what's my expression, etc.
- ghaff 9y ago
  The ability to function as a virtual assistant, even at the level of a not-so-sharp intern [1], would be a killer app. Give it some parameters for a trip you're taking. It comes back with some options and follow-up questions. We are a long way from that point. Even that not-so-sharp intern has a huge amount of internalized knowledge about general preferences, cities, airports, etc. and probably knows questions to ask to narrow things down. I strongly suspect there are other domains where a lot of people are assuming we're 90% there and we're not. [1] Not to insult interns or any other group. I just mean you don't need to be at experienced executive assistant level to be really useful.
- forgot-my-pw 9y ago
  Makes me wonder if Watson will do a better contextual search and single result answer. IBM should release a voice assistant too.
dahart 9y ago
What's interesting to me reading this is the expectation that voice search should do things that text search currently doesn't do. If I ask Google about Bill Murray running for president, I presumably am looking for articles that mention the words "Bill", "Murray", and "President". I would expect to return articles that best match the query, and I would never expect Google to be able to tell me the difference between real and fake articles. For a server to answer the author's question it has to understand the question so well it can change the question into "What is the list of people that have run for U.S. president?" and then check if Bill Murray is in that list. That's a tall order. We are moving past expecting the most word matches into expecting the server to understand what we really want. Voice search is more like the "I'm feeling lucky" button, because it takes longer; you only have time for one answer and the first answer has to be right. It comes without the expectation that you're lucky if the answer happens to be right, now we need the first result to be the rightest result there is. So the glass is half empty. I personally prefer to see it half-full, but it's also true and the critique is more valuable and interesting than optimism.
- seiferteric 9y ago
  Because it was sold as something better. When siri first came out it was billed as something revolutionary and that it would continuously improve as more people used it. Did that even happen? Outside of the "happy path" sort of questions, I find it rather disappointing with pre-canned responses, or just showing me search results most of the time. Now I really only use it for setting reminders or alarms.
  - dahart 9y ago
    Totally. Me too, my use of Google Voice and Siri and Alexa has declined because I don't usually get what I want the first time. Reminders and alarms it always gets right, it's faster to set a reminder by voice than by typing. But I think you and I are illustrating how we expect more from voice search than text search. My use of text search hasn't declined like my use of voice search, and text search is just as fundamentally bad as voice search. I think it's because I can see & sift many results, and because I can easily iterate on my query when it's not quite right. Voice search can't do either easily. Perhaps Amazon, Apple, Google and Microsoft all initially thought that the revolutionary part was being able to speak a query and have the query match what you said, and that the search part was already good enough.
csomar 9y ago
It's interesting that while many fails, there is still one that wins. That is if you combine the efforts of these 4 digital assistants, you'll get a much smarter one. Do they have an API? Can you query siri, cortana, etc..?
- giobox 9y ago
  There are APIs for both Amazon and Google's voice assistant services. Not surprisingly Siri doesn't expose a public one, I've no idea about Cortana. I've messed around a little with them on the Raspberry Pi. This idea, while simple in principle, might be kinda annoying in practice. You're still left with similar issues - how do you decide which talking cylinder service answered the question best? Do you play all of the answers? For me I'm fairly sure listening to all of them in a row would frustrate me even further - just waiting for Alexa to finish telling me the news headlines is sometimes kinda annoying, especially when that information in visual form can be grokked almost instantly. Many of these devices, especially the Google one, are getting better at context based followup questions - managing who to send your follow up question to could be kinda crappy as well. I suppose you could do one device that could ask each service individually ("Alexa...", "Ok Google..."), but in my experience as soon as I get one bad answer, I inevitably just use google.com to find what I need rather than risk wasting my time on another failed conversation. The main part that I've found hard to do in home rolled voice assistants is microphone arrays. Almost all these devices use pretty sophisticated microphone technologies for things like noise cancelling, subject isolation etc, which so far has been non-trivial to do to a similar standard in homemade versions of them. It also certainly used to be the case that creating your own "hotword" system to call the Alexa API was technically against the ToS (it allowed you to use a button press to call Alexa instead), as naturally Amazon would rather you buy a real Echo. No idea if this is still the case, and at any rate Amazon can't really enforce this either, but worth mentioning.
  - forgot-my-pw 9y ago
    A funny popular experiment is the seebotschat Twitch account who livestreamed 2 google homes running Cleverbot API talking to each others. Here's a short highlight: https://www.youtube.com/watch?v=WoI6_z2mfdY https://www.youtube.com/watch?v=WoI6_z2mfdY Some implementation details in AMA: https://redd.it/5nz3eb https://redd.it/5nz3eb
davidw 9y ago
Google's thing can't even figure out my wife's Italian name most of the time. It's quite frustrating.
- octalmage 9y ago
  My girlfriend has the name Taryn and Siri really struggles with it, usually correcting it to Karen or Terell (both names in my address book). Alexa does better but probably because it doesn't know about the other names.
  - evilduck 9y ago
    Not that it excuses Siri's shortcomings or helps if you're referring to her by name mid-sentence texting to someone else, but you can give assign nicknames in your Contacts app. It might make it less frustrating to dictate texts or start calls. So instead of saying "Call Taryn" and getting it misheard you could say "Call my girlfriend".
paradite 9y ago
Ensemble for the win, I suspect that they can perform better than the combined individual best when sharing training resources and models.
GCA10 9y ago
As much fun as this test is, it dodges the most interesting question of all: "Are these machines supposed to be talking search engines?" I'm increasingly believing that the answer is: "No." These machines (especially Alexa) are rapidly gaining popularity while still providing pretty ragged answers to search queries. So we should start asking: "Are they taking on a different function that didn't match our early expectations?" In a word, yeah. Alexa is a really nifty jukebox for those of us that don't have the good sense to create formal playlists. It's a handy kitchen timer, especially if you've got multiple pots doing different things. It's a better alarm clock and a better purveyor of soothing bedtime sounds. (If you're asking: Good god, how many people really want or need that, think: Fussing infants.) Smartphones already provide pretty excellent search results on the fly. I'm not sure voice-powered assistants will re-solve that problem with great success. But there are a surprising number of rudimentary needs around the house for which a voice-enabled device becomes quite handy.
- ghaff 9y ago
  I expect there are a lot of questions that lend themselves to concise answers. BUT if sensible informally phrased questions don't get answered properly a decent percentage of the time, we learn not to bother. I agree that voice interfaces aren't good for a lot of things. How do I cook XYZ? probably isn't suited. But overall performance just isn't that great.
  - bluGill 9y ago
    > How do I cook XYZ? probably isn't suited That is perfectly suited to voice if voice worked. When I call my mom for the recipe for cake it would be a whole lot easier if my mom would say "beat the eggs for 1 minute", listen for the beater to start and then say stop after one minute. My mom has better things to do with her time than walk me through the recipe, but an assistant should be able to do this. Of course I have just transformed the problem into something that technology isn't able to do. However the problem isn't with the voice interface it is our AI isn't yet up to all that. (poor AI, every time they do something useful we rename it and move the goal posts)
rojobuffalo 9y ago
Might be interesting to test with Wolfram Alpha as well. It looks like some of the questions wouldn't fit the WA API, but I'm curious how it would score.
EGreg 9y ago
Why isn't siri as half as smart as Wolfram Alpha's box? Someone should license them! I want to be able to ask basic factual questions while driving, get the answers and dig deeper. Until then I would like an audio service like Google Helpouts used to be, on demand. Like Magic service.
- freeone3000 9y ago
  Wolfram Alpha is slow. Even if it's right, it's only good for knowledge questions out of its database - things like public figures ("how many children does barak obama have"), physical statistics ("what is the melting point of tungsten"), and so on work fine. However, topical ("what about aluminium?"), temporal ("what's the weather?"), and location-based ("show me restaurants nearby") are outside of scope for wolfram alpha entirely - so a given app must aggregate. Why don't apps aggregate? The "can you handle this?" api endpoint is frequently returns false positives, and the proper API is really slow (multiple seconds) for negatives . If we get a false positive, or something hard to detect as a negative, that's the only answer we can show. And since a voice assistant is expected to return one answer quickly, this is straight out.
- fooker 9y ago
  Siri had wolfram alpha integration at launch. Then Apple got overconfident and removed it. Google "Siri getting dumber" for reference.
notadoc 9y ago
I find Google Assistant to be very good at answering most questions. Also, I can accurately get the weather from Siri most of the time.
majani 9y ago
I'm curious about how accurate the assistants were at listening. That used to be the most pressing issue with voice commands that had relegated the technology to a running joke. It appears that's understandably what the companies have been focusing on so far, but there's still work to be done to get to 100% accuracy of listening, especially when you take into account exotic names and interchanging between languages and slang.
colinbartlett 9y ago
Forget about questions, I cannot even get Siri on Apple TV to recognize what I am saying. I have often wanted to keep a kind of journal like this poster but I suspect it would recognize the correct words about 30% of the time. My wife who, unlike me, is not a native English speaker has probably a 10% success rate. This is why any kind of forthcoming voice-response Apple device is completely a nonstarter to me.
- Domenic_S 9y ago
  What a weird conclusion, that future -- and presumably better -- tech would be a nonstarter because current tech doesn't work for you.
  - michaelmrose 9y ago
    The parent poster is dubious that improvements in features will coincide with improvements in recognizing his voice. Voice assist functionality could be 200% more awesome but if it specifically doesn't seem good at just recognizing what he has said such functionality is useless to him. This isn't terribly strange at all.
- forgot-my-pw 9y ago
  Google voice recognition has 5% word error rate now: https://venturebeat.com/2017/05/17/googles-speech-recognition-technology-now-has-a-4-9-word-error-rate/ https://venturebeat.com/2017/05/17/googles-speech-recognitio... It might actually be better of Apple were to license the speech recognition or use the Cloud Speech API.
coldcode 9y ago
I would like to know how much the pack of lies costs?
- anentropic 9y ago
  "It’s right there on their website. These numbers do not accurately represent the price you will pay"
ksk 9y ago
One problem is that the computing power dedicated to each user is minuscule. If you could dedicate a super computer for processing every input, you could have a much more sophisticated system that could easily deal with all of those queries.
nmstoker 9y ago
I'm surprised by a number of the failures, such that I did wonder if the failure might be happening on the speech recognition side rather than the response generation side. Both Google Home and Alexa have very little problem recognising my speech whilst friends often struggle even when they seem to say the precise same phrase, to the point it's mildly entertaining. With Google I suspect they've tailored to my voice (I've used voice commands extensively for several years) but I've only had a Dot briefly and it worked well from the start. Another surprise is that they cope well with my perculiarly English English phrasing and pronunciation, but I'm sure there are lots of less widely spoken dialects that would throw them.
- ghaff 9y ago
  Alexa is the first thing I've owned that really does quite a solid job in the voice recognition department (whatever its failings to return something useful based on that recognition). Siri on my phone is rather hit or miss by contrast. I suspect that the microphone array has a lot to do with it. Anecdotally, I've read pieces by people saying that homebrew "Echos" together with the Alexa APIs aren't as good as an actual Alexa.
rojobuffalo 9y ago
I keep coming back to the idea that progress towards AGI might be made by someone working on a "coordinator" agent. We might have several narrowly focused agents with deep knowledge in particular domains: a mathematician, a fact-checker, a botanist, a structural engineer, etc.; then have an agent that broadly understands how to route requests to the right vertical. Maybe that's already descriptive of the underlying architecture for some of these agents. The alternative might be that we interface with several different conversational agents, and like interfacing with people, we use our judgement to decide which specialist to ask.
- Bjartr 9y ago
  That's kind of what Watson did, but that level of architecture hasn't made it into personal assistants yet
kelchm 9y ago
One of the most 'magical' experiences I've ever had using Google Assistant was the following exchange: Me: "Okay Google, What's the latest album by Death Cab For Cutie?" Google: "The latest ablum by Death Cab For Cutie is Kintsugi" Me: "Okay Google, Play the album Kintsugi on Spotify" Google: "Okay, asking to play Kintsugi. [album starts playing]"
- sib 9y ago
  Wouldn't it have been a lot more magical if it had simply asked, "Would you like me to play it?" (knowing that you have a Spotify subscription) at the end of answering your question? Or, at least, for you to be able to say "Play it!" rather than the unnatural "Okay Google, play the album Kintsugi on Spotify"...
  - forgot-my-pw 9y ago
    The context searching might work with: "Play it". I haven't tried myself.
  - tomc1985 9y ago
    I would have preferred a straight answer, without a servile question at the end.
maerF0x0 9y ago
I'm gonna make a service where you ask my service and it answers the 4 answers given :D
LesZedCB 9y ago
i wonder if there would be any use in services that don't respond in real-time. I think these digital assistants are nerfed by the real-time response requirement. I'd be happy to ask some of those questions, and get a pop up in a few minutes. And they could be of much higher quality as they can be processed and better researched.
netvarun 9y ago
Shameless Plug: I work at Semantics3 [https://semantics3.com/ https://semantics3.com/] - an API for product and pricing data. These 4 digital assistants should partner with us to help their users find the prices for a pack of Lays chips, iphone, etc. ;)
contingencies 9y ago
"Okay Google, spend my money."
- glitcher 9y ago
  The Alexa version :) https://xkcd.com/1807/ https://xkcd.com/1807/
  - spurlock 9y ago
    "Okay Google, infer semantic meaning from my words"
boznz 9y ago
English is a terrible language for this, unfortunately it's the only one I speak. Not sure how other languages cope, I suspect the simpler ones cope much better. We almost need a spoken equivalent of SQL
- thaumasiotes 9y ago
  English is best known for being simpler than average, not more complex. It's one of the flagships (along with Latin / Mandarin Chinese / Swahili) for the theory "languages which are widely learned by adults become simplified over time".
lowbloodsugar 9y ago
Man asks Amazon digital assistant about the price of Lay's chips, is disappointed when it "has little interest in having a conversation about it" and wants to sell it to him instead. o_0
myrandomcomment 9y ago
So I decided to ask Siri some of these that he listed as giving a Bing search answer that I felt Wolfram would have answered correctly for Siri. I my case I did get the correct answer, not a Bing result. Where does the Jackfruit grow? What is the boiling point of water at an altitude of 1km? What is 1km in feet? How far away is Disneyland? For "km" I said "kilometer" and not "km".
jandrese 9y ago
> Siri took the crown on factual questions, but surprisingly did poorly on reasoning (“Queries”) where I expected the Wolfram Alpha-backed service to get flying colours. Didn't Apple ditch the WolframAlpha integration pretty quickly after Siri was released? I remember a lot of the Wolfram type queries stopped working shortly after release.
- valleyer 9y ago
  This very article shows examples of Siri responding with Wolfram Alpha results.
- yosito 9y ago
  They didn't ditch it, but they did fuck it up and seem to prefer Bing searches now over Wolfram Alpha queries.
greggman 9y ago
I found it curious the author didn't like that google was quoting webpages. I actually liked that Google was not taking credit for the info and Google was informing where the info was coming from so I can decide if I want to trust that info or not. Of course if I ask for a fact like "How many inches in a meter" I just want the answer. But, if I ask "what's the weather going to be tomorrow" I might prefer an answer like "badweather.com says it's going to rain tomorrow" so I can then think (ugh, badweather.com is always wrong) and ask "What does goodweather.com say about tomorrow's weather". Ideally I could ask the assistant to use a particular site by default. This is specially true for me because Siri's default doesn't seem very accurate to me being that I live on the other side of the world from the offices of the company they use for weather info.
- ancalimon 9y ago
  Heya, I'm sorry for the confusion. My issue with quoting webpages is not that it quotes them -- that's fine -- but that it does so in a very verbose manner. This leads to information overload. For example: Me: What is the boiling point of water at an altitude of 1km? Google: At sea level, water boils at 212 °F. With each 500-feet increase in elevation, the boiling point of water is lowered by just under 1 °F. At 7,500 feet, for example, water boils at about 198 °F. Because water boils at a lower temperature at higher elevations, foods that are prepared by boiling or simmering will cook at The problem is with the vast quantity of information, and the fact that some is both irrelevant and truncated. The last sentence is incomplete and cut off, yet as a listener I have no way of knowing this. I will thus try to remember it, at the expense of the facts that came previously. When reading a webpage, the important part is to read the specific parts of interest, and not overload the user. If it can't do that, it risks providing irrelevant or, quite frankly, confusing data (such as the odd answer to how much a Dreamliner weighs). I don't know if that's better than not providing an answer at all.
benzoate 9y ago
I’d be interested to see how Siri performs on different devices. The Siri on your phone is not the same Siri that you have on your Apple TV. The Apple TV version is much better at giving information about tv shows, movies and music, but terrible in comparison for everything else.
dheera 9y ago
It's far less than half for me. Maybe less than 5%. Examples of queries that I frequently need that won't work: - OK Google, please download an offline maps area for Yosemite National Park and about 80 kilometers around it. - OK Google, navigate to Yosemite National Park. Highway 120 is closed so please remember that when we get to an area without reception, do not route me via 120 in the offline maps. - OK Google, let me know when we reach a place along our route where I can buy an SD card. - OK Google, let me know when we reach the last Trader Joe's or Safeway along our route that is still open. - OK Google, when is the next Caltrain arriving in Palo Alto that stops in San Bruno? - OK Google, if I miss that train, when is the next train? - OK Google, get me an UberPool for 2 people to Castro St. in Mountain View as long as it's under $10 and would arrive within 30 minutes. - OK Google, what is the name of the driver and what is the license plate? - OK Google, navigate to my friend's party on Facebook. - OK Google, call my friend 5 minutes before we arrive. Their number is in the wall on the event page for my friend's party. - OK Google, find me a restaurant that has non-americanized Chinese food, recommended with higher frequency on Chinese language websites than English websites, and has vegetarian options. - OK Google, connect to my Nest thermostat. My username is XXX and my password is YYY. Turn on the air conditioner 30 minutes before we arrive home based on the navigation. - OK Google, close that Java error that popped up and is covering the navigation. - OK Google, please zoom out the map slightly so I can see how far we are from the destination. - OK Google, please go back to the normal navigation view. - OK Google, navigate to my next calendar event's location. - OK Google, install Facebook Messenger, login as XXX with password YYY, and message ZZZ saying that I'll be late by however much the Uber app estimates. - OK Google, turn off my alarm clock whenever I am biking or driving. - OK Google, let me know if I get an e-mail from XXX in the next 2 hours. - OK Google, please block all calls except from XXX for the next 1 hour. XXX's phone number is in their e-mail signature. Yeah. We're a LONG way before assistants are useful. The pieces are all there. It's not a machine learning problem anymore. It's just that there are just way too many walled gardens between the various parties that hold the data necessary to be useful.
- spurlock 9y ago
  > there are just way too many walled gardens between the various parties that hold the data necessary to implement any of the above. This is changing. If Google had their way, they would be assigning IPV6 addresses to bits of dust lying around in your house and trying to assign semantic meaning to them. If they had their way.

298 points by forgot-my-pw 9y ago

I got 22 out of the "40 verbose Assistant questions" correct. Not bad! I beat them all (as a percentage). Maybe not a bad idea for a gameshow.

chris_overseas 9y ago
That's similar to how The Chase[1] works, except contestants go head to head with a professional quiz master instead of a digital assistant. [1] https://en.wikipedia.org/wiki/The_Chase_(UK_game_show) https://en.wikipedia.org/wiki/The_Chase_(UK_game_show)
forgot-my-pw 9y ago
Makes me wonder if Watson will do a better contextual search and single result answer. IBM should release a voice assistant too.

51Cards 9y ago

I notice some different answers on my devices. For example on the "Are tomatoes vegetables?" question my Google Home states that they are definitely a fruit. (quoting Oxford Dictionary) Edit: And "What's the height in meters of the Empire State Building?" gets me "381 meters, 443 meters to tip"

IanCal 9y ago

I tried the conversational weather one on google assistant on my phone. "Should I take an umbrella tomorrow?" "No..." and shows me tomorrows forecast. "What about the day after?" "No..." and shows a forecast that when I look more closely at I notice is for today. Neither of these also spotted that I'm heading to another city tomorrow, which is in my calendar. If I changed it to "do I need to take an umbrella for my trip tomorrow" it just searches google and gives me a search result suggesting I take a small folding umbrella... for a trip to Thailand.

visarga 9y ago
It might be so, but remember that experts were predicting computer will beat top humans in Go in 10 years. Maybe next month there will be a breakthrough with NLP. The amount of research and compute going into this problem is amazing, and we don't know what's around the corner. Also, it might be possible to create a much better assistant today, but it would be too expensive to offer to the public for free. What if it requires 100 TPUs to run?
- IanCal 9y ago
  The thing for me in that example is it's not a particularly complex set of options. It knows I'm asking for the weather, it knows that it should give me the weather in a particular location, and google knows where I'll be tomorrow. The other main mistake was (despite it getting the conversational part right) thinking "the day after" means today. I don't think these things require TPUs or new NLP.
  - visarga 9y ago
    Yes, they managed assistants badly - Google, Apple and Amazon. They could have added more APIs, recognized many more patterns of user requests, and it's just plain old programming, so what's keeping them? For example, Google Assistant doesn't play nice with YouTube. I guess it isn't in Google's interest to have a free agent software serve a free music site when the same service makes money for Amazon (Alexa). They wanted to make their agent profitable, so didn't let it be extended for free. Just my guess.
    - johnsmith21006 9y ago
      Google Home with YouTube is amazing. I say play Gwen Bottle and get Gwen Stefanni sing Message in a bottle. Or say play One Love Peas and get Blach Eye Peas ad One Love concert. What are you talking about?
- marcosdumay 9y ago
  This one problem of NLP is kinda solved¹ for more than 10 years already. What you are seeing here is a much simpler CS problem, but much harder social problem. It is "why can't my applications talk to each other?". I really doubt it will be solved in 10 years. 1 - Humans don't have a perfect solution for it, and machines are still worse, but not that worse.

codekilla 9y ago

more people are starting to appreciate that AGIish stuff is actually really, really hard.

snarf21 9y ago
Agreed, determining human intent is really, really, really hard. There are some many context clues we use in everyday life that the voice interface will never have access to like where am I standing, what's my expression, etc.
ghaff 9y ago
The ability to function as a virtual assistant, even at the level of a not-so-sharp intern [1], would be a killer app. Give it some parameters for a trip you're taking. It comes back with some options and follow-up questions. We are a long way from that point. Even that not-so-sharp intern has a huge amount of internalized knowledge about general preferences, cities, airports, etc. and probably knows questions to ask to narrow things down. I strongly suspect there are other domains where a lot of people are assuming we're 90% there and we're not. [1] Not to insult interns or any other group. I just mean you don't need to be at experienced executive assistant level to be really useful.
forgot-my-pw 9y ago
Makes me wonder if Watson will do a better contextual search and single result answer. IBM should release a voice assistant too.

dahart 9y ago

What's interesting to me reading this is the expectation that voice search should do things that text search currently doesn't do. If I ask Google about Bill Murray running for president, I presumably am looking for articles that mention the words "Bill", "Murray", and "President". I would expect to return articles that best match the query, and I would never expect Google to be able to tell me the difference between real and fake articles. For a server to answer the author's question it has to understand the question so well it can change the question into "What is the list of people that have run for U.S. president?" and then check if Bill Murray is in that list. That's a tall order. We are moving past expecting the most word matches into expecting the server to understand what we really want. Voice search is more like the "I'm feeling lucky" button, because it takes longer; you only have time for one answer and the first answer has to be right. It comes without the expectation that you're lucky if the answer happens to be right, now we need the first result to be the rightest result there is. So the glass is half empty. I personally prefer to see it half-full, but it's also true and the critique is more valuable and interesting than optimism.

seiferteric 9y ago
Because it was sold as something better. When siri first came out it was billed as something revolutionary and that it would continuously improve as more people used it. Did that even happen? Outside of the "happy path" sort of questions, I find it rather disappointing with pre-canned responses, or just showing me search results most of the time. Now I really only use it for setting reminders or alarms.
- dahart 9y ago
  Totally. Me too, my use of Google Voice and Siri and Alexa has declined because I don't usually get what I want the first time. Reminders and alarms it always gets right, it's faster to set a reminder by voice than by typing. But I think you and I are illustrating how we expect more from voice search than text search. My use of text search hasn't declined like my use of voice search, and text search is just as fundamentally bad as voice search. I think it's because I can see & sift many results, and because I can easily iterate on my query when it's not quite right. Voice search can't do either easily. Perhaps Amazon, Apple, Google and Microsoft all initially thought that the revolutionary part was being able to speak a query and have the query match what you said, and that the search part was already good enough.

csomar 9y ago

It's interesting that while many fails, there is still one that wins. That is if you combine the efforts of these 4 digital assistants, you'll get a much smarter one. Do they have an API? Can you query siri, cortana, etc..?

giobox 9y ago
There are APIs for both Amazon and Google's voice assistant services. Not surprisingly Siri doesn't expose a public one, I've no idea about Cortana. I've messed around a little with them on the Raspberry Pi. This idea, while simple in principle, might be kinda annoying in practice. You're still left with similar issues - how do you decide which talking cylinder service answered the question best? Do you play all of the answers? For me I'm fairly sure listening to all of them in a row would frustrate me even further - just waiting for Alexa to finish telling me the news headlines is sometimes kinda annoying, especially when that information in visual form can be grokked almost instantly. Many of these devices, especially the Google one, are getting better at context based followup questions - managing who to send your follow up question to could be kinda crappy as well. I suppose you could do one device that could ask each service individually ("Alexa...", "Ok Google..."), but in my experience as soon as I get one bad answer, I inevitably just use google.com to find what I need rather than risk wasting my time on another failed conversation. The main part that I've found hard to do in home rolled voice assistants is microphone arrays. Almost all these devices use pretty sophisticated microphone technologies for things like noise cancelling, subject isolation etc, which so far has been non-trivial to do to a similar standard in homemade versions of them. It also certainly used to be the case that creating your own "hotword" system to call the Alexa API was technically against the ToS (it allowed you to use a button press to call Alexa instead), as naturally Amazon would rather you buy a real Echo. No idea if this is still the case, and at any rate Amazon can't really enforce this either, but worth mentioning.
- forgot-my-pw 9y ago
  A funny popular experiment is the seebotschat Twitch account who livestreamed 2 google homes running Cleverbot API talking to each others. Here's a short highlight: https://www.youtube.com/watch?v=WoI6_z2mfdY https://www.youtube.com/watch?v=WoI6_z2mfdY Some implementation details in AMA: https://redd.it/5nz3eb https://redd.it/5nz3eb

davidw 9y ago

Google's thing can't even figure out my wife's Italian name most of the time. It's quite frustrating.

octalmage 9y ago
My girlfriend has the name Taryn and Siri really struggles with it, usually correcting it to Karen or Terell (both names in my address book). Alexa does better but probably because it doesn't know about the other names.
- evilduck 9y ago
  Not that it excuses Siri's shortcomings or helps if you're referring to her by name mid-sentence texting to someone else, but you can give assign nicknames in your Contacts app. It might make it less frustrating to dictate texts or start calls. So instead of saying "Call Taryn" and getting it misheard you could say "Call my girlfriend".

paradite 9y ago

Ensemble for the win, I suspect that they can perform better than the combined individual best when sharing training resources and models.

GCA10 9y ago

As much fun as this test is, it dodges the most interesting question of all: "Are these machines supposed to be talking search engines?" I'm increasingly believing that the answer is: "No." These machines (especially Alexa) are rapidly gaining popularity while still providing pretty ragged answers to search queries. So we should start asking: "Are they taking on a different function that didn't match our early expectations?" In a word, yeah. Alexa is a really nifty jukebox for those of us that don't have the good sense to create formal playlists. It's a handy kitchen timer, especially if you've got multiple pots doing different things. It's a better alarm clock and a better purveyor of soothing bedtime sounds. (If you're asking: Good god, how many people really want or need that, think: Fussing infants.) Smartphones already provide pretty excellent search results on the fly. I'm not sure voice-powered assistants will re-solve that problem with great success. But there are a surprising number of rudimentary needs around the house for which a voice-enabled device becomes quite handy.

ghaff 9y ago
I expect there are a lot of questions that lend themselves to concise answers. BUT if sensible informally phrased questions don't get answered properly a decent percentage of the time, we learn not to bother. I agree that voice interfaces aren't good for a lot of things. How do I cook XYZ? probably isn't suited. But overall performance just isn't that great.
- bluGill 9y ago
  > How do I cook XYZ? probably isn't suited That is perfectly suited to voice if voice worked. When I call my mom for the recipe for cake it would be a whole lot easier if my mom would say "beat the eggs for 1 minute", listen for the beater to start and then say stop after one minute. My mom has better things to do with her time than walk me through the recipe, but an assistant should be able to do this. Of course I have just transformed the problem into something that technology isn't able to do. However the problem isn't with the voice interface it is our AI isn't yet up to all that. (poor AI, every time they do something useful we rename it and move the goal posts)

rojobuffalo 9y ago

Might be interesting to test with Wolfram Alpha as well. It looks like some of the questions wouldn't fit the WA API, but I'm curious how it would score.

EGreg 9y ago

Why isn't siri as half as smart as Wolfram Alpha's box? Someone should license them! I want to be able to ask basic factual questions while driving, get the answers and dig deeper. Until then I would like an audio service like Google Helpouts used to be, on demand. Like Magic service.

freeone3000 9y ago
Wolfram Alpha is slow. Even if it's right, it's only good for knowledge questions out of its database - things like public figures ("how many children does barak obama have"), physical statistics ("what is the melting point of tungsten"), and so on work fine. However, topical ("what about aluminium?"), temporal ("what's the weather?"), and location-based ("show me restaurants nearby") are outside of scope for wolfram alpha entirely - so a given app must aggregate. Why don't apps aggregate? The "can you handle this?" api endpoint is frequently returns false positives, and the proper API is really slow (multiple seconds) for negatives . If we get a false positive, or something hard to detect as a negative, that's the only answer we can show. And since a voice assistant is expected to return one answer quickly, this is straight out.
fooker 9y ago
Siri had wolfram alpha integration at launch. Then Apple got overconfident and removed it. Google "Siri getting dumber" for reference.

notadoc 9y ago

I find Google Assistant to be very good at answering most questions. Also, I can accurately get the weather from Siri most of the time.

majani 9y ago

I'm curious about how accurate the assistants were at listening. That used to be the most pressing issue with voice commands that had relegated the technology to a running joke. It appears that's understandably what the companies have been focusing on so far, but there's still work to be done to get to 100% accuracy of listening, especially when you take into account exotic names and interchanging between languages and slang.

colinbartlett 9y ago

Forget about questions, I cannot even get Siri on Apple TV to recognize what I am saying. I have often wanted to keep a kind of journal like this poster but I suspect it would recognize the correct words about 30% of the time. My wife who, unlike me, is not a native English speaker has probably a 10% success rate. This is why any kind of forthcoming voice-response Apple device is completely a nonstarter to me.

Domenic_S 9y ago
What a weird conclusion, that future -- and presumably better -- tech would be a nonstarter because current tech doesn't work for you.
- michaelmrose 9y ago
  The parent poster is dubious that improvements in features will coincide with improvements in recognizing his voice. Voice assist functionality could be 200% more awesome but if it specifically doesn't seem good at just recognizing what he has said such functionality is useless to him. This isn't terribly strange at all.
forgot-my-pw 9y ago
Google voice recognition has 5% word error rate now: https://venturebeat.com/2017/05/17/googles-speech-recognition-technology-now-has-a-4-9-word-error-rate/ https://venturebeat.com/2017/05/17/googles-speech-recognitio... It might actually be better of Apple were to license the speech recognition or use the Cloud Speech API.

coldcode 9y ago

I would like to know how much the pack of lies costs?

anentropic 9y ago
"It’s right there on their website. These numbers do not accurately represent the price you will pay"

ksk 9y ago

One problem is that the computing power dedicated to each user is minuscule. If you could dedicate a super computer for processing every input, you could have a much more sophisticated system that could easily deal with all of those queries.

nmstoker 9y ago

I'm surprised by a number of the failures, such that I did wonder if the failure might be happening on the speech recognition side rather than the response generation side. Both Google Home and Alexa have very little problem recognising my speech whilst friends often struggle even when they seem to say the precise same phrase, to the point it's mildly entertaining. With Google I suspect they've tailored to my voice (I've used voice commands extensively for several years) but I've only had a Dot briefly and it worked well from the start. Another surprise is that they cope well with my perculiarly English English phrasing and pronunciation, but I'm sure there are lots of less widely spoken dialects that would throw them.

ghaff 9y ago
Alexa is the first thing I've owned that really does quite a solid job in the voice recognition department (whatever its failings to return something useful based on that recognition). Siri on my phone is rather hit or miss by contrast. I suspect that the microphone array has a lot to do with it. Anecdotally, I've read pieces by people saying that homebrew "Echos" together with the Alexa APIs aren't as good as an actual Alexa.

rojobuffalo 9y ago

I keep coming back to the idea that progress towards AGI might be made by someone working on a "coordinator" agent. We might have several narrowly focused agents with deep knowledge in particular domains: a mathematician, a fact-checker, a botanist, a structural engineer, etc.; then have an agent that broadly understands how to route requests to the right vertical. Maybe that's already descriptive of the underlying architecture for some of these agents. The alternative might be that we interface with several different conversational agents, and like interfacing with people, we use our judgement to decide which specialist to ask.

Bjartr 9y ago
That's kind of what Watson did, but that level of architecture hasn't made it into personal assistants yet

kelchm 9y ago

One of the most 'magical' experiences I've ever had using Google Assistant was the following exchange: Me: "Okay Google, What's the latest album by Death Cab For Cutie?" Google: "The latest ablum by Death Cab For Cutie is Kintsugi" Me: "Okay Google, Play the album Kintsugi on Spotify" Google: "Okay, asking to play Kintsugi. [album starts playing]"

sib 9y ago
Wouldn't it have been a lot more magical if it had simply asked, "Would you like me to play it?" (knowing that you have a Spotify subscription) at the end of answering your question? Or, at least, for you to be able to say "Play it!" rather than the unnatural "Okay Google, play the album Kintsugi on Spotify"...
- forgot-my-pw 9y ago
  The context searching might work with: "Play it". I haven't tried myself.
- tomc1985 9y ago
  I would have preferred a straight answer, without a servile question at the end.

maerF0x0 9y ago

I'm gonna make a service where you ask my service and it answers the 4 answers given :D

LesZedCB 9y ago

i wonder if there would be any use in services that don't respond in real-time. I think these digital assistants are nerfed by the real-time response requirement. I'd be happy to ask some of those questions, and get a pop up in a few minutes. And they could be of much higher quality as they can be processed and better researched.

netvarun 9y ago

Shameless Plug: I work at Semantics3 [https://semantics3.com/ https://semantics3.com/] - an API for product and pricing data. These 4 digital assistants should partner with us to help their users find the prices for a pack of Lays chips, iphone, etc. ;)

contingencies 9y ago

"Okay Google, spend my money."

glitcher 9y ago
The Alexa version :) https://xkcd.com/1807/ https://xkcd.com/1807/
- spurlock 9y ago
  "Okay Google, infer semantic meaning from my words"

boznz 9y ago

English is a terrible language for this, unfortunately it's the only one I speak. Not sure how other languages cope, I suspect the simpler ones cope much better. We almost need a spoken equivalent of SQL

thaumasiotes 9y ago
English is best known for being simpler than average, not more complex. It's one of the flagships (along with Latin / Mandarin Chinese / Swahili) for the theory "languages which are widely learned by adults become simplified over time".

lowbloodsugar 9y ago

Man asks Amazon digital assistant about the price of Lay's chips, is disappointed when it "has little interest in having a conversation about it" and wants to sell it to him instead. o_0

myrandomcomment 9y ago

So I decided to ask Siri some of these that he listed as giving a Bing search answer that I felt Wolfram would have answered correctly for Siri. I my case I did get the correct answer, not a Bing result. Where does the Jackfruit grow? What is the boiling point of water at an altitude of 1km? What is 1km in feet? How far away is Disneyland? For "km" I said "kilometer" and not "km".

jandrese 9y ago

> Siri took the crown on factual questions, but surprisingly did poorly on reasoning (“Queries”) where I expected the Wolfram Alpha-backed service to get flying colours. Didn't Apple ditch the WolframAlpha integration pretty quickly after Siri was released? I remember a lot of the Wolfram type queries stopped working shortly after release.

valleyer 9y ago
This very article shows examples of Siri responding with Wolfram Alpha results.
yosito 9y ago
They didn't ditch it, but they did fuck it up and seem to prefer Bing searches now over Wolfram Alpha queries.

greggman 9y ago

I found it curious the author didn't like that google was quoting webpages. I actually liked that Google was not taking credit for the info and Google was informing where the info was coming from so I can decide if I want to trust that info or not. Of course if I ask for a fact like "How many inches in a meter" I just want the answer. But, if I ask "what's the weather going to be tomorrow" I might prefer an answer like "badweather.com says it's going to rain tomorrow" so I can then think (ugh, badweather.com is always wrong) and ask "What does goodweather.com say about tomorrow's weather". Ideally I could ask the assistant to use a particular site by default. This is specially true for me because Siri's default doesn't seem very accurate to me being that I live on the other side of the world from the offices of the company they use for weather info.

ancalimon 9y ago
Heya, I'm sorry for the confusion. My issue with quoting webpages is not that it quotes them -- that's fine -- but that it does so in a very verbose manner. This leads to information overload. For example: Me: What is the boiling point of water at an altitude of 1km? Google: At sea level, water boils at 212 °F. With each 500-feet increase in elevation, the boiling point of water is lowered by just under 1 °F. At 7,500 feet, for example, water boils at about 198 °F. Because water boils at a lower temperature at higher elevations, foods that are prepared by boiling or simmering will cook at The problem is with the vast quantity of information, and the fact that some is both irrelevant and truncated. The last sentence is incomplete and cut off, yet as a listener I have no way of knowing this. I will thus try to remember it, at the expense of the facts that came previously. When reading a webpage, the important part is to read the specific parts of interest, and not overload the user. If it can't do that, it risks providing irrelevant or, quite frankly, confusing data (such as the odd answer to how much a Dreamliner weighs). I don't know if that's better than not providing an answer at all.

benzoate 9y ago

I’d be interested to see how Siri performs on different devices. The Siri on your phone is not the same Siri that you have on your Apple TV. The Apple TV version is much better at giving information about tv shows, movies and music, but terrible in comparison for everything else.

dheera 9y ago

It's far less than half for me. Maybe less than 5%. Examples of queries that I frequently need that won't work: - OK Google, please download an offline maps area for Yosemite National Park and about 80 kilometers around it. - OK Google, navigate to Yosemite National Park. Highway 120 is closed so please remember that when we get to an area without reception, do not route me via 120 in the offline maps. - OK Google, let me know when we reach a place along our route where I can buy an SD card. - OK Google, let me know when we reach the last Trader Joe's or Safeway along our route that is still open. - OK Google, when is the next Caltrain arriving in Palo Alto that stops in San Bruno? - OK Google, if I miss that train, when is the next train? - OK Google, get me an UberPool for 2 people to Castro St. in Mountain View as long as it's under $10 and would arrive within 30 minutes. - OK Google, what is the name of the driver and what is the license plate? - OK Google, navigate to my friend's party on Facebook. - OK Google, call my friend 5 minutes before we arrive. Their number is in the wall on the event page for my friend's party. - OK Google, find me a restaurant that has non-americanized Chinese food, recommended with higher frequency on Chinese language websites than English websites, and has vegetarian options. - OK Google, connect to my Nest thermostat. My username is XXX and my password is YYY. Turn on the air conditioner 30 minutes before we arrive home based on the navigation. - OK Google, close that Java error that popped up and is covering the navigation. - OK Google, please zoom out the map slightly so I can see how far we are from the destination. - OK Google, please go back to the normal navigation view. - OK Google, navigate to my next calendar event's location. - OK Google, install Facebook Messenger, login as XXX with password YYY, and message ZZZ saying that I'll be late by however much the Uber app estimates. - OK Google, turn off my alarm clock whenever I am biking or driving. - OK Google, let me know if I get an e-mail from XXX in the next 2 hours. - OK Google, please block all calls except from XXX for the next 1 hour. XXX's phone number is in their e-mail signature. Yeah. We're a LONG way before assistants are useful. The pieces are all there. It's not a machine learning problem anymore. It's just that there are just way too many walled gardens between the various parties that hold the data necessary to be useful.

spurlock 9y ago
> there are just way too many walled gardens between the various parties that hold the data necessary to implement any of the above. This is changing. If Google had their way, they would be assigning IPV6 addresses to bits of dust lying around in your house and trying to assign semantic meaning to them. If they had their way.