3 ms·

The explanation for this is fairly unremarkable (https://techcrunch.com/2018/05/24/family-claims-their-echo-sent-a-private-conversation-to-a-random-contact/ htt

The explanation for this is fairly unremarkable (https://techcrunch.com/2018/05/24/family-claims-their-echo-sent-a-private-conversation-to-a-random-contact/ https://techcrunch.com/2018/05/24/family-claims-their-echo-s...): Echo woke up due to a word in background conversation sounding like “Alexa.” Then, the subsequent conversation was heard as a “send message” request. At which point, Alexa said out loud “To whom?” At which point, the background conversation was interpreted as a name in the customers contact list. Alexa then asked out loud, “[contact name], right?” Alexa then interpreted background conversation as “right”. As unlikely as this string of events is, we are evaluating options to make this case even less likely.

titzer 8y ago
Actually it's very remarkable on how bad the voice recognition is, and how Amazon can knowingly ship this crap to people, putting it in their homes, recording their private interactions. Ya'll recorded people having sex yet? It sounds like the machine learning models are tuned for specific phrases and have terrible, terrible false positive rates. "Evaluating options to make this less likely"? Now I don't even know what mental model you guys are using (if you are from Amazon). This shouldn't be a matter of "likelihood". But OK, fine, let's use math. AFAICT there is about a 0% chance that a human would make such a string of errors interpreting human speech, but let's call it 1 in a trillion. If you guys aren't doing better than about 1 in a trillion for this string of at least 4 interactions, then one of those terms is stupidly, stupidly high.[1] It actually interpreted random conversation as the name of someone on a contact list? Horrible. trust-- [1] Not even to mention, a human has context, understands boundaries, preferences, and has a ML voice recognition model developed and tuned over decades of interacting with real people. A human would also be smart enough to understand the other human's situation, context, state of mind, and realize that even just the cadence of the conversation not changing in response to queries was indicative of the humans not acknowledging the query. Machines are f'in stupid.
- andreyk 8y ago
  It's a 1-in-a-million edge case.... I would hope people on HN at least would understand this stuff happens in software.
  - titzer 8y ago
    1 in a million happens about 4000 times a second on a single CPU running at 4ghz... Did you really want to talk concrete numbers? Because if so, I am wondering about the probability that Amazon's voice recognition mistakes random conversation as a valid entry in someone's contact list, as well as the other terms in this equation. I don't know if you work for Amazon, and if so I don't want to single you out specifically, but this is a pretty bad screwup, and it does not inspire confidence. Please don't brush it away with "oh it's a 1-in-a-million edge case". That attitude is even worse, and if it is indeed that attitude inside of Amazon, then I am even more strongly against this and I hope that further, deeper scrutiny is applied here, because this screwup is actually illegal.
    - andreyk 8y ago
      I do not work in Amazon. I side with them on this because look at this protocol: 1 Wake up w “Alexa.” 2 respond to “send message” 3 respond to “To whom?” 4 respond to “[contact name], right?” As an engineer (well, an AI researcher who used to engineer), that looks to me like they were not negligent and it was hard to predict background conversation would produce this unlikely set of inputs - it would be nice to see stats but this is the first time it is covered in the media to my knowledge. And as with the laughter story, they will now change the inputs to make it likely 1-in-a-trillion this will happen. However, it does seem like Alexa etc. will have to be better about recognizing audio from TVs/conversations and stuff directed at it - and I am sure they are working on it. PS comparison to CPU is not great obvs since it's about number of instances (how many times does "Alexa" get woken up by background audio - not 4ghz)
      - titzer 8y ago
        They would like to put this in a billion homes no doubt, and suppose that people are home, talking, a couple hours a day. Now we are talking real numbers. A few billion conversation-hours per day, 365 days in a year--suddenly one in a trillion is starting to look like it's gonna happen a couple times a year. Now if Amazon knows these probabilities--which they don't, because they clearly have not done due diligence in understanding their rapidly evolving, inscrutable voice models--they are now knowingly violating eavesdropping laws, probabilistically. This is part of the problem. People want to handwave away small probabilities when they should be busting their asses to make probabilities actually 0--solutions like not having this crap in their house at all.

titzer 8y ago

Actually it's very remarkable on how bad the voice recognition is, and how Amazon can knowingly ship this crap to people, putting it in their homes, recording their private interactions. Ya'll recorded people having sex yet? It sounds like the machine learning models are tuned for specific phrases and have terrible, terrible false positive rates. "Evaluating options to make this less likely"? Now I don't even know what mental model you guys are using (if you are from Amazon). This shouldn't be a matter of "likelihood". But OK, fine, let's use math. AFAICT there is about a 0% chance that a human would make such a string of errors interpreting human speech, but let's call it 1 in a trillion. If you guys aren't doing better than about 1 in a trillion for this string of at least 4 interactions, then one of those terms is stupidly, stupidly high.[1] It actually interpreted random conversation as the name of someone on a contact list? Horrible. trust-- [1] Not even to mention, a human has context, understands boundaries, preferences, and has a ML voice recognition model developed and tuned over decades of interacting with real people. A human would also be smart enough to understand the other human's situation, context, state of mind, and realize that even just the cadence of the conversation not changing in response to queries was indicative of the humans not acknowledging the query. Machines are f'in stupid.

andreyk 8y ago
It's a 1-in-a-million edge case.... I would hope people on HN at least would understand this stuff happens in software.
- titzer 8y ago
  1 in a million happens about 4000 times a second on a single CPU running at 4ghz... Did you really want to talk concrete numbers? Because if so, I am wondering about the probability that Amazon's voice recognition mistakes random conversation as a valid entry in someone's contact list, as well as the other terms in this equation. I don't know if you work for Amazon, and if so I don't want to single you out specifically, but this is a pretty bad screwup, and it does not inspire confidence. Please don't brush it away with "oh it's a 1-in-a-million edge case". That attitude is even worse, and if it is indeed that attitude inside of Amazon, then I am even more strongly against this and I hope that further, deeper scrutiny is applied here, because this screwup is actually illegal.
  - andreyk 8y ago
    I do not work in Amazon. I side with them on this because look at this protocol: 1 Wake up w “Alexa.” 2 respond to “send message” 3 respond to “To whom?” 4 respond to “[contact name], right?” As an engineer (well, an AI researcher who used to engineer), that looks to me like they were not negligent and it was hard to predict background conversation would produce this unlikely set of inputs - it would be nice to see stats but this is the first time it is covered in the media to my knowledge. And as with the laughter story, they will now change the inputs to make it likely 1-in-a-trillion this will happen. However, it does seem like Alexa etc. will have to be better about recognizing audio from TVs/conversations and stuff directed at it - and I am sure they are working on it. PS comparison to CPU is not great obvs since it's about number of instances (how many times does "Alexa" get woken up by background audio - not 4ghz)
    - titzer 8y ago
      They would like to put this in a billion homes no doubt, and suppose that people are home, talking, a couple hours a day. Now we are talking real numbers. A few billion conversation-hours per day, 365 days in a year--suddenly one in a trillion is starting to look like it's gonna happen a couple times a year. Now if Amazon knows these probabilities--which they don't, because they clearly have not done due diligence in understanding their rapidly evolving, inscrutable voice models--they are now knowingly violating eavesdropping laws, probabilistically. This is part of the problem. People want to handwave away small probabilities when they should be busting their asses to make probabilities actually 0--solutions like not having this crap in their house at all.