48 ms·

Amazon device recorded private conversation, sent it out to random contact

• newman8r 8y ago
  The article doesn't explain much, but it sounds to me like the device mistakenly interpreted a voice command to record and send a message.
  • Fwirt 8y ago
    The article makes it sound like the engineers didn't know what was going on, so it could be an actual bug, unless support was just trying to placate her.
    • macintux 8y ago
      The fact that the device didn't audibly confirm the message appears to be a bug, from my reading.
  • untog 8y ago
    Sure, but isn't that still terrifying? It brings me back to something I wonder about with these voice activated devices: you have no idea what it can do. Voice UIs are so utterly opaque, and Amazon(/Google/whoever) pushes new software updates without informing you. So it'll add new commands all the time, and you won't know about it until you accidentally trigger it, making it do... whatever. But either way, they should surely be confirming this before send, the same way Siri does when I ask it to send a text.
    • deleted 8y ago
      [deleted]
    • newman8r 8y ago
      yeah it's still off-putting. I haven't used a smartphone in years. Looking back on the days when I did use a smartphone, there was always this tiny fear in the back of my head that maybe I'd pocket-dialed someone and they'd be able to hear my conversation. It actually had a subtle chilling effect. The Hawthorne effect (which discusses how people react to being observed) seems to be especially relevant nowadays as there's a real possibility that we're being recorded at any time. https://en.wikipedia.org/wiki/Hawthorne_effect https://en.wikipedia.org/wiki/Hawthorne_effect It's actually a big theme in 1984 where the mere possibility that you were being listened to via telescreen was enough to keep you in line and influence your behavior.
  • xyzzy_plugh 8y ago
    I could be mistaken but I believe there are confirmations required? Could it be that the volume was set very low and they inadvertently triggered their way through the entire dialog? I've accidentally set alarms from a few rooms away but this is a whole other level.
    • simion314 8y ago
      >I've accidentally set alarms from a few rooms away but this is a whole other level This seems dangerous, some smart house could open the doors or stop the heat or stream your entire live on the internet.
  • maherbeg 8y ago
    POST requests should require secondary verification from the user. GET requests are probably fine to execute. Some POST requests may only need to let you know that they did it, such as adding a calendar reminder on to YOUR calendar. It should ask if it wants to send a calendar invite to someone else's calendar.
• awat 8y ago
  My concern is that this almost sounds like something not customer facing was accidentally made customer facing imho.
• gnicholas 8y ago
  > But Danielle is hoping Amazon gives her a refund for her devices, which she said their representatives have been unwilling to do. Based on Amazon's admission and apologies, I'd think they'd be willing to give a lot more than just a refund. Seems like a huge liability for them.
  • awat 8y ago
    Unless it’s expected behavior that malfunctioned. Antecdotally I’ve found a lot of companies support have almost no leeway for appeasement if the triggering behavior is considered “expected behavior” or a bug based off of it.
    • chaostheory 8y ago
      But this is Amazon. I guess things have changed for the worse on the customer service front though. This happened not too long ago (2-3 years) - probably around the time they posted a net profit In terms of the big picture, this was dumb PR wise and this story should have never happened. For less than a $1000, Amazon could have stopped the story if the PR dept were more in sync with customer service
      • awat 8y ago
        I don’t disagree at all on the PR side. Spent sometime in the contact center tools space and my general experience is if a company allows it exempt higher level employees avoid engaging with customer service directly at pretty much all costs. Not saying it happened here but thats how you get $10/hour vendors making judgement calls on customer situations that are front page news.
        • chaostheory 8y ago
          Yes, I also used to work in that space, our software / platform was used by Fortune 500 companies. My point still stands. This is Amazon and not random company X's customer service. There are very few companies like this that people can name (e.g. Nordstrom, Patagonia, ...) It this used to not happen at Amazon. Customer service used to be empowered to make customers happy, which was what made Amazon legendary. Things have changed. Customer service is now crippled like almost every other mediocre company. This is just one of the symptoms. Maybe Amazon feels that it has enough market share now, so it doesn't matter as much?
• falcolas 8y ago
  Well, when you consider that Amazon's devices once interpreted random noises as a "<trigger>, laugh" command (as per their explanation), that a voice command to record and send a message could be created from random conversation is not in the least bit surprising. Inappropriate, worrying, but not surprising.
• jschwartzi 8y ago
  This is the story I'm going to link to when my tech geek friend asks me why I don't have an Alexa like he does.
• simion314 8y ago
  This devices should by law always have a big LED on when recording, you would have the option to set it's brightness though. Also why could it send data out without any confirmation from the user.
  • kinsomo 8y ago
    > This devices should by law always have a big LED on when recording And the LED should be forbidden by law from being software-controlled. They should also have a physical inline toggle switch to disconnect the microphone(s) entirely, whose state is easy to visually confirm.
    • imrehg 8y ago
      I wonder how would you even do that? It's not a tape recorder, where someone has to physically press a button, there are no moving parts, thus it must be software controlled.
      • rolha-capoeira 8y ago
        I assume they mean that the light should not be accessible to software, just wired so that "if camera is on, light is on"
        • dpark 8y ago
          Which means the light is always on. The device must be listening or the wake word won't work.
          • millstone 8y ago
            Maybe the light could be tied to network activity? The light is on if the WiFi chip is powered.
            • dpark 8y ago
• softwarefounder 8y ago
  Pays Amazon for a listening device and installs in home...gets upset when it listens.
  • DoofusOfDeath 8y ago
    That seems overly simplistic to me. It's like saying that someone buys a car that's powered by an explosive liquid, and then complains when all the gas in the tank explodes at the same time.
    • chapium 8y ago
      Both a car and an amazon echo have a switch that turns that functionality off.
      • peeters 8y ago
        I feel like you're being purposefully reductive here. You know they're different right? The entire value proposition (and marketing strategy) of Alexa and related devices is that they are voice activated, so you don't have to physically interact with the device to make a request. Given that, it would be ridiculous to expect paying customers to switch it off when not in use. The onus on Amazon is to protect their customers' privacy while the device is being used as intended. If you really want to pick apart the car analogy, maybe this is closer to having automatic start and driving capability on a car, but the car sometimes randomly starts and drives away. Are you really going to blame the customer for that, even though they could disable the feature every time they exit the car?
        • chapium 8y ago
          The car analogies are usually quite poor. However, it depends on your use case. Echo Dot's have a mute button which prevents monitoring, or at least it seems so. I'm not an echo power user, so my needs are definitely different than for intended uses like home automation. I think for some, at least users like me would like a feature similar to "su". I'd like to confirm that Alexa should start doing stuff if it has not been confirmed for x number of minutes. I think the real problem with these home assistant devices is they are not designed with a confirmation message. This would be annoying in some cases, but surely a balance can be struck. I'm not trying to be reductive at all about the car analogy. When you turn on a car, you acknowledge that the engine is running. You can also mute an echo if you have the foreknowledge that you will not be doing activities that require voice activation.
• asn0 8y ago
  Summary of vague technical details (which may be all we hear about this): > an Alexa engineer investigated ... they said 'our engineers went through your logs, and they saw exactly what you told us, they saw exactly what you said happened, and we're sorry.' He apologized like 15 times in a matter of 30 minutes and he said we really appreciate you bringing this to our attention, this is something we need to fix!" > the engineer did not provide specifics about why it happened, or if it's a widespread issue. > "He told us that the device just guessed what we were saying" The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do.
  • DoofusOfDeath 8y ago
    > The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do. Apparently it's not programmed to do that. Unless this was a hardware glitch or cosmic-ray event.
    • cortesoft 8y ago
      As developers, we use 'programmed' in that manner all the time, to mean what we were trying to get it to do rather than what it actually does.
      • DoofusOfDeath 8y ago
        I don't. And in this particular scenario, it's an important distinction to make because there's a risk of Amazon using equivocation to deflect responsibility.
        • cortesoft 8y ago
          If this was intentional, then yes they shouldn't be able to deflect responsibility. However, if it was just a bug, I think it is a bit unfair to vilify either the company or the developers. Bugs happen, and hopefully they can learn what caused this and prevent this class of bugs in the future.
• rectang 8y ago
  "Always listening" is a fundamentally unsafe design. Once recordings of private conversations leave the local environment and make it to the cloud, eventually they will leak. It's akin to data collection by law enforcement: once the data exists, eventually it will be abused.
  • wilsonnb 8y ago
    So you're saying that all data that has made it to the cloud will eventually leak? That seems like a strange assumption to me.
    • rectang 8y ago
      Yes. Assume that all data uploaded to the cloud will eventually be compromised. That is the only safe assumption. Not even the most responsible companies (e.g. Google) can hold out 100% of the time in the face of determined assault by government. Some of that data is going to leak to three-letter agencies, or similar. Somewhat less responsible companies (e.g. Amazon) will leak data more often, to a wider range of threat agents. So then we have to consider how valuable this data is. Random sampling of private conversations within the home? Sometimes innocuous -- but if the wrong moment gets leaked, the consequences are potentially life-shattering.
      • wilsonnb 8y ago
        That seems like an extreme perspective. Kind of like the tech version of abstinence only sex ed. Sure, it's the only 100% safe way, but it's not useful for most people. It doesn't weigh the benefits against the potential cons, or even take into consideration the actually likelihood of data being leaked.
        • rectang 8y ago
          The alternative perspective you're presenting sounds to me utterly cavalier about the prospect of ruining people's lives. It's like Equifax's attitude towards identity theft: it doesn't affect their profitability, so why care? It's because such blithe dismissal of the damage caused by data gathering is so prevalent in the industry that the likelihood of devastating compromise is so high and the costs borne by the populace are spiraling upwards. Some data should never be collected. Some data should never even be uploaded.
• iooi 8y ago
  How soon until Amazon starts sending the private conversations of people discussing illegal things directly to law enforcement? It will be pretty easy to justify by bringing up bomb threats, but it would mostly be people buying/using drugs.
  • asn0 8y ago
    Amazon could add this to the suite of law enforcement products they've already been selling[1] 1. https://www.nytimes.com/2018/05/22/technology/amazon-facial-recognition.html https://www.nytimes.com/2018/05/22/technology/amazon-facial-...
• _jal 8y ago
  If you bug your own home and consent to third-party, opaque management of the surveillance, things like this will happen[1]. I expect worse to come. [1] Am not saying anyone deserves this, or that AZON is either evil or beneficent, or anything else other than precisely what was stated.
  • fastball 8y ago
    Amazon's ticker is AMZN.
    • deleted 8y ago
      [deleted]
    • _jal 8y ago
      Oops, no idea what my fingers were doing there.
• tambourine_man 8y ago
  People are volunteerelly paying for their houses to be tapped for the convenience of being able to shout: play me some song. I’m so out of touch with this world it’s scary.
  • williamscales 8y ago
    Nevermind that we are already in a world where you can play any song you like from the wireless computer you carry in your pocket. That's not convenient enough. /s
    • mikeash 8y ago
      That pocket computer also has all the same potential threats to privacy. Somehow nobody cares about them, though.
      • mieseratte 8y ago
        > That pocket computer also has all the same potential threats to privacy. Somehow nobody cares about them, though. This is just whataboutism. We can complain about Amazon in a thread about Amazon without having to tackle a whole different issue.
        • mikeash 8y ago
          It’s not “whataboutism” to point out that (almost all of) the people who can’t conprehend why anyone would ever have one of these devices have, in fact, already had one for years.
          • mieseratte 8y ago
            There's a big difference between a device that occasionally listens and can easily be powered off or muffled and a device that is actively listening at all times and can easily "accidentally" capture your conversations.
      • 8y ago
• DiabloD3 8y ago
  So when will such devices just be made illegal entirely? It doesn't seem they can be made to be safe for average people if stuff like this keeps happening and companies are unwilling to compensate people. What if this was a lawyer talking about clients or a doctor talking about patients? Is Amazon (or Google, or Microsoft) willing to deal with such legal liabilities? Is this what move fast and break things looks like? Edit: To those trying to downvote, remember, the Silicon Valley lives in a bubble, and for the safety of people inside and outside of the bubble, the bubble must be poked every so often. This is one of those times.
  • dcbadacd 8y ago
    I think this tech is just in it's infancy, calling it inherently dangerous or saying it should be banned is silly. I think it will be done properly some time in the future, it's most likely going to be similar to the Windows vs. Linux vs. MacOS battle.
  • wooter 8y ago
    illegal? what a joke. if you don't want to take that risk, don't buy one.
  • function_seven 8y ago
    You ever butt dial someone? Should all cell phones be outlawed? I'd say the Echo needs to have a more difficult activation routine available. Maybe have the option of setting the wake word to "Alexa, can you please" or something similarly long and unlikely to appear in normal conversation. Also, I wonder if they had their wake word set to "Computer". I did that once, and quickly reverted after so many false activations.
• dep_b 8y ago
  My experience is totally anecdotal and I'm not even an Alexa owner, but I had a Skype conversation with a client that had one them set up in his office and it was constantly triggering, even when the word wasn't used. So if it though it heard "Alexa, send this message to John: {conversation of ten minutes}" it just did what it thought it was supposed to do. But it's weird they didn't hear any audible confirmation or anything.
• michael_j_ward 8y ago
  I received one of these messages just a week ago. Alexa sent me a message of my friend and his girlfriend having a private conversation. I immediately texted him to ask if he intended to do that and he did not- so weird.
  • nicky0 8y ago
    Can you explain a bit more. What was the context of the message? was it an email? What was the subject line and the text? Was the audio an attachment or a link? A link to where...? "Amazon sent me a message" is so frustratingly vague...
    • amhokies 8y ago
      You're able to send voice messages to other people that have an Alexa. Basically like voicemail.
      • p49k 8y ago
        That means that, unlike butt-dialing, you're more likely to accidentally send a message to someone you're actually talking about since the person's name is part of the command. Yikes!
      • nicky0 8y ago
        Is this something you have to actively set up or does it come enabled by default?
    • cdcarter 8y ago
      Alexa has voice messaging built in to the device/Alexa app. You receive a notification in the app and can ask your device to play it.
    • thefifthsetpin 8y ago
      Supposing that you're not tasked with resolving this problem for Amazon, why would those details matter?
• deleted 8y ago
  [deleted]
• amzn_engineer1 8y ago
  What actually happened: Alexa misinterpreted some voice commands and activated a "call" skill. The people involved and local news got very excited and escalated this into a conspiracy story. Amazon takes customer privacy EXTREMELY seriously. There's no way a team would get the "ok" to build a skill that randomly records private conversations then sends them to a random contact. It also doesn't make any logical sense to build such a skill. Yes, I might sound biased because I am an engineer at Amazon. This statement is my own and unrelated to Amazon's opinion.
  • function_seven 8y ago
    It's pretty clear to me that this was an inadvertent activation of the call skill. The equivalent of butt-dialing someone from your phone. I get random voicemails from people all the time that are clearly recordings of their phone in their pocket. If that's the case, then you need to provide additional features to reduce the chances of this happening. Longer and more unique wake word options, or more complex and deliberate confirmations before the call is placed. Something that a user can enable if they're worried about this sort of thing.
    • njstraub608 8y ago
      100% this. Seems like there should at the very least be a product feature that automatically disables any outbound messaging by default (similarly to how you can block purchases without a PIN). This doesn't solve the problem of always listening, but at least prevents this particular situation from inadvertently happening. Additionally you should be able to set an automatic deletion of voice recording after a specified amount of time. I'm sure some of this has to be in the pipeline with GDPR.
  • titzer 8y ago
    The obvious, predictable and serious screwup occurs and people shrug it off. Engineers from Amazon post how it's all a regrettable mistake and it won't happen again. I am not gonna unload on you specifically here, but if you could maybe pass along to whoever is making business and product decisions over there that yeah, maybe 80% are stupid enough to buy this crap, but there are a huge number of people that are extremely creeped out by this and avoid it like the plague. That deafening silence you here from them should not be interpreted as license to push the boundary even further. If there was an Alexa in my place of residence, I would rip it out of the wall, smash it to bits with a hammer, and fire it out of a cannon into the sun. Hell no to this crap from my side. I hope there are more incidents like this until people wake up to the fact they are putting telescreens in their homes. You guys can't be trusted!
• tapatio 8y ago
  People who use these "microphones" are idiots. They are right on up there with the people that give away their DNA to web companies.
  • mikeash 8y ago
    I’m one of those “idiots.” Can you explain how it’s any worse than carrying a cell phone?
    • fouric 8y ago
      Android has per-app microphone permissions that you can disable, if you want. At least in theory, Google apps honor these permissions, and on LineageOS, you can verify that this is the case - while with the Alexa, not only is there no way to audit any of the code running on the device, but it's explicitly marketed as being always listening.
      • mikeash 8y ago
        The baseband is a black box and has direct access to the microphone.
        • fouric 8y ago
          Oh, this is true - although the likelihood and ease of it being exploited as a listening device are much lower than that of a device explicitly designed to do so. Edit: Not to mention that the entire Alexa is a black box, too. Second Edit: Not trying to claim that you're an idiot, just provide some perspective.
          • mikeash 8y ago
            I suspect that the capability is already there, to be turned on at the request of law enforcement.
  • ryanwaggoner 8y ago
• parliament32 8y ago
  I feel that while these devices are great in principal, there still needs to be work done on whether the "trigger word" activation is the best way to enable them. Maybe we need longer / more complex trigger words? Maybe there's a different way we can activate them alltogether, avoiding the "always listening" problem?
• NVRM 8y ago
  But seriously what do you more or less expect from this. Anyone littlely involved in tech will tell to throw this shit out the window.
  • ryanwaggoner 8y ago
    I'm "involved with tech" (developer for 15+ years) and I have two of them. You might also be horrified to learn that my house contains multiple smartphones, computers, routers, and other devices, all of which are connected to the internet! I enjoy living dangerously.
    • NVRM 8y ago
      I am also a developer,living in countless universes, each enveloped in its own shell, compelled by the wheel of time, like particles of dust blowing about in the sky, and I don't understand what's wrong with you.
• deleted 8y ago
  [deleted]
• m3kw9 8y ago
  Hmm the AI model which is widely considered a undebugable black box glitches on a percentage of users. What else could happen next?
• jadedhacker 8y ago
  I feel creeped out by these home listening devices and I don't own one, but don't our phones already have this capability? You can turn "Ok Google" on on an android phone. I sometimes record audio, and the mic is incredibly good. Is there a substantial difference between our phones and these devices? EDIT: Just realized the substantial difference is that Google and Amazon own all of these things. They don't control all makes and models of phones.
  • twinkletwinkle 8y ago
    Not sure about Android phones with "Ok Google" but on iOS, all of Siri's voice processing is on the device. As opposed to Amazon Alexa which sends the data to the cloud for processing.
    • ryanwaggoner 8y ago
      I don't think this is true. I think they upload the audio to the cloud to process, but it's not linked to your user ID, only to the device ID.
    • millstone 8y ago
      Siri does speech recognition on Apple’s servers, but commands are performed on the device. The server doesn’t know who you are.
      • merinowool 8y ago
        Is there an independent audit that can prove that?
      • jadedhacker 8y ago
        If they have your voice, they can identify you regardless of metadata can't they?
  • mns 8y ago
    I got an Echo Dot a month ago, because there was a special offer on Amazon for the Philips Hue system, and it included one free. After reading this and having a discussion with my girlfriend last night about how she needs a backup of something, but we don't have a DVD writer at home (she's not tech savvy and thinks people still back-up stuff on DVDs), and now, the second day my Amazon daily offers is filled with portable DVD writers, it starts to creep me out and the first thing that I want to do when I go home is to unplug it. I know it can be a crazy coincidence and I was never the kind of guy that believed this Facebook is listening to you, but still, what if they actually just listen for some keywords?
• dustinmoorenet 8y ago
  I have this idea of a system I would like to have in my house. It contains cameras in every room that are constantly watching where people are and relaying the coordinates to a central server. That server makes decisions on if lights should be on or if A/C should be running in that room. But I would never buy this system. I would have to make it myself. I am hopeful that open source software and hardware can produce individual components that I can trust to piece together.
  • jasonkostempski 8y ago
    Even if you build it, knowing guest may never feel completely comfortable visiting your house.
    • fouric 8y ago
      They should feel more comfortable than an equivalent system built by a company that is looking to profit off of your data - and additionally, you can give the guest stronger guarantees that when you say that the system is "off", it actually is.
      • ggg9990 8y ago
        The chance that a random implementer has a security vulnerability is much higher than that Jeff Bezos is listening to me watch TV. A private system is more vulnerable to target attack and an Amazon system is more vulnerable to mass surveillance.
        • fouric 8y ago
          That's true - and I suppose that some people are much more concerned about the former threat than the latter.
        • bluGill 8y ago
          You are on the right track, but I don't think you are quite right. Amazon like systems are more vulnerable to mass surveillance. Your vulnerability isn't a targeted attack: you are not valuable enough to be worth the effort to figure out your system. As an attacker on your system I'd have to figure out how to break in, and then how to use the hardware you have. You are more valuable as part of a botnet - attacks that already exist. (if you are a politician then maybe, but that person is also vulnerable to a targeted attack on their amazon system - probably more so because the target is easier to figure out).
• mundo 8y ago
  I wonder how similar this is to https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf ? That's the "Facial recognition fooled by funny-colored glasses" study from Carngie Mellon, where researchers were able to make machine learning algorithms fail disastrously (e.g. mistake a man for Milla Jovovich) with a pair of glasses printed with what looks like a random assortment of colorful pixels, but is in fact a targeted attack specifically designed to trick the algorithm. This Alexa failure is obviously not a targeted attack; but when your system is exposed to enough data, eventually you will stumble over some input that happens to resemble a targeted attack by pure chance, right? It's equivalent to saying that if you aimed the facial recognition algos in that paper at millions of faces wearing randomly-colored glasses, eventually some of the glasses would be close enough to the targeted-attack glasses to produce the same effect. Obviously I'm theorizing on almost no data here - I don't know anything about Alexa's voice recognition, maybe it contains no ML at all. But it seems plausible that this might be what happened here - not a bug per se, but the natural and totally expected result of giving an opaque, machine-generated system with a very low failure rate so much input data that the failure rate is significant.
  • wilburTheDog 8y ago
    I recently stayed at a house which had an alexa device. In a conversation where I said the words light switch several times Alexa beeped and responded to me each time. I think most of us just believe Amazon when they say "Alexa responds to its name" and don't stop to consider the possible failure modes. We want to believe it can understand our words, when it's really just guessing. Over long enough time it's inevitable that it will misunderstand you and do something you don't want. Still, I wonder how it heard "record this conversation and send it to someone on my contact list".
• trumped 8y ago
  How did they receive the conversation? email? was the receiving address similar to some other address (the real recipient)?
• ddtaylor 8y ago
  It's okay because they take privacy seriously.
• titzer 8y ago
  These devices are all driven by machine learning voice recognition. I have no idea how I am supposed to derive trust in the software running on them when even the engineers themselves are arms-distance away from their machine models, and the machine models are tuned to terabytes of input data and are trained with reinforcement learning. I don't want to disparage engineers working on these products, but hell, we don't even know how big software systems really work anymore, and we've got neural networks thrown in the mix. We're all just plugging shit together until it limps these days. I vote no on telescreens.
• djsumdog 8y ago
  I've told my roommate I'm moving out if he ever buys an Alex/Google/Apple assistant device. I have a microphone and I've been intending to get one of the open source solutions working and just tie it in to mpd, weather and a few other things. But all the processing should really be done on your own device, by hardware you own, software that's open and that you configure, and not send up to someone else's computer (aka "the cloud"). Of course non-tech people probably wouldn't bother because that's a steep curve and you're talking about more expensive devices to do on-board processing. There are some companies that are trying to make this more accessible to regular consumers. I hope we see a move in this direction.
  • tejaswiy 8y ago
    Just out of curiosity - what open solutions are you referring to?
    • djsumdog 8y ago
      Mycroft https://github.com/MycroftAI https://github.com/MycroftAI Jarvis https://github.com/alexylem/jarvis https://github.com/alexylem/jarvis Jasper https://jasperproject.github.io https://jasperproject.github.io there are others if you look around
  • evv 8y ago
    > I've told my roommate I'm moving out if he ever buys an Alex/Google/Apple assistant device. Do you have an Apple or Google device near you right now? Does it have a microphone, battery power, and a connection to the internet? What about your roommate? Do you ever have sensitive conversations near these devices? Of course I'm not suggesting you should abandon such technology. I'm just wondering why you draw the line at "Alex/Google/Apple assistant device"
    • Bartweiss 8y ago
      This is actually a pretty interesting question about threat models. A lot of "it can listen" fears are inconsistent with "I carry a smartphone", but I do think there's a meaningful distinction. My expectation isn't secrecy, but it is privacy. I carry a smartphone, but I keep voice-command activation turned off. I'm sure it could be activated remotely, but I expect that would require a targeted effort. I assume that Google isn't constantly recording everything in the vicinity of all Android phones - the news would probably get out, and honestly I'd see the battery impact pretty quickly. As the saying goes, "Quantity has a quality all its own". My basic objection is to systems that are likely to store/upload all available data, with the attendant risks of leakage or commercialization I object to. Anything I say around my phone is potentially forfeit, but I don't think everything is.
• merinowool 8y ago
  Wouldn't that be a crime to record someone without consent and share that to a 3rd party? I would have called the police or is Amazon too big for the law to grasp?
  • macintux 8y ago
    That's an interesting point. In California and some other states, having a device actively recording any conversation without consent from everyone involved is illegal, much less sharing it.
• gshakir 8y ago
  Even though I like Apple stance on privacy, I always turn off microphone access to apps like 'WhatsApp' and others.
• thefifthsetpin 8y ago
  Google let me listen to my own queries following, "okay, google." Mostly they were reasonable, but sometimes it heard "okay google" in what sounded to me like regular driving noise. It's a good reminder that even though we've trained computers to recognize sounds and images, we didn't train them to listen & see the way that we do. The mistakes that these algorithms make need not be anything like the mistakes that a person would make.
• rm_-rf_slash 8y ago
  This is why I unplug my Echo whenever I’m not using it. Side note: why is it that after years and years of privacy breaches, there has been almost no support to use hardware switches for cameras and microphones? I would feel 100000% safer using my internet-connected cameras and microphones knowing that I can turn off those devices independently of their host devices and that no hacker can monitor my cameras and microphones, even if they’ve rootkitted every device I own.
• ReedJessen 8y ago
  I am very skeptical of this person's claim.
• bmans94 8y ago
  It's absurd to me that someone would wire up "every room in their house" with internet connected microphones (in a technology that is still in it's infancy!) and then claim you felt like your privacy was "invaded". At best, it's a Trojan Horse, but the naivety here is astounding. Of course your Echo shouldn't send out private conversations, and it looks like a legitimate bug occurred. But man, the buy-now think-later mindset some people have is worrying.
  • saas_sam 8y ago
    You literally carry an audio & video recording device with a GPS chip and multiple wireless attack vectors on you for nearly every waking hour of your life.
    • bmans94 8y ago
      Of course I do. But that's not my point. They wallpapered their house with Echos! And now they are 'shocked'. When I get home, my phone usually gets tossed on a charger in my room, and forgotten about (while checking for messages every once in a while). And my phones sole purpose, while capable of it, is not to listen to everything I say and interpret voice commands. I never use Siri, and I would be 'shocked' at this point if it sent out a message, that I didn't intend, without alerting me. It's a older, tested technology (in my opinion). I'm not surprised that my GPS capable phone knows where I am at all times. I wouldn't be surprised if my Echo recorded my voice and sent a message to someone. I wouldn't be surprised if there were some major bugs yet to be addressed in the software for the Echo. I guess this is a little harsh, but my sympathies are somewhat lost on these people. Not their fault for what happened, but they did take a risk. "A husband and wife in the privacy of their home have conversations that they're not expecting to be sent to someone (in) their address book," It just doesn't sit well with me.
      • saas_sam 8y ago
        > I would be 'shocked' at this point if it sent out a message, that I didn't intend, without alerting me. Yep. So was couple. About Alexa. Same thing.
• jefe_ 8y ago
  I imagine there are a number of normal conversations that can trigger unexpected behaviors from these smart devices, especially when you consider the thousands of different regional accents, along with the thousands of ways of speaking the same words with various speeds, pitches, and annunciations. The article mentioned they were talking about wood floors: ...Oak in the annex and for the record... recording begins ...it's just the annex, stop fretting... recording stops just send [flooring person] a quick message with samples you like. sends recording to someone with same name as flooring person It sounds like the problem is that it was overly confident it understood the user's intentions, and did all of this silently.
• leroy_masochist 8y ago
  Tangential point, but you would think that Amazon would have immediately given this woman the refund she asked for, rather than instead offering to take her through a technically complex de-provisioning process. It just makes them look even worse during the inevitable bad publicity that they had to know they were going to get from this.
  • deleted 8y ago
    [deleted]
• ComodoHacker 8y ago
  >"The person on the other line said, 'unplug your Alexa devices right now,'" she said. "'You're being hacked.'" Perhaps they were indeed hacked. Got into test group for some black hats project.
• yason 8y ago
  I am oversimplifying here but: * install device that is designed to listen to speech in the house * the device is connected to internet * the device is capable of contacting other internet peers/services/hosts * the device knows a bit about its owner's internet presence such as contacts * the device is equipped with simple conversational user interface based on fuzzy human speech-command detection These basically bound the failure modes of the complete system. I am not surprised that a glitch like that happened. I can certainly attest to how shocking it might be to discover first hand that it really did happen, but given the context I can't say it's hard to not foresee something like this.
  • tj-teej 8y ago
    If you replace "fuzzy human speech-command detection" with "buttons which can accidentally be pressed" - how is this different from butt-dialing?
    • zymhan 8y ago
      I don't think there is much of a difference. People took a while to understand butt-dialing, and not do it as much. It'll take a while for people to get used to voice assistants that trigger seemingly randomly.
      • sdenton4 8y ago
        AFAIK, pocket dials have mainly been reduced by improvements in phones, rather than changes in user behavior; if the camera just sees inside of a pocket, it's unlikely to be an intentional button press.
        • zymhan 8y ago
          I mean way back in the day when people had candy-bar phones and had to remember to enable the keypad lock :D
          • EvanAnderson 8y ago
• gesman 8y ago
  >>>But Danielle is hoping Amazon gives her a refund for her devices, which she said their representatives have been unwilling to do. She says she’s curious to find out if anyone else has experienced the same issue. If that's true - that dumb greedy move is going to cost AMZN way more than simply giving customer refund.
• jonjojr 8y ago
  "My husband and I would joke and say I'd bet these devices are listening to what we're saying," Think about this statement for a sec. Of course is listening to you. It has to listen to you because it needs to be able to respond to "Alexa". So yes, this is always listening. Now, once you come to terms with that, do you feel comfortable having a technology with access to the internet, location, habits, account, contacts, email, phone, CC information etc etc, to be actively listening to your most intimate private conversations? It does not take a 5 year old two seconds to figure this out.
  • superkuh 8y ago
    People make the same jokes about cell phones and location tracking. But it seems like with these invasions of privacy they're always unable to make the leap from joking about it to the reality itself.
  • wilsonnb 8y ago
    The problem with this logic is that it applies to phones, tablets, and laptops as well. There isn't much additional risk from having an Echo or a Google Home if you already keep a smartphone within 10 feet of you at all times, which most of the people who are buying these devices do.
    • pessimizer 8y ago
      Why is that a problem?
• Bhilai 8y ago
  I am seeing comments saying, Alexa misinterpreted some voice commands and activated a "call" skill. This sounds like a good excuse but does absolve Amazon of responsibility. If their voice and command recognition is broken they should add second step verifications like "Are you sure you want to call X" or "Okay, I am calling 'X' in 10s" or something similar. Crazy to imagine Amazon is shipping this device with capability of video calling people and buying stuff online without having fully testing and thinking about all scenarios. If there is even a slightest possibility that Alexa/Siri/Google/Cortana is going to misinterpret commands (with privacy implications) then they should do two step verification of some sort.
• randyrand 8y ago
  How does alexa get your contacts? I don't think I've given it mine.
• draw_down 8y ago
  It's crazy to use these things at the moment, in my opinion. Maybe in a couple years the failure modes and ways these things can be used to terrible ends will be better understood. But for now, it's not well-understood how many things could go wrong with an internet-connected always-listening microphone that knows all your contacts and can buy things.
• jacksmith21006 8y ago
  Not sure why people would be buying the Echo over the Google Home any longer? The GH offers a much better UX and now is well ahead of Alexa in sales last quarter. "Google takes top spot in global smart speaker market, HomePod nowhere to be found" https://appleinsider.com/articles/18/05/23/google-takes-top-spot-in-global-smart-speaker-market-homepod-nowhere-to-be-found https://appleinsider.com/articles/18/05/23/google-takes-top-... Came to the market 2 years later and already passed. This is before we even get Duplex. https://www.youtube.com/watch?v=bd1mEm2Fy08 https://www.youtube.com/watch?v=bd1mEm2Fy08 Google goes in the lead on something they tend to not give it up.
  • ocdtrekkie 8y ago
    Uh, dude, I know being a fan of Google is your thing here, but this was less than a year ago: http://money.cnn.com/2017/10/11/technology/google-home-mini-security-flaw/index.html http://money.cnn.com/2017/10/11/technology/google-home-mini-... And it's a flaw so bad they had to outright remove the button entirely, because they couldn't fix it without replacing the hardware. It's a little disingenuous to insinuate a bug with recording audio is a reason people shouldn't be buying company X's product over company Y's, which also has had a bug with recording audio.
    • jacksmith21006 8y ago
      Google quickly fixed the issue caused by the person putting it in the bathroom and causing the button to stick. The person that it happened to got a pre-release version. Google even went to the persons house on a Friday evening to resolve. The person found the problem on the same day. None of the units sold to the general public had the issue. Realize Google literally dropped everything to take care of the issue. Now that is what you want to see. The reason you buy a Google Home is because they provide a much better user experience and why Google has already passed Amazon even though they did not get to the market until 2 years later. "Google takes top spot in global smart speaker market, HomePod nowhere to be found" https://appleinsider.com/articles/18/05/23/google-takes-top-spot-in-global-smart-speaker-market-homepod-nowhere-to-be-found https://appleinsider.com/articles/18/05/23/google-takes-top-... We started with an Echo and then purchased the Google Home and then several more Google Homes. My son had graduation earlier this week at an arena that has a name that is difficult to pronounce. It is named after someone that gave a lot of money to the University. There was also a MLB game that night that started 15 minutes later than the graduation. Walking through the kitchen asked drive time thinking no problem as people are leaving the city usually not going to the city. Google got it the first try and gave drive time which was 2x more than normal. Google saved my butt as quickly got ready and left and would not have otherwise. Asking our Echo and Siri the same question and neither could handle. That is why you buy the Google Home. Well also because of new things like Duplex coming. https://www.youtube.com/watch?v=bd1mEm2Fy08 https://www.youtube.com/watch?v=bd1mEm2Fy08 Google is just so far ahead in every layer of the AI stack and just going to be able to provide a far superior product.
• PeterStuer 8y ago
  Obligatory https://www.youtube.com/watch?v=sAz_UvnUeuU https://www.youtube.com/watch?v=sAz_UvnUeuU
• myself248 8y ago
  Back in the early 90s, I remember seeing a cartoon in one of the tech magazines, possibly BYTE or PC/Computing. It featured an elevator opening onto a floor of office cubicles. The guy in the elevator screams "COMPUTER! FORMAT SEE COLON SLASH WHY ENTER" across the whole cube farm. The doors close, and the elevator is gone. Caption was something along the lines of "Speaker-independent voice recognition might be a bit tricky."
• gricardo99 8y ago
  The most shocking thing about this story is how a customer was able to get qualified engineering support for an unusual support request, from a large company. Or... what's missing is the hours spent going through support tiers 1 through N, and somehow not getting dropped/lost along the way.
• rhacker 8y ago
  Same thing with my Android phone. When I am in an area without internet access, but phone-calling signal (at least) the dumb ass phone defaults to using an internal low-quality OK-Google speech recognition. So instead of looking up hours to a nearby restaurant I look at my phone and it is suddenly calling someone in my contact list (that I haven't really talked to in years)... and of course since the speech recognition blasted the CPU's horsepower, nothing on the screen is responsive and my thumbs are constantly slamming the screen with no feedback - the calls goes through for about a second or ringing before my phone decide to hang up fully.
• amingilani 8y ago
  I turned on recording indication sounds on my Google home in accessibility options. Now I hear it beep and randomly start listening when my wife and I talk. Sometimes it does it randomly without me making a sound. I ask it what's up and it ignores me and goes to sleep. Smart speakers are so dumb I'm hardly surprised by this.
  • dmix 8y ago
    Yeah Amazon said there should have been an audible warning it was recording and sending the message. It's apparently just a 'rare' glitch. Based on Amazon's response I assume the device didn't notify the user at all and it wasn't just a matter of not hearing it.
• blunte 8y ago
  The chances of this happening all the time are unknown but believable. We as tech users are constantly trying to balance convenience with safety
• stevenhubertron 8y ago
  This just seems so unlikely. How was it sent? Was a MP3 file generated and attached to an email? Was it a voice call? I don't really understand a circumstance that this could have happened.
• deleted 8y ago
  [deleted]
• IronWolve 8y ago
  And Bank of America is now pushing its Digital Assistant, Erica. Just what I need, another robot to talk too... https://promo.bankofamerica.com/Erica/she-is-here/ https://promo.bankofamerica.com/Erica/she-is-here/
• bootsz 8y ago
  Every day I become more convinced that we've really let tech get a little too out of control, and could probably benefit from putting the brakes on a bit to get some very critical stuff under control first (e.g. security/privacy). Don't get me wrong, technology is amazing and the cutting edge stuff going on today is super exciting... But it seems the capabilities of software now are far, far outpacing our abilities to ensure an adequate level of security and personal privacy. We've been blinded by flashy tech, too busy being amazed at everything all the time ("look at what AI can do!") and not stopping to consider the huge hidden price we're paying. The reasons this has happened are kind of obvious... I just fear that it's going to take things getting to a point where we have a really major catastrophe on our hands for anyone to be willing to really do anything about it. A lot of people can and will get seriously hurt. I can imagine such an event resulting in a great "cooling" period in tech where advances are slowed while bigger investments are made into security, cryptography, solving the "identity problem", etc. But it would be nice if we could just have the foresight to fix things before it gets to that point...
• stefanwlb 8y ago
  The exact same thing happened with Ok Google a year ago, and I will repeat the same thing again for those who are able to hear. This happened due to a bug, except not the way you would think. The only bug was that the recording was sent to an anonymous user instead of the Amazon HQ. Don't trust them.
• tschellenbach 8y ago
  I have 3 google home devices and they work very well. It also doesn't feel like a bigger privacy threat than my laptop or my phone. Also, we all want JARVIS!
• IkmoIkmo 8y ago
  I recall a few weeks ago how some researchers figured out that the voice-commands could be triggered with subliminal messages, too. i.e., you wouldn't have to go 'Ok Google', but rather just play a fingerprint of that sound that makes the algorithm think you said that, without it being audible or understandable to human beings. And you could hide it in other audio like music. That feels pretty scary particularly because these devices have such a high mandate. Right now you can shop for things online by voice command. Record and send messages. What's next, sending money, sending a data dump of sensitive data like emails, passwords, contact lists? We've already heard of some reports where you'd have a smart home device listen to a commercial on television where they were demo'ing a purchase of some product via such a device, and interpret it as a command from the owner. Amazon took steps to avoid that with their superbowl commercial, but it seems to have done so by changing the commercial, not the product itself. Just another attack vector to worry about. I'd happily buy these home speakers if I could limit them to just downloading information, and only uploading limited pieces of information, e.g. a music playlist. I'd want to be able to shut off any of the commercial/financial or social capabilities. I just don't care about them and they're risky.
  • kevingrahl 8y ago
    Do you happen to have a link to said research? I’d be curious to fiddle around with that.. for educational purposes of course..
• mandeepj 8y ago
  Please help me in understanding this - are these features built on purpose? If no then what type of testing has their team done?
• mihaela 8y ago
  I do see a dveice like that useful in some scenarios. Like a lonely elderly person, living alone.. Talking to Alexa might be nice.
• gnicholas 8y ago
  Amazon's response, from Ars' article: > Echo woke up due to a word in background conversation sounding like "Alexa." Then, the subsequent conversation was heard as a "send message" request. At which point, Alexa said out loud "To whom?" At which point, the background conversation was interpreted as a name in the customers contact list. Alexa then asked out loud, "[contact name], right?" Alexa then interpreted background conversation as "right." As unlikely as this string of events is, we are evaluating options to make this case even less likely. https://arstechnica.com/gadgets/2018/05/amazon-confirms-that-echo-device-secretly-shared-users-private-audio/ https://arstechnica.com/gadgets/2018/05/amazon-confirms-that...
  • RyanShook 8y ago
    Seems like a pretty big stretch...
    • bisby 8y ago
      I mean, if this was a one in a million chance of happening, and there have been billions of conversations happening near devices like these, I'm surprised this hasn't happened more often. If you have an echo and it kicks on unexpectedly, you can go to the app to see what it thought you said, along with a recording of you saying it. I've had "alright, uhh..." trigger it. and on the grainy bad across the room recording, I could see how the computer thought it was "Alexa". So I could see how if the computer is purposefully listening for "right" or "yes" that it could mistake any number of other words for that. I would much prefer an open source + open hardware alternative that works reliably, but I received an Echo dot as a gift, so after a few months of debating, I plugged it in. Having an always listening internet connected device that has access to your contact list is a pretty big stretch. But somehow we go along with that just fine.
      • jacobsenscott 8y ago
        Given the numbers it has almost certainly happened at least hundreds of times and probably thousands of times. But the remaining sequence of events - the receiver notifies the sender, and a media outlet picks it up, and amazon comments seems to have only happened once so far.
• mdeeks 8y ago
  My Echo somehow woke up when we were in another room, then heard itself talking to itself, and called my friend in the middle of the night. You can see the log of what it said and how it heard the last word as the command: https://www.facebook.com/mike.deeks/posts/10215464075417775 https://www.facebook.com/mike.deeks/posts/10215464075417775 It was both hilarious and infuriating. I immediately turned the calling feature off (you have to contact support btw) and later we switched to Google Home.
• madrox 8y ago
  This is something I don’t believe we’ve really figured out in terms of QAing stochastic processes. Doing QA on normal code is binary...did it work or not? Doing a QA on a model is a lot more difficult. You need to think about Type I and Type II error. If I were a product manager thinking about wake words, I’d want my Type I error to be as small as possible...to the point where I’d accept a pretty high Type II error. Lately I’ve been seeing Alexa wake on accident more often. It’s like someone isn’t thinking critically about what kind of Type I error they’re willing to accept.
• 8bitsrule 8y ago
  Really, folks... we can't even get the net to work the way we want, but then we heap these enormously sophisticated, delicate tasks onto relatively-untested, non-deterministic warez, and pray all goes well. Much like running into a human on a bike, but less deadly. (Unless there's a psycho in my contact list.)
• tinycakes 8y ago
  It seems like an easy safeguard to implement to prevent sending unwanted emails/texts would be for Alexa to generate a unique confirmation phrase that the user has to repeat back in order to actually execute the send command. Something like Alexa: "Say 'purple people eater' to send email." Real Person: "Purple people eater." Alexa: "Email sent."
• yani 8y ago
  I am so happy with GDPR right now
• deleted 8y ago
  [deleted]
• guilt 8y ago
  This is what I think: https://imgur.com/a/EY1mc5L https://imgur.com/a/EY1mc5L
• andreyk 8y ago
  The explanation for this is fairly unremarkable (https://techcrunch.com/2018/05/24/family-claims-their-echo-sent-a-private-conversation-to-a-random-contact/ https://techcrunch.com/2018/05/24/family-claims-their-echo-s...): Echo woke up due to a word in background conversation sounding like “Alexa.” Then, the subsequent conversation was heard as a “send message” request. At which point, Alexa said out loud “To whom?” At which point, the background conversation was interpreted as a name in the customers contact list. Alexa then asked out loud, “[contact name], right?” Alexa then interpreted background conversation as “right”. As unlikely as this string of events is, we are evaluating options to make this case even less likely.
  • titzer 8y ago
    Actually it's very remarkable on how bad the voice recognition is, and how Amazon can knowingly ship this crap to people, putting it in their homes, recording their private interactions. Ya'll recorded people having sex yet? It sounds like the machine learning models are tuned for specific phrases and have terrible, terrible false positive rates. "Evaluating options to make this less likely"? Now I don't even know what mental model you guys are using (if you are from Amazon). This shouldn't be a matter of "likelihood". But OK, fine, let's use math. AFAICT there is about a 0% chance that a human would make such a string of errors interpreting human speech, but let's call it 1 in a trillion. If you guys aren't doing better than about 1 in a trillion for this string of at least 4 interactions, then one of those terms is stupidly, stupidly high.[1] It actually interpreted random conversation as the name of someone on a contact list? Horrible. trust-- [1] Not even to mention, a human has context, understands boundaries, preferences, and has a ML voice recognition model developed and tuned over decades of interacting with real people. A human would also be smart enough to understand the other human's situation, context, state of mind, and realize that even just the cadence of the conversation not changing in response to queries was indicative of the humans not acknowledging the query. Machines are f'in stupid.
    • andreyk 8y ago
      It's a 1-in-a-million edge case.... I would hope people on HN at least would understand this stuff happens in software.
• crazy_monkey 8y ago
  Why is anyone surprised? We're talking about an extremely super terribly ethical mega-corporation here... There ought to be monetary penalties each time something like this happens for them to proactively care.
• aphextron 8y ago
  Is this a surprise to anyone? That was literally my first thought the moment they announced these things.
• binnesjohn 8y ago
  I will never have these cia nsa sponsored mics in my home. Fact ! I do not trust amazon, google, facebook or absolutely any organization anymore.
• TheSpiceIsLife 8y ago
  "He told us that the device just guessed what we were saying," she said. Danielle said the device did not audibly advise her it was preparing to send the recording, something it’s programmed to do. Is it possible there's an exploit out in the wild for these devices that allows an attacker to control the device remotely. Is it possible an attacker told the device not to make an audible announcement?
  • wepple 8y ago
    Possible, technically, of course. But if someone is going to exfil voice, why did they send it to a random contact who would then alert the owners?
  • sleepychu 8y ago
    Yes, but it's improbable. You can communicate with Alexa at human inaudible frequency and instruct it to set it's volume to 1/10(0/10?) but of course you need to be reasonably local and targeted. Is it possible that there's a remote exploit? Yeah of course but if it's in the wild enough that this particular individual was struck by a seemingly random (and valueless) attack we'd be seeing a lot more of these in the wild than we are.
• 2T1Qka0rEiPr 8y ago
  Question - how can these potentially nefarious acts be mitigated, where I plug a device into my home network which listens/watches "periodically". I guess naively I'm thinking some sort of light/display which says "Amazon is sending/retrieving data" - it would need to be from a third party to have integrity, and I suppose in order to actually allay the fears of the masses it would need to be plug-and-playable. Does such a thing actually exist?
  • faitswulff 8y ago
    You could always set up a dedicated router for your smart home devices if you really wanted.
• ataturk 8y ago
  All devices with mics and cameras should come with physical switches to disable them. Even a couple of DIP switches on the back of the device would be fine.
• mabbo 8y ago
  I was chatting with my wife about something the other day and my android phone, from my pocket, randomly added to the conversation "That's good to know". O_O
• albi_lander 8y ago
  This makes private-by-design alternatives like Snips (https://snips.ai/ https://snips.ai/) even more legit. When everything is processed locally, the user does not have to trust anyone/anything.
• perseusprime11 8y ago
  I am trying to think what kind of architecture, code design could have led to such a defect if indeed it's a defect. My guess this is a feature that got triggered by an easter egg.
• alistproducer2 8y ago
  I've said it before, but this solidifies it for me - I'm never installing a internet enabled hot mic in my home. I'd rather spend the 2 seconds finding the song on my phone.
• fusl 8y ago
  I don't see where Amazon would be at fault here. Remember back in the old days of T9 keyboard phones, remember that one time where you forgot to lock your keyboard when you put it into your pocket and it called a random person from your contact list? You didn't immediately call the press to say it was Nokias, Ericssons, Siemens' (or whatever your phone manufacturer is) fault you left your phone unlocked and it called a random person because of the hardware keys being pressed within your pocket.
• patsplat 8y ago
  Voice is not for personal computing. It is best for social computing. This is a completely new metaphor for interactivity, and problems are to be expected. This incident speaks more to Amazon's speech recognition rather than a weakness of voice. Am not an expert in the topic, but it certainly seems like Alexa tries to hear patterns rather than translating speech to text then text to commands. Google Home's recognition is much better.