17 ms·
Alexa and Siri Can Hear Hidden Commands
- NyrIsNotDead 8y agoAnything with a mic attached has the same problem, why is it special if it’s something from Amazon or Apple?
- TeMPOraL 8y ago1) You have no control over it, a third party in the cloud does. 2) Scale of deployment.
- CurtHagenlocher 8y agoMost things with attached microphones aren't using them as a command port.
- ocdtrekkie 8y agoThe main concern would be that these voice assistants are designed to auto activate on that audio and can do everything from make purchases for you to activating devices in your home.
- Jun8 8y agoThe interesting thing is that Google Assistant has teh same problem, it's right there in the subtitle. It's interesting that it was omitted from the main title. Rather than journalistic oversight I think this verifies what people have commented many times: that the fact that GA does not have a personalized name makes it to refer to it. SO much so that a very distant third product is included rather than GA.
- baxtr 8y agoMaybe the main title would be too long if it included “and google’s voice assistant”
- GW150914 8y agoThis was the title I used, it must have been changed: “Alexa and Siri Can Hear This Hidden Command. You Can’t.” I couldn’t have added the subtitle because as you say, it would have been too long.
- osteele 8y ago"Alexa, Siri, and Google Home, can hear inaudible commands" "Alexa, Siri, and Google Home, can hear commands that humans can't"
- srtjstjsj 8y agoThat's less clickbaity than "This Hidden Command" and "You can't." that are specifically designed to hack your psychology and make you feel uncomfortable and threatened by not knowing what they are referring to.
- GW150914 8y agoI agree with you and Osteele, but I didn’t feel it was my place to editorialize the title, but I’m happy for the moderators to do it.
- lozenge 8y agoUsually the attack requires the source code (or weightings of the neural network), I'd be surprised if they are able to actually attack these systems.
- ninguem2 8y agoCan't you just "machine learn" the attack?
- saagarjha 8y agoDoes it really? As far as I was aware, it is still possible to perform a black box attack without knowing the weights of a network. Using specially crafted input, it's even possible to "steal" weights from a network!
- srtjstjsj 8y agohttps://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/carlini https://www.usenix.org/conference/usenixsecurity16/technical... "We evaluate these attacks under two different threat models. In the black-box model, an attacker uses the speech recognition system as an opaque oracle. We show that the adversary can produce difficult to understand commands that are effective against existing systems in the black-box model. Under the white-box model, the attacker has full knowledge of the internals of the speech recognition system and uses it to create attack commands that we demonstrate through user testing are not understandable by humans."
- dirkgently 8y agoAnd ".... hidden commands that are undetectable to the human ear to Apple’s Siri, Amazon’s Alexa and Google’s Assistant." And then "digital assistants like Amazon’s Alexa or Apple’s Siri are set to outnumber people by 2021". Why is Google mentioned in one context but not other?
- tinus_hn 8y agoDark times are ahead for Assistant. You must warn her of your discovery!
- ColanR 8y agoThis kind of thing has been on HN before...the new thing here is that the command is embedded in a human-audible sound clip.
- newsbinator 8y agoI can't get Siri to turn on the flashlight or lock my phone. What can Siri do that's dangerous?
- exodust 8y agoSend emails is one example. I don't use Siri, but apparently it can send emails including to multiple recipients. The "danger" is only limited by your imagination in the scenario where a malicious stranger has access to your email client.
- newsbinator 8y agoI guess you could tell Siri, > Send an email to my mom that says, "I have an emergency and I need $2000. Here's the account number to send it to: 12345. Mom, please don't ask questions. This is urgent. Send the money now." Would Siri lookup your mom's email and send that?
- exodust 8y agoI have no idea, but a "smarter" email would be "mum can I borrow $100, pay you back ASAP, just a bit short today sorry and thanks!"... Mum would be less inclined to phone you in a panic.
- calimac 8y agoYes, Siri knows family
- Infernal 8y agoIf you have previously identified one of your contacts as your mother, yes. If not, Siri will ask who your mother is and if Siri should remember that piece of information.
- Infernal 8y agoTagging on to say I just tried this - even though I activate Siri without unlocking, I was asked to unlock my phone to continue with the email.
- exodust 8y agoI hadn't thought of this, it's quite concerning. I don't see how they can safeguard against this without reducing the effectiveness of the voice recognition. A secret command to "paste clipboard into new email, send to [address]" is a shiny new attack vector without any apparent straight forward way to plug the security hole.
- djrogers 8y agoThe obvious plug would be to not allow such a ridiculously unsafe command without requiring you to unlock your device, much like my iPhone does today.
- exodust 8y agoSure, but your phone is sometimes already unlocked because you used it 30 seconds ago and it now sits on the table. Or it's playing music, or your kid has it etc. I don't think I was thinking about a phone anyway, more the dedicated devices that sit there listening all the time.
- DecoPerson 8y agoSo lock your phone every time you set it down. Never leave it unlocked. I used to have my iPhone lock 5min after I pressed the sleep button. Now that TouchID makes it very easy to unlock, I have it locking immediately. When I let my friend's 4yo use my iPad, I triple tap the home button and press "Guided Access", which can prevent the user from accessing other apps until I disable it. (I do this because I'm worried about what he may accidentally search on the web, not because I'm worried he'll steal my data!)
- kalleboo 8y agoSiri is supposed to be tailored to your own voice and not accept commands from anyone else. Sounds like they need to approve that fingerprinting. (or this different on the HomePod since it's supposed to be used by multiple people?)
- bsenftner 8y agoHidden audio is simply too easy. Hidden audio is the knife that kills desire for any financial services access through a voice assistant - for those smart enough to not follow the horde.
- Shivetya 8y agofor me it kills all desire to have a voice assistant. I am already in the camp of taping over the camera's in my computers now will I need to worry about the microphone or what comes from the speakers? so the question is, shouldn't they be able to detect the wavelength of what they are processing to weed out some of the more obvious tricks? with voice recognition could it also not be limited to a voice it is trained to know?
- djsumdog 8y agoI don't bother taping over stuff. If you think about it, there are probably 10+ microphones in your room (Samsung TVs, phones, laptops, tablets, etc.) I run 3rd party roms, Linux on all my dev/tv machines, disable Cortana on my gaming laptop, and hope there isn't something listening in all that trusted, untrusted and oss code I'm running. I told my roommate I'd move out if he ever got an Alexa or Google home device. I do want to run Jarvis, or one of the OSS alternatives. Many of them send your data to Google/Amazon as well if you enable using their Speech-to-Text services, but they also have options for using local OSS decoders as well (and typically enable those by default). Our phones are so powerful today there is no reason to send your speech to the cloud (someone else's computer). It should just be done locally; and tech should be improved so accuracy is improved locally without needing the larger datasets that Google/Amazon/Apple use. More devs need to use the OSS assistance instead, and maybe that will push other engineers to no go the easy route and opt to protect their privacy instead.
- freehunter 8y agoYou should bother taping over cameras you're not using though. It's way too easy to hijack them and keep the light from turning on when you do.
- banku_brougham 8y agoSerious off-topic question: Are there docs for Siri so that i can learn what it can/can’t do? I have tried skipping songs, playing a genre, set random play, and similar in iTunes — generally a failure, often initiates an unwanted phone call. On the phone I can successfully call the intended contact about 50% of the time, possibly because I have ~250 contacts. I suspect that if i knew the right words to interact with the API I could have a more enjoyable Siri experience. Alternatively, is there a way to disable it completely — as in long hold on headphones button does not initiate.
- hunter2_ 8y agoThis doesn't address your question but might be interesting. As someone who has used Android forever and never tried Siri, those anecdotes are mind blowing to me. For me with Google Assistant, media commands work about 80% and successful call initiation is about 90%. And I haven't looked for documentation either. YMMV of course.
- rayiner 8y agoSo a simple media command fails 1 in 5 times? This is the best SV can do with a incredibly well-studied technology (voice recognition)?
- hunter2_ 8y agoWell, I try new things all the time. For example, sometimes it finds an artist or song whose name contains the genre I was trying to play. I could approach 100% with "hey google, play" when something is paused (but not "hey google, stop" when it's playing, because of the ambient noise of whatever is playing making the wake word fail).
- matwood 8y agoAs someone who has used both, the problem with Siri is when it works it's great, but it will fail on the exact same command the next time you try it. Consistency is key in these things working.
- 8y ago
- devy 8y agoJust out of curiosity, what's the frequency range that a typical mic can pick up the signal from? The article did not specially mention about the range instead it said inaudible. And here is another article I found that mentions the normal 20-20kHz frequency response range: http://blog.shure.com/mic-basics-frequency-response/ http://blog.shure.com/mic-basics-frequency-response/ Isn't that mostly overlap with the human ear capability? I understand each person is different, etc. But just curious the specifics.
- hunter2_ 8y agoEvery model will be different, but the important thing is that the boundaries represent the frequencies within which the signal will stay above a particular threshold of amplitude. A good spec sheet will tell you that threshold, and I've seen things like -3, -6, or -10 dB. It will still pass audio outside of the range but at an undisclosed attenuation.
- jerf 8y agoYes, typical mics tend to pick up the typical human frequency range, though cheaper mics may have some really poor characteristics at the edges. Usually in the speech range they'll be pretty solid. However, there's a lot of play within the space. One difference is that microphones do a very direct recording of the sound waves, but what we hear is actually very distorted compared to the "real" sound by the nature of our ear. One of the big differences is that if there is a very loud 4000Hz sound, we can't hear a soft 4005Hz sound near it very well, but the microphone "hears" it just fine. So for instance, you could put out a loud sound for a user, but embed a very quiet command in frequencies the human couldn't hear, but if the listening model doesn't account for that (and there are reasons it wouldn't necessarily want to, because it wants to hear commands even in the presence of significant background noise), you could get commands in to a system. See https://en.wikipedia.org/wiki/Psychoacoustics https://en.wikipedia.org/wiki/Psychoacoustics for discussion about how our ears fail to pick up the "real audio" signal, and how much we've exploited that in music compression. Now, that was a very brute force example. It sounds to me like what this article is talking about are called "adversarial examples" (https://blog.acolyer.org/2017/02/28/when-dnns-go-wrong-adversarial-examples-and-what-we-can-learn-from-them/ https://blog.acolyer.org/2017/02/28/when-dnns-go-wrong-adver... ). Voice recognition doesn't listen the same way we do, it doesn't necessarily take a holistic view of the signal, but is looking for specific frequency patterns and changes and turning that into phonemes, into words, etc. (There's a lot of ways of doing this and I don't specifically know what Alexa and Siri are doing, so that's a really vague overview.) If you know what they are looking for, you can use filters to very, very selectively remove the patterns from a bit of music or something that Alexa might trigger on, and then insert just the bare minimum skeleton of the sounds that it is really recognizing. A human won't be able to hear the difference (most likely; depends on how badly the original is mangled but even if it is audible it is almost certainly not audible without an A/B test and very good ears), but the probably-neural-nets monitoring for sounds will end up superstimulated and interpret the adversarial example as words. While the adversarial examples work best with tuning to the target network, widely-shared networks like Alexa or Siri mean that such tuning is practical where attacking some custom-trained model used by one person isn't, and experiments have shown that adversarial examples travel between separately-trained nets and even non-neural-net models to a much, much greater degree than what at least my own intuition would have suggested before hand. (See previous link and look for the discussion of "Practical black-box attacks against deep learning systems using adversarial examples". It is extremely counter-intuitive to me how easy this is.)
- coleca 8y agoIsn't there an inaudible tone that you can use to disable the assistant? I recall reading somewhere that Amazon used it in their commercials for Alexa so that everyone's Echos weren't lighting up during the commercials. I know when a commercial for the Echo comes on and the voices repeat "Alexa, do X" the Echo I have near the TV speaker doesn't light up.
- adambowles 8y agoIf I recall correctly, the adverts omit a certain frequency range from the assistant's invocation phrase which human's won't notice is missing Edit: Yep, omit / reduce tones in the 3000 - 6000 Hz range https://www.reddit.com/r/amazonecho/comments/5oer2u/i_may_have_found_how_amazon_prevents_the_echo/ https://www.reddit.com/r/amazonecho/comments/5oer2u/i_may_ha...
- SlowRobotAhead 8y agoWhich makes WAY more sense than a range outside of human hearing that OP implicated ("inaudible"). Expecting TV's to play tones outside the range people can hear is ridiculous.
- shawn 8y ago"Inaudible" doesn't necessarily mean a range outside of human hearing. For example you could embed a 6kHz tone in a way that's inaudible to humans due to the other frequencies in the waveform.
- sideshowb 8y agoAnd promptly lose it to lossy compression
- jimrandomh 8y agoTV channels already contain inaudible-sound identifiers. Nielsen has people put listeners in their homes, and uses the identifiers to track which channels are getting played.
- teachrdan 8y agoThe real problem is that Alexa and Siri increase your attack surface area to include every speaker in your home--including any cheapo Bluetooth or internet-connected speakers that could get hacked to produce these human-inaudible sounds. (To be fair, I recall this specific point coming up in an HR thread about Alexa being able to open your door for Amazon deliveries, but I thought it was worth reiterating here.)
- hedora 8y agoAm I the only one that wants a nice cherry mechanical keyboard that transcribes typed commands to inaudible voice commands?
- ct0 8y agoyes
- Skunkleton 8y agoUsually when a comment starts with "am I the only one", the answer is NO. I think this is the first time I have ever seen a possible exception. Well done.
- freeone3000 8y agoWhy use the voice commands, then? Why not just type?
- jrochkind1 8y agoit's Art.
- tudelo 8y agoThe only thing I can think of is to either not be heard by others nearby or to mess with people who have these devices. 1 can be done by just typing to something that can actually natively store what you want and 2 is just for fun I guess.
- asdsa5325 8y agoyes, that sounds silly
- deleted 8y ago[deleted]
- sailfast 8y agoI can't recommend highly enough that everyone at least turn on the audible sound when their assistants are listening. At a minimum you should know the kinds of things that end up triggering the device. There's a really wide area of detection and it's interesting to see where that is. I would also recommending changing your default word at a minimum. Then again, I might also recommend ditching the device entirely but I happen to have one in my kitchen that I like OK sometimes.
- jackhack 8y agobut "Hey Spybot-Listening-To-Every-Conversation-In-This-Household-And-Building-A-Transcript-For-The-NSA" doesn't roll off the tongue the same way.
- TeMPOraL 8y ago"Hey, NSA snitch"? "Hello Big Brother?"
- crowbahr 8y agoCome now, on a tech and hacking website of all places we should all be better informed than this. There is a voice activation only chip tied directly into the lights that only listens for commands. Once the activation is heard it sends it off to the servers to get processed etc. With everyone scrutinizing the web traffic around them trying to prove NSA/Google spying with them we'd definitely have found something by now. So it's probably just yours that is sending off to the NSA. I'd send it back to them for repairs, see if they don't give you a free home mini in compensation!
- TheGuyWhoCodes 8y agoCan't this just be fixed so that Alexa, Siri, etc. will only accept your voice pattern?
- tinus_hn 8y agoWhile that’s really difficult it would also mean you’d have to train the assistant before you’d be able to use it, which is a big hurdle most customers probably don’t want.
- ComputerGuru 8y agoiOS now has a “text to siri” feature that can disable spoken interfacing but retain the “smart” capabilities of the digital assistant. Not that digital assistants are worth the risk they bring. Until now I can’t get Siri to do anything useful that isn’t very artificially and carefully phrased.
- deleted 8y ago[deleted]
- mkirklions 8y agoI got an iphone from work and I was shocked at how little Siri could do. Given all the hype from my friends and the commercials, I expected something outstanding. Nope, significantly worse than google's assistant. That was the start of my complete disappointment in apple as I continued to use an iphone and wonder- Why is anyone buying this?
- kfrzcode 8y agoThe same reason people buy Coach bags etc... status and brand.
- mulletbum 8y agoEveryone who likes iPhones owns one, so that removes status. Personal anecdote, I dislike Apple computers, but I like their phones. So I am at least one person who doesn't buy them for brand. This notion is outdated.
- mkirklions 8y agoCan you explain what is good about their phones? Things that drive me crazy- >No widgets >constant reminders to sign-in >constant reminders to update >no double tap/settings seem harder to find >Finger print scanner sucks so bad. >Little annoyances like the animations to change screen take 0.5 seconds too long. I'm not sure what I'm supposed to be enjoying on my iphone.
- lerie82 8y agoThis is old news.
- frenchie4111 8y ago> In the wrong hands, the technology could be used to unlock doors, wire money or buy stuff online — simply with music playing over the radio. Why hide it in radio content? Couldn't they just play it out loud when I am not home?
- callumprentice 8y agoDolphinAttack: Inaudible Voice commands: https://youtu.be/21HjF4A3WE4 https://youtu.be/21HjF4A3WE4
- Froyoh 8y agoCan't they just limit the activation frequency? Seems easy enough
- tasty_freeze 8y agoI don't know anything about it, but based on the name, I'll venture a guess. The audio system has an A/D converter which samples audio at a specific rate -- say 48 KHz. Aliasing occurs when the input to the D/A convert is above 1/2 the sample rate. A 24001 Hz signal is indistinguishable from a 23999 Hz signal. A 25000 Hz signal is indistinguishible from a 23000 Hz signal, etc. To eliminate these types of problems, there will be an analog lowpass filter before the sampling circuit. There is a gradual rolloff of signal sensitivity. Aliasing still occurs, but the energy of the aliased signals is significantly reduced. My guess is you take a voice command, even if it constrained to be say 200 Hz to 2KHz, then invert the spectrum and shift it to the 46-48 KHz range. When this high frequency is played back, due to aliasing, the software after the A/D converter sees it as a 0-2KHz signal, though greatly attenuated. To overcome that, the source audio can be tremendously loud. Humans can't hear it, so it remains stealthy.
- cozzyd 8y agoThat's clever but that's so many dB down with any sane anti-aliasing filter that it would require quite the sound source. Based on flipping through the pages of the paper (https://arxiv.org/pdf/1708.09537.pdf https://arxiv.org/pdf/1708.09537.pdf), it looks like they're taking advantage of the non-linearity in the response at high frequencies to effectively demodulate a lower-frequency signal that was mixed up to ~22 KHz. Which, if that's what they're doing, is totally awesome!
- deleted 8y ago[deleted]
- jacksmith21006 8y agoReplaced our Echos with Google homes. Curious if they are also vulnerabile?
- saagarjha 8y ago> Amazon said that it doesn’t disclose specific security measures, but it has taken steps to ensure its Echo smart speaker is secure. So, security by obscurity?
- notsofastbuddy 8y ago> So, security by obscurity? Obscurity is a perfectly valid layer in a security system. It's just not sufficient as the primary security mechanism.
- danShumway 8y ago> This month, some of those Berkeley researchers published a research paper that went further Pet peeve, I really wish that this was a link. Was I just blind? Is the actual paper linked anywhere in the article?
- srtjstjsj 8y agoIt's obliquely linked at "More recently, Mr. Carlini and his colleagues at Berkeley have [LINK: incorporated commands] into audio recognized by Mozilla’s DeepSpeech voice-to-text translation software, an open-source platform." https://nicholas.carlini.com/code/audio_adversarial_examples/ https://nicholas.carlini.com/code/audio_adversarial_examples... It's probably this paper: https://nicholas.carlini.com/papers/2018_dls_audioadvex.pdf https://nicholas.carlini.com/papers/2018_dls_audioadvex.pdf discussed in January when it went up on Arxiv: https://news.ycombinator.com/item?id=16220376 https://news.ycombinator.com/item?id=16220376
- danShumway 8y agoThanks, I guess I was just blind :)
- mdekkers 8y agoIt is beyond me why people would want to put a live mic in their home. Every dystopian story, real or fiction, features some element of constant observation, and here we go, happily placing these devices in our homes. Insane