13 ms·

Hey, Alexa, What Can You Hear? And What Will You Do With It?

200 points by SREinSF 9y ago

z_open 9y ago
In past discussions, I was led to believe we were safe from these devices, in terms of pervasive data collection, because the main processor that was always running was primitive and was only powerful enough to pick up a key phrase. This phrase would power the main processor which would then listen to your query, respond, and then deactivate. But now, as the article mentions, I realize that this initial processor could be made to listen for many key words like "love", "hate", or other words that would pick up on sensitive personal information. I really don't think these devices should exist.
- pdkl95 9y ago
  > because the main processor that was always running was primitive and was only powerful enough to pick up a key phrase That's only true if you're trying to do complex analysis on the device. Cell phones in the early 1990s had even less computing power, but it was enough to encode speech down to 13.2 kbit/s[1] (or 5.6 kbit/s[2]). A simple noise gate[3] would reduce recording duty cycle down to maybe 1% while costing a trivial amount of CPU load. Modern hardware - even tiny embedded devices - can probably do a lot more than simple gat4ed+compressed audio. [1] https://en.wikipedia.org/wiki/Full_Rate https://en.wikipedia.org/wiki/Full_Rate [2] https://en.wikipedia.org/wiki/Half_Rate https://en.wikipedia.org/wiki/Half_Rate [3] https://en.wikipedia.org/wiki/Noise_gate https://en.wikipedia.org/wiki/Noise_gate
  - qvrjuec 9y ago
    So let's assume this is technically possible, and only examine the process under the lens of detectability. Assuming the codec is 8kbps, you'd see roughly 3MB upload for every hour of audio, and I would say it's not out of the question for the noise gate to be active from the television being on in the same room, or music playing. For an utterance, seeing a 3MB upstream would be super abnormal. It would be impossible to transmit all of this data for processing without somebody noticing.
    - jononor 9y ago
      Why would it record continiously? If I were developing such a device I'd use random sampling. Don't have use for all the audio in every home, but a little bit from everywhere can be useful. Can be used to improve the acoustic model of the room maybe, or improve targetting of ads[1]. Some snippets from the minutes before and activation could be especially interesting, to understand the context. 1. Or more nefarious things, of course.
      - soared 9y ago
        Or record audio, do speech to text, some light analysis, and upload the metadata. Or just upload a basket of words and their frequency, "Toilet Paper, 2, tv, 7, online, 4,"
        jononor 9y ago
        Detecting who speaks is also a (relatively) lightweight analysis. Combined with bag-of-words can build personality/interest profile. Speech based gender detection can also be done, probably also detecting kids from adults. Now have good data about demographics of the household.
    - pdkl95 9y ago
      While a noise gate is an obvious starting point, I suspect that much smarter filters are possible. A wide range of design choices are possible that trade CPU <-> complexity <-> accuracy. If a high error rate (including "no data") is acceptable, a filter might simply default to "off" when any unusual condition like "noise gate has been on continuously for 15 minutes" is detected. If I was designing this kind of spyware, I would put a hard limit that cuts off uploading any "extra" data after sending some low multiple of the "legitimate" data.
oulipo 9y ago
You can't trust the large companies who have a culture of profiting off your data to protect your privacy. This is why we need AI at the edge, and not in the clouds, and Privacy-by-design thinking in our architectures. This is the only way for people to know their data won't be compromised and misused, because it never leaves their devices. Disclaimer: I'm a co-founder of https://snips.ai https://snips.ai and we are building a 100% on-device and private-by-design platform to build Voice AI assistants We would be happy to know what you do with it! You can take a look at what some people have built with it already https://github.com/snipsco/awesome-snips https://github.com/snipsco/awesome-snips We are open-sourcing it over time, starting with the NLU: https://medium.com/snips-ai/snips-nlu-is-an-open-source-private-by-design-alternative-to-dialogflow-amazon-lex-and-other-nlu-a95dbe16f4a1 https://medium.com/snips-ai/snips-nlu-is-an-open-source-priv.... Snips is available in English, French, German, and soon Japanese and Korean with more European languages coming this year. You can start building your own private-by-design smart speaker on the platform in under 1h with this tutorial: https://medium.com/snips-ai/building-a-voice-controlled-home-sound-system-using-snips-and-sonos-2aaf16523ce9 https://medium.com/snips-ai/building-a-voice-controlled-home...
- jacquesm 9y ago
  > I'm a co-founder of https://snips.ai https://snips.ai and we are building a 100% on-device and private-by-design platform to build Voice AI assistants Excellent! How will you guarantee that your devices will not leak data?
  - oulipo 9y ago
    We are open-sourcing the code, and people can have their own network monitoring to see that there is no data leaking
    - jacquesm 9y ago
      Please check into 'reproducible builds' and ways to ensure the software is not modified between the point where you build it and the point where it gets deployed. There will most likely at some point be an attempt to use your reputation to achieve the exact opposite of your stated goals so beware of that.
rb808 9y ago
With a court order can authorities turn on the microphone and just listen to everything? Seems easy to do but I haven't heard about it yet. I guess phones can do the same. Presumably the Russians and Chinese do this already. :)
- pdkl95 9y ago
  A warrant is required only as long as "voice assistant" technology "is not in general public use"[1]. Kyllo v United States created a bright-line[3] test that removes the warrant requirement to see "details of a private home that would previously have been unknowable without physical intrusion"[2] when use of the technology is normalized. Note that this removes the warrant requirement in general, even if you personally don't own an Alexa/etc. The test is if the public expects audio in the home might be recorded and sent to a 3rd party. If the answer is "yes", then the police can use their own hardware to do the recording. [1] http://caselaw.findlaw.com/us-supreme-court/533/27.html http://caselaw.findlaw.com/us-supreme-court/533/27.html [2] Ibid. [3] https://en.wikipedia.org/wiki/Bright-line_rule https://en.wikipedia.org/wiki/Bright-line_rule
  - shawn-butler 9y ago
    Amazon resisted a warrant from a prosecutor seeking all Alexa recordings from a house that was the site of murder. Trial in Arkansas I believe. I don't think it ever went anywhere precedent setting however because the defendant eventually gave permission to Amazon to turn it over. Still pretty muddy waters. I would trust Amazon a little more than I would trust the smart tv makers, but that's just my bias.
    - pdkl95 9y ago
      > I would trust Amazon Note that re: Kyllo v United States, trust in any specific manufacturer is irrelevant. The SCOTUS ruled that use of a technology - not a product - requires a warrant when it is "is not in general public use". The Kyllo case involved two federal agents that used their own thermal imaging camera to search a house for the presence of people and grow lights.
pdkl95 9y ago
> including an “algorithmic transparency requirement” that would help people understand how their data was being used and what automated decisions were then being made about them. This needs to be required for any type of algorithmic decision making. Without algorithmic transparency ulterior motives, intentional or unintentional biases, and unnoticed mistakes are hidden from public review. A common response is that we don't know how some types of machine learning make their decisions. I agree this is occasionally true. Find a way generate an explanation, or use a different algorithm; transparency is a critical requirement.
deleted 9y ago
[deleted]
amelius 9y ago
Why don't we have open-source voice-assistants yet? I mean, if we can have an open-source OS (e.g. Linux), then surely we can have open-source speech recognition, right?
- robotrout 9y ago
  Voice assistants are difficult, both electro-acoustically to get a good clean voice signal from the ambient room noise and voice-assistant generated audio (this is called barge-in), as well as the software to actually parse that speech. A great open-source voice-assistant is ambitious. But why isn't there a great open-source version of any $iot_appliance? IOT, in general, is crawling with companies that are looking for a monthly cash flow, and are therefore creating an ecosystem of non-federated, walled garden type devices that report back to some central server to keep the user dependent upon and paying monthly cash to some company. All of these IOT projects are generally simpler than voice-assistants, and I look forward to the day when we have IOT projects that talk to each other or a local server, without linking us to some company forever.
  - amelius 9y ago
    > A great open-source voice-assistant is ambitious. Yes, I get that. But so was writing the Linux kernel.
    - ggg9990 9y ago
      Linux kernel is the outlier, not the norm. You can’t take a regular runner and ask him why he isn’t running as fast as Usain Bolt.
      - amelius 9y ago
        Yes, but we only need one Usain Bolt to push the boundary of running.
        ConceptJunkie 9y ago
musage 9y ago
It's time we stop, hey, what's that sound Everybody look what's going down
fortythirteen 9y ago
The things people are willing to give up for the most minimal of conveniences are only going to get worse in the next decade. I wouldn't be willing to put an open wiretap in my home, even if it did something amazing, like extend my lifespan. The quality of my life is not significantly improved with these devices, and all of their practical uses can be duplicated with the minimal effort of tapping a screen a couple of times.
- SlowRobotAhead 9y ago
  I'm seriously shocked by the number of people that are putting always on "assistants" in their home - but more so on the number of people that should know better, like HN people.
  - amelius 9y ago
    Even if you know better and act like it, you're still going to be monitored while visiting friends.
  - _rpd 9y ago
    One of my friends has one of these because she likes the way it helps her build a grocery shopping list over the week. She once thought of herself as a privacy activist, and is fully conscious of the hypocrisy. It really made me think twice about how small the convenience has to be for someone to compromise their privacy.
  - criddell 9y ago
    What's shocking about it? For just the Echo, Amazon claims to have sold 20 million devices so they are probably in at least 10 million homes. How many of the people that buy an Echo have been harmed so far? I have an Echo and use it daily for music, weather, timers, home automation (ie controlling lights and fans), etc... I also drive a car almost every day and that has a relatively high probability of killing me. Compared to that, having an Echo in my house seems somewhat harmless.
    - StanislavPetrov 9y ago
peg_leg 9y ago
I had a dream that Alexa recognized my voice when I was visiting a friend's house. Won't be long for that one.
- wackspurt 9y ago
  That's funny. I wonder how human intrigues (and the literature/narratives around it) will be affected by ubiquitous computing and networking.
- Johnny555 9y ago
  Your phone already knows you visited your friend's house, why are you worried about Alexa knowing too?
sqdbps 9y ago
Companies use patents defensively and the Times is fully aware of that so this comes across as cynical clickbait. What I find interesting is that they keep quoting "consumer watchdog" an anti google organization turned anti technology (they're against self driving cars and robots now). The scary thing is that mainstream respected news outlets casually traffic in technophobia as evident form articles about automation and AI for example and how every piece mentioning a tech company is permeated with FUD about their motivations or their "power".
- brynjolf 9y ago
  That is a lot of buzzwords.
EGreg 9y ago
One application details how audio monitoring could help detect that a child is engaging in “mischief” at home by first using speech patterns and pitch to identify a child’s presence, one filing said. A device could then try to sense movement while listening for whispers or silence, and even program a smart speaker to “provide a verbal warning.” This worries me. First the children are watched 24/7, them the adults.
r3nrut 9y ago
If you're not planning to release your own hardware, could you provide a list of devices that the software is tested with? I looked on the site a bit but the crazy art thing triggered my ADHD and OCD as well. I imagine the idea is that you can compile it for whatever device on any platform with some change but makes it harder for community engagement without a go to for sure platform to test some builds and POCs.
akeck 9y ago
Alexa could enable subtle insider trading by both Amazon employees and Amazon AIs.

200 points by SREinSF 9y ago

In past discussions, I was led to believe we were safe from these devices, in terms of pervasive data collection, because the main processor that was always running was primitive and was only powerful enough to pick up a key phrase. This phrase would power the main processor which would then listen to your query, respond, and then deactivate. But now, as the article mentions, I realize that this initial processor could be made to listen for many key words like "love", "hate", or other words that would pick up on sensitive personal information. I really don't think these devices should exist.

pdkl95 9y ago
> because the main processor that was always running was primitive and was only powerful enough to pick up a key phrase That's only true if you're trying to do complex analysis on the device. Cell phones in the early 1990s had even less computing power, but it was enough to encode speech down to 13.2 kbit/s[1] (or 5.6 kbit/s[2]). A simple noise gate[3] would reduce recording duty cycle down to maybe 1% while costing a trivial amount of CPU load. Modern hardware - even tiny embedded devices - can probably do a lot more than simple gat4ed+compressed audio. [1] https://en.wikipedia.org/wiki/Full_Rate https://en.wikipedia.org/wiki/Full_Rate [2] https://en.wikipedia.org/wiki/Half_Rate https://en.wikipedia.org/wiki/Half_Rate [3] https://en.wikipedia.org/wiki/Noise_gate https://en.wikipedia.org/wiki/Noise_gate
- qvrjuec 9y ago
  So let's assume this is technically possible, and only examine the process under the lens of detectability. Assuming the codec is 8kbps, you'd see roughly 3MB upload for every hour of audio, and I would say it's not out of the question for the noise gate to be active from the television being on in the same room, or music playing. For an utterance, seeing a 3MB upstream would be super abnormal. It would be impossible to transmit all of this data for processing without somebody noticing.
  - jononor 9y ago
    Why would it record continiously? If I were developing such a device I'd use random sampling. Don't have use for all the audio in every home, but a little bit from everywhere can be useful. Can be used to improve the acoustic model of the room maybe, or improve targetting of ads[1]. Some snippets from the minutes before and activation could be especially interesting, to understand the context. 1. Or more nefarious things, of course.
    - soared 9y ago
      Or record audio, do speech to text, some light analysis, and upload the metadata. Or just upload a basket of words and their frequency, "Toilet Paper, 2, tv, 7, online, 4,"
      - jononor 9y ago
        Detecting who speaks is also a (relatively) lightweight analysis. Combined with bag-of-words can build personality/interest profile. Speech based gender detection can also be done, probably also detecting kids from adults. Now have good data about demographics of the household.
  - pdkl95 9y ago
    While a noise gate is an obvious starting point, I suspect that much smarter filters are possible. A wide range of design choices are possible that trade CPU <-> complexity <-> accuracy. If a high error rate (including "no data") is acceptable, a filter might simply default to "off" when any unusual condition like "noise gate has been on continuously for 15 minutes" is detected. If I was designing this kind of spyware, I would put a hard limit that cuts off uploading any "extra" data after sending some low multiple of the "legitimate" data.

oulipo 9y ago

You can't trust the large companies who have a culture of profiting off your data to protect your privacy. This is why we need AI at the edge, and not in the clouds, and Privacy-by-design thinking in our architectures. This is the only way for people to know their data won't be compromised and misused, because it never leaves their devices. Disclaimer: I'm a co-founder of https://snips.ai https://snips.ai and we are building a 100% on-device and private-by-design platform to build Voice AI assistants We would be happy to know what you do with it! You can take a look at what some people have built with it already https://github.com/snipsco/awesome-snips https://github.com/snipsco/awesome-snips We are open-sourcing it over time, starting with the NLU: https://medium.com/snips-ai/snips-nlu-is-an-open-source-private-by-design-alternative-to-dialogflow-amazon-lex-and-other-nlu-a95dbe16f4a1 https://medium.com/snips-ai/snips-nlu-is-an-open-source-priv.... Snips is available in English, French, German, and soon Japanese and Korean with more European languages coming this year. You can start building your own private-by-design smart speaker on the platform in under 1h with this tutorial: https://medium.com/snips-ai/building-a-voice-controlled-home-sound-system-using-snips-and-sonos-2aaf16523ce9 https://medium.com/snips-ai/building-a-voice-controlled-home...

jacquesm 9y ago
> I'm a co-founder of https://snips.ai https://snips.ai and we are building a 100% on-device and private-by-design platform to build Voice AI assistants Excellent! How will you guarantee that your devices will not leak data?
- oulipo 9y ago
  We are open-sourcing the code, and people can have their own network monitoring to see that there is no data leaking
  - jacquesm 9y ago
    Please check into 'reproducible builds' and ways to ensure the software is not modified between the point where you build it and the point where it gets deployed. There will most likely at some point be an attempt to use your reputation to achieve the exact opposite of your stated goals so beware of that.

rb808 9y ago

With a court order can authorities turn on the microphone and just listen to everything? Seems easy to do but I haven't heard about it yet. I guess phones can do the same. Presumably the Russians and Chinese do this already. :)

pdkl95 9y ago
A warrant is required only as long as "voice assistant" technology "is not in general public use"[1]. Kyllo v United States created a bright-line[3] test that removes the warrant requirement to see "details of a private home that would previously have been unknowable without physical intrusion"[2] when use of the technology is normalized. Note that this removes the warrant requirement in general, even if you personally don't own an Alexa/etc. The test is if the public expects audio in the home might be recorded and sent to a 3rd party. If the answer is "yes", then the police can use their own hardware to do the recording. [1] http://caselaw.findlaw.com/us-supreme-court/533/27.html http://caselaw.findlaw.com/us-supreme-court/533/27.html [2] Ibid. [3] https://en.wikipedia.org/wiki/Bright-line_rule https://en.wikipedia.org/wiki/Bright-line_rule
- shawn-butler 9y ago
  Amazon resisted a warrant from a prosecutor seeking all Alexa recordings from a house that was the site of murder. Trial in Arkansas I believe. I don't think it ever went anywhere precedent setting however because the defendant eventually gave permission to Amazon to turn it over. Still pretty muddy waters. I would trust Amazon a little more than I would trust the smart tv makers, but that's just my bias.
  - pdkl95 9y ago
    > I would trust Amazon Note that re: Kyllo v United States, trust in any specific manufacturer is irrelevant. The SCOTUS ruled that use of a technology - not a product - requires a warrant when it is "is not in general public use". The Kyllo case involved two federal agents that used their own thermal imaging camera to search a house for the presence of people and grow lights.

pdkl95 9y ago

> including an “algorithmic transparency requirement” that would help people understand how their data was being used and what automated decisions were then being made about them. This needs to be required for any type of algorithmic decision making. Without algorithmic transparency ulterior motives, intentional or unintentional biases, and unnoticed mistakes are hidden from public review. A common response is that we don't know how some types of machine learning make their decisions. I agree this is occasionally true. Find a way generate an explanation, or use a different algorithm; transparency is a critical requirement.

deleted 9y ago

[deleted]

amelius 9y ago

Why don't we have open-source voice-assistants yet? I mean, if we can have an open-source OS (e.g. Linux), then surely we can have open-source speech recognition, right?

robotrout 9y ago
Voice assistants are difficult, both electro-acoustically to get a good clean voice signal from the ambient room noise and voice-assistant generated audio (this is called barge-in), as well as the software to actually parse that speech. A great open-source voice-assistant is ambitious. But why isn't there a great open-source version of any $iot_appliance? IOT, in general, is crawling with companies that are looking for a monthly cash flow, and are therefore creating an ecosystem of non-federated, walled garden type devices that report back to some central server to keep the user dependent upon and paying monthly cash to some company. All of these IOT projects are generally simpler than voice-assistants, and I look forward to the day when we have IOT projects that talk to each other or a local server, without linking us to some company forever.
- amelius 9y ago
  > A great open-source voice-assistant is ambitious. Yes, I get that. But so was writing the Linux kernel.
  - ggg9990 9y ago
    Linux kernel is the outlier, not the norm. You can’t take a regular runner and ask him why he isn’t running as fast as Usain Bolt.
    - amelius 9y ago
      Yes, but we only need one Usain Bolt to push the boundary of running.
      - ConceptJunkie 9y ago

musage 9y ago

It's time we stop, hey, what's that sound Everybody look what's going down

fortythirteen 9y ago

The things people are willing to give up for the most minimal of conveniences are only going to get worse in the next decade. I wouldn't be willing to put an open wiretap in my home, even if it did something amazing, like extend my lifespan. The quality of my life is not significantly improved with these devices, and all of their practical uses can be duplicated with the minimal effort of tapping a screen a couple of times.

SlowRobotAhead 9y ago
I'm seriously shocked by the number of people that are putting always on "assistants" in their home - but more so on the number of people that should know better, like HN people.
- amelius 9y ago
  Even if you know better and act like it, you're still going to be monitored while visiting friends.
- _rpd 9y ago
  One of my friends has one of these because she likes the way it helps her build a grocery shopping list over the week. She once thought of herself as a privacy activist, and is fully conscious of the hypocrisy. It really made me think twice about how small the convenience has to be for someone to compromise their privacy.
- criddell 9y ago
  What's shocking about it? For just the Echo, Amazon claims to have sold 20 million devices so they are probably in at least 10 million homes. How many of the people that buy an Echo have been harmed so far? I have an Echo and use it daily for music, weather, timers, home automation (ie controlling lights and fans), etc... I also drive a car almost every day and that has a relatively high probability of killing me. Compared to that, having an Echo in my house seems somewhat harmless.
  - StanislavPetrov 9y ago

peg_leg 9y ago

I had a dream that Alexa recognized my voice when I was visiting a friend's house. Won't be long for that one.

wackspurt 9y ago
That's funny. I wonder how human intrigues (and the literature/narratives around it) will be affected by ubiquitous computing and networking.
Johnny555 9y ago
Your phone already knows you visited your friend's house, why are you worried about Alexa knowing too?

sqdbps 9y ago

Companies use patents defensively and the Times is fully aware of that so this comes across as cynical clickbait. What I find interesting is that they keep quoting "consumer watchdog" an anti google organization turned anti technology (they're against self driving cars and robots now). The scary thing is that mainstream respected news outlets casually traffic in technophobia as evident form articles about automation and AI for example and how every piece mentioning a tech company is permeated with FUD about their motivations or their "power".

brynjolf 9y ago
That is a lot of buzzwords.

EGreg 9y ago

One application details how audio monitoring could help detect that a child is engaging in “mischief” at home by first using speech patterns and pitch to identify a child’s presence, one filing said. A device could then try to sense movement while listening for whispers or silence, and even program a smart speaker to “provide a verbal warning.” This worries me. First the children are watched 24/7, them the adults.

r3nrut 9y ago

If you're not planning to release your own hardware, could you provide a list of devices that the software is tested with? I looked on the site a bit but the crazy art thing triggered my ADHD and OCD as well. I imagine the idea is that you can compile it for whatever device on any platform with some change but makes it harder for community engagement without a go to for sure platform to test some builds and POCs.

akeck 9y ago

Alexa could enable subtle insider trading by both Amazon employees and Amazon AIs.