5 ms·

> In principle a home assistant should maintain a ring-buffer that listens for its "wake word" and then immediately discards any audio that doesn't match the tr

> In principle a home assistant should maintain a ring-buffer that listens for its "wake word" and then immediately discards any audio that doesn't match the trigger. Should this be easy to verify with network monitoring? audio data is quite heavy. I'm assuming if audio information gets uploaded, it will be quite obvious on the data. We haven't heard about it, and I'm for that reason I don't think it actually does not do it (i.e. the ring buffer hypothesis is likely correct).

ipaddr 4y ago
What if it gets turned on selectively?
fladrif 4y ago
It doesn't need to upload audio data though, transcribing it to text would accomplish 90% of their use cases.
- drstewart 4y ago
  How do you think it gets transcribed to text?
- gr4yb34rd 4y ago
  I doubt the guts of something like an Echo would be capable of containing any kind of speech to text model. It'll probably be a lot more of a concern in a few years when speech models like that are getting baked into cheap chips for consumer devices, though.
  - dehrmann 4y ago
    Early versions of Dragon Naturally Speaking ran on Pentiums.
    - chickenchicken 4y ago
      I remember seeing this on a trade fair as a child with my mother in 1993
    - PointyFluff 4y ago
      [dead]
    - phire 4y ago
      Dragon Naturally Speaking required extensive training on the voice of the user, and the result wasn't as accurate as modern machine-learning speech-to-text models. You had to watch it and actively correct it's results. IMO, the accuracy of modern speech-to-text models is still nowhere near accurate enough. Maybe they should bring back the per-user training
  - woodruffw 4y ago
    I'm not as doubtful: iPhones have had on-device speech recognition since at least 2019, with iOS 13[1]. Amazon might have legal or policy reasons for not doing offline speech recognition, but the technology has been there for a while. [1]: https://developer.apple.com/documentation/speech/sfspeechrecognitionrequest/3152603-requiresondevicerecognition?language=objc https://developer.apple.com/documentation/speech/sfspeechrec...
    - sokoloff 4y ago
      Has the tech been available in a $29 device that also includes a speaker and a microphone array?
      - woodruffw 4y ago
        $29 is the sale price, not the BOM price. The entire point of "smart" home devices is to reduce consumer friction around revenue-driving actions; it would be surprising if Amazon were to prioritize making a profit on a $29 device rather than the sales and subscriptions it facilitates.
        jrockway 4y ago
        I don't know if it's every smart home vendor's goal to drive revenue. Apple sells their smart speakers for $100 or $300, which doesn't seem like a "loss leader" price, and it doesn't ask you to buy anything else. Their marketing page mentions what it can integrate with (smart locks, smart lights, etc.) but those things are notably not subscriptions. If you buy a smart light switch and a smart speaker, then you can just say "Hey Siri turn off the lights in the kitchen" and nobody ever gets any more money, and you didn't have to go downstairs and turn off the lights in the kitchen. This seems like a very justified smart home. I am skeptical of all things proprietary technology, but Apple's stuff bothers me the least.
      - gigel82 4y ago
        You can run whisper.cpp (a very solid TTS) on a Raspberry Pi (https://github.com/ggerganov/whisper.cpp/discussions/166 https://github.com/ggerganov/whisper.cpp/discussions/166), so I'm pretty sure with a modest CPU upgrade an Echo would be more than capable of running it locally.
  - vineyardmike 4y ago
    Not gen 1 devices but in 2023 some of the echos do partial on-device processing of commands (same with HomePods and same with Google home).
kkielhofner 4y ago
I was curious about this at one point in time. I whipped up a full-blown Rube Goldberg contraption that involved a lot of pcap filters on my router and kludgy hacks to Home Assistant for graphing, etc. I even integrated it to a separate LED on an esp32 sitting next to the Echo where the esp32 LED would activate when traffic to/from the Echo was detected. It's a tiny and anecdotal sample but even when looking at historical graphs I never once "caught" the Echo device transferring any even remotely suspicious amount of data unless it was wake-word activated.
- whatshisface 4y ago
  Patch notes: Transmission of customer's private conversations is now delayed until the wake word is spoken. See Bug 39482: "Suspicion aroused"
zamnos 4y ago
It doesn't seem like it would be impossibly hard to program "cocaine" as a quiet (ie, no visual or audio activity when activated with that particular word), save the audio on either side of that wake weird word, then wait for a "normal" activation that sends audio to the cloud, and then uploads the "cocaine" audio at the same time. Not remotely claiming any of the assistants do this, but if I were trying to program an evil voice assistant, it's what I would do to avoid detection. If the devices have the hardware for it, even imperfectly, run the transcription locally and it's even easier to stenographically hide just the text in the later upload. It doesn't need to perfect transcription to English either. Recording the phonemes for the reciever to reconstruct the language from would be enough. It's more important to get the names in the conversation anyway. Bonus points for also recording+transcribing phone numbers/DTMF tones it hears and then uploading all of them to the NSA.

ipaddr 4y ago

What if it gets turned on selectively?

fladrif 4y ago

It doesn't need to upload audio data though, transcribing it to text would accomplish 90% of their use cases.

drstewart 4y ago
How do you think it gets transcribed to text?
gr4yb34rd 4y ago
I doubt the guts of something like an Echo would be capable of containing any kind of speech to text model. It'll probably be a lot more of a concern in a few years when speech models like that are getting baked into cheap chips for consumer devices, though.
- dehrmann 4y ago
  Early versions of Dragon Naturally Speaking ran on Pentiums.
  - chickenchicken 4y ago
    I remember seeing this on a trade fair as a child with my mother in 1993
  - PointyFluff 4y ago
    [dead]
  - phire 4y ago
    Dragon Naturally Speaking required extensive training on the voice of the user, and the result wasn't as accurate as modern machine-learning speech-to-text models. You had to watch it and actively correct it's results. IMO, the accuracy of modern speech-to-text models is still nowhere near accurate enough. Maybe they should bring back the per-user training
- woodruffw 4y ago
  I'm not as doubtful: iPhones have had on-device speech recognition since at least 2019, with iOS 13[1]. Amazon might have legal or policy reasons for not doing offline speech recognition, but the technology has been there for a while. [1]: https://developer.apple.com/documentation/speech/sfspeechrecognitionrequest/3152603-requiresondevicerecognition?language=objc https://developer.apple.com/documentation/speech/sfspeechrec...
  - sokoloff 4y ago
    Has the tech been available in a $29 device that also includes a speaker and a microphone array?
    - woodruffw 4y ago
      $29 is the sale price, not the BOM price. The entire point of "smart" home devices is to reduce consumer friction around revenue-driving actions; it would be surprising if Amazon were to prioritize making a profit on a $29 device rather than the sales and subscriptions it facilitates.
      - jrockway 4y ago
        I don't know if it's every smart home vendor's goal to drive revenue. Apple sells their smart speakers for $100 or $300, which doesn't seem like a "loss leader" price, and it doesn't ask you to buy anything else. Their marketing page mentions what it can integrate with (smart locks, smart lights, etc.) but those things are notably not subscriptions. If you buy a smart light switch and a smart speaker, then you can just say "Hey Siri turn off the lights in the kitchen" and nobody ever gets any more money, and you didn't have to go downstairs and turn off the lights in the kitchen. This seems like a very justified smart home. I am skeptical of all things proprietary technology, but Apple's stuff bothers me the least.
    - gigel82 4y ago
      You can run whisper.cpp (a very solid TTS) on a Raspberry Pi (https://github.com/ggerganov/whisper.cpp/discussions/166 https://github.com/ggerganov/whisper.cpp/discussions/166), so I'm pretty sure with a modest CPU upgrade an Echo would be more than capable of running it locally.
- vineyardmike 4y ago
  Not gen 1 devices but in 2023 some of the echos do partial on-device processing of commands (same with HomePods and same with Google home).

kkielhofner 4y ago

I was curious about this at one point in time. I whipped up a full-blown Rube Goldberg contraption that involved a lot of pcap filters on my router and kludgy hacks to Home Assistant for graphing, etc. I even integrated it to a separate LED on an esp32 sitting next to the Echo where the esp32 LED would activate when traffic to/from the Echo was detected. It's a tiny and anecdotal sample but even when looking at historical graphs I never once "caught" the Echo device transferring any even remotely suspicious amount of data unless it was wake-word activated.

whatshisface 4y ago
Patch notes: Transmission of customer's private conversations is now delayed until the wake word is spoken. See Bug 39482: "Suspicion aroused"

zamnos 4y ago

It doesn't seem like it would be impossibly hard to program "cocaine" as a quiet (ie, no visual or audio activity when activated with that particular word), save the audio on either side of that wake weird word, then wait for a "normal" activation that sends audio to the cloud, and then uploads the "cocaine" audio at the same time. Not remotely claiming any of the assistants do this, but if I were trying to program an evil voice assistant, it's what I would do to avoid detection. If the devices have the hardware for it, even imperfectly, run the transcription locally and it's even easier to stenographically hide just the text in the later upload. It doesn't need to perfect transcription to English either. Recording the phonemes for the reciever to reconstruct the language from would be enough. It's more important to get the names in the conversation anyway. Bonus points for also recording+transcribing phone numbers/DTMF tones it hears and then uploading all of them to the NSA.