3 ms·

I wouldn't be so sure you'd know. Modern phones have coprocessors that are dedicated to transcribing speech into text in a very power efficient manner (hell, my

I wouldn't be so sure you'd know. Modern phones have coprocessors that are dedicated to transcribing speech into text in a very power efficient manner (hell, my Pixel phone has a feature which allows it to identify songs playing at all times ins a very power efficient manner, even when in "airplane mode", even when the screen is off). Days worth of text could be compressed and sent off in literally dozens of kilobytes. And that's not even accounting for side-channels (a hypothetical targeted attack could conceivably look for "target phrases" and send a single bit of information using a side channel to some server to alert that the phrase has been said).

LinuxBender 7y ago
I was not aware they have speech transcription co-processors now. That would certainly be a game changer. Back when I used to help the FBI monitor cell phones, they were very slow processors and the user could tell when it was sending anything due to the heat. Best we could do was play with MWI on/off to get cell locations and use the mainframe (cell switch) to do audio monitoring, but that was very limited at the time.
- Klathmon 7y ago
  I know it's ironic given the topic we're in, but the technology is actually really impressive! The coprocessors are really just ASICs designed to run neural networks efficiently. The real cool shit is in the compression and simplification of the networks that allow constant recognition from a small power-sipping chip. They're also doing a bunch of work with federated machine learning processes. So all of the training data can be kept on the device itself, but they can (i'm grossly oversimplifying this) basically "merge" a full neural net from somewhere else with data learned on-device. It's that last part that I'm extremely interested in, as it means we can have all the benefits of machine learning systems, but also keep all of our privacy in that all the processing happens locally, our training data doesn't need to leave the device at all, and the algorithms can still adapt and improve for each person individually. It apparently also has side effects that the neural network on your device can adapt to your usage individually much faster and more accurately than a "master" neural network ever could.
jerf 7y ago
Last I knew, the coprocessors that are running all the time are very, very small and incapable, and only capable of detecting the wake word, and that not even terribly reliably; it sent the wake word off to the main processor if it thought it heard it for further checks. This would be, say, two or three years ago. Are there now co-processors running all the time that could do full speech recognition (not just keyword recognition, which is way easier, but full speech recognition) all the time without such significant battery drain that it would be noticed? I ask not because I consider this some sort of technical impossibility, but because if that is the case, I'd like to know so I can update my understanding of the situation.
- Klathmon 7y ago
  [1] is google's paper on their implementation of a setup similar to this for identifying songs entirely offline without using much power at all. You are correct that the actual processors that are "always" listening are fairly limited (They describe it as a DSP, so that should give you an idea of its capabilities), but when layered the way they are in Pixel phones, it makes it possible for the DSP to detect "something" interesting is happening, then wake up and pass the information to an "AI processor" (they call it the "Pixel visual core" but i find that a misnomer because it's a pretty capable TPU on it's own) which would then do the transcribing, and go back to sleep. I'm making some assumptions that I haven't researched myself, but I'm assuming there are patterns, frequencies, and other simply detected "signals" present in most speech that make it easy to detect "someone is speaking", and from there can wake up a more powerful (but still surprisingly efficient) processor to do more work. And Google also has some simplified and compressed neural networks dedicated to transcribing speech which are designed to run on lower powered devices[2]. I don't know if they are actually using the PVC in the Pixel phones to do audio transcription or if they're just using the general CPU for that, but I have a feeling that a TPU would be able to scream through buffered audio very quickly and would go back to sleep very quickly leading to minimal power draw. In fact I'd be willing to bet good money that an "always transcribing" system could be done today with current generation phones in a way that wouldn't impact battery life enough to be obviously noticeable or cause the user to start looking for what was draining it in most cases. And considering how often I'm personally actively speaking in a day, I bet it would have virtually no impact on my personal device if implemented this way. (and man did it feel wrong to type that last paragraph out!) [1] https://ai.google/research/pubs/pub46522 https://ai.google/research/pubs/pub46522 [2] https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp...

LinuxBender 7y ago

I was not aware they have speech transcription co-processors now. That would certainly be a game changer. Back when I used to help the FBI monitor cell phones, they were very slow processors and the user could tell when it was sending anything due to the heat. Best we could do was play with MWI on/off to get cell locations and use the mainframe (cell switch) to do audio monitoring, but that was very limited at the time.

Klathmon 7y ago
I know it's ironic given the topic we're in, but the technology is actually really impressive! The coprocessors are really just ASICs designed to run neural networks efficiently. The real cool shit is in the compression and simplification of the networks that allow constant recognition from a small power-sipping chip. They're also doing a bunch of work with federated machine learning processes. So all of the training data can be kept on the device itself, but they can (i'm grossly oversimplifying this) basically "merge" a full neural net from somewhere else with data learned on-device. It's that last part that I'm extremely interested in, as it means we can have all the benefits of machine learning systems, but also keep all of our privacy in that all the processing happens locally, our training data doesn't need to leave the device at all, and the algorithms can still adapt and improve for each person individually. It apparently also has side effects that the neural network on your device can adapt to your usage individually much faster and more accurately than a "master" neural network ever could.

jerf 7y ago

Last I knew, the coprocessors that are running all the time are very, very small and incapable, and only capable of detecting the wake word, and that not even terribly reliably; it sent the wake word off to the main processor if it thought it heard it for further checks. This would be, say, two or three years ago. Are there now co-processors running all the time that could do full speech recognition (not just keyword recognition, which is way easier, but full speech recognition) all the time without such significant battery drain that it would be noticed? I ask not because I consider this some sort of technical impossibility, but because if that is the case, I'd like to know so I can update my understanding of the situation.

Klathmon 7y ago
[1] is google's paper on their implementation of a setup similar to this for identifying songs entirely offline without using much power at all. You are correct that the actual processors that are "always" listening are fairly limited (They describe it as a DSP, so that should give you an idea of its capabilities), but when layered the way they are in Pixel phones, it makes it possible for the DSP to detect "something" interesting is happening, then wake up and pass the information to an "AI processor" (they call it the "Pixel visual core" but i find that a misnomer because it's a pretty capable TPU on it's own) which would then do the transcribing, and go back to sleep. I'm making some assumptions that I haven't researched myself, but I'm assuming there are patterns, frequencies, and other simply detected "signals" present in most speech that make it easy to detect "someone is speaking", and from there can wake up a more powerful (but still surprisingly efficient) processor to do more work. And Google also has some simplified and compressed neural networks dedicated to transcribing speech which are designed to run on lower powered devices[2]. I don't know if they are actually using the PVC in the Pixel phones to do audio transcription or if they're just using the general CPU for that, but I have a feeling that a TPU would be able to scream through buffered audio very quickly and would go back to sleep very quickly leading to minimal power draw. In fact I'd be willing to bet good money that an "always transcribing" system could be done today with current generation phones in a way that wouldn't impact battery life enough to be obviously noticeable or cause the user to start looking for what was draining it in most cases. And considering how often I'm personally actively speaking in a day, I bet it would have virtually no impact on my personal device if implemented this way. (and man did it feel wrong to type that last paragraph out!) [1] https://ai.google/research/pubs/pub46522 https://ai.google/research/pubs/pub46522 [2] https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp...