7 ms·

Speaking without vocal cords, thanks to a new AI-assisted wearable device

121 points by geox 3y ago

khimaros 3y ago
i'm really excited to see progress being made in this space. subvocal speech recognition seems to be an underfunded area of research. my sense is that it has the potential to make hands free interaction with our devices in public spaces less obnoxious and, consequently, more socially acceptable. however, i notice that the article doesn't mention anything about dictionary size, which is a very important consideration for a tool of this kind.
- graphe 3y ago
  https://en.m.wikipedia.org/wiki/Basic_english https://en.m.wikipedia.org/wiki/Basic_english this communicates English efficiently with 850 words. I don't think it's basic English is any good but I can see them making simplified English the lingua franca to boost 'literacy rates' in the future.
- thorum 3y ago
  > The research team demonstrated the system’s accuracy by having the participants pronounce five sentences — both aloud and voicelessly — including “Hi, Rachel, how are you doing today?” and “I love you!” (…) Going forward, the research team plans to continue enlarging the vocabulary of the device through machine learning and to test it in people with speech disorders. It's a proof of concept at this stage but very cool.
- dontreact 3y ago
  While subvocal is cool and would allow for speech in more places, something that’s earlier on the tech tree and that I would like to see is just robust lipreading. I already am comfortable talking to my phone quietly using my AirPods while looking at my screen, but it seems like in loud public places the accuracy becomes unusable. I imagine it could be easily recovered by the additional signal of lipreading.
anonylizard 3y ago
This seems only useful to people who once had a voice, then lost their voice. Because only this way, would they have a unified mapping of voice cord movements to actual voices. Deaf and mutes can't really use this. It also basically mandates a patch to your throat, because no way of detecting vibrations otherwise. I wonder if there are visual based ways, like sign->text, expression->text, that would benefit from the larger developments in LLMs. Like an LLM that has access to your conversation history, so when you give your smartphone camera a hand sign and a smile, it can guess and output an entire intended speech.
zharknado 3y ago
Very cool! This is an insanely impressive sensor, but the proposed application is still in dream phase. > Going forward, the research team plans to continue enlarging the vocabulary of the device through machine learning and to test it in people with speech disorders. They haven’t tried giving it to a person with a voice disorder. So it just might not work in that application at all. That will likely depend on the degree to which laryngeal muscles are implicated in a given person’s disorder. That’s certainly a valid starting place for research purposes, but it’s very early days. And I imagine you’ll need some very interesting cabling attached to a somewhat beefy device to actually run live inference from this data, plus to drive the speech synthesis.
tbenst 3y ago
This is a super cool device. Note that the decoding is highly limited: they decode into one of five different sentences. This is easier than five words for example as there is more information to distinguish. Unfortunately the media is blowing this way of out proportion as the larynx alone does not contain sufficient information to decode silent speech. If you also sense the lips, tongue articulators, and jaw, then general English decoding becomes possible with high accuracy (eg see our recent work here: https://x.com/tbenst/status/1767952614157848859 https://x.com/tbenst/status/1767952614157848859). It’s not in the preprint but I’ve done experiments with only the larynx recorded and performance is pretty abysmal on even a 10 word vocabulary—-hence why they did a five sentence task.
- irviss 3y ago
  > If you also sense the lips, tongue articulators, and jaw, then general English decoding becomes possible with high accuracy A bit OT but I see this frequently and I'm curious. Why do you English speakers (or just a US phenomenon?) tend to use the word "English" instead of "language", "linguistic" or one of its related words to refer to a general concept?
  - atopal 3y ago
    There are about 6000 spoken languages around the world with an extreme variety in how they produce meaning. How could you make sweeping statements about all of them?
  - x1798DE 3y ago
    Not OP, but as a native English speaker and former scientist (though not in this area), I would interpret "x does y on English tasks" to mean "we tested this in English and don't know if the effect generalizes to other languages".
    - thaumasiotes 3y ago
      In this case we do know if the effect generalizes to other languages. It cannot fail to; the larynx, lips, tongue, and jaw are almost all there is. For example, vowels are conventionally defined by jaw position ("height"), tongue position ("frontness"), and lip configuration ("rounded" or not). You might miss some things like creaky voice or ejectives, you'll probably miss aspiration, but all that does is give you a worst-case scenario analogous to a native speaker trying to understand someone with a foreign accent. Extremely high accuracy will be possible.
ImHereToVote 3y ago
I'll take the whole lot.
feverzsj 3y ago
How it compares to electrolarynx, which give you robotic sound.
croemer 3y ago
"Speaking" is a hyperbole. It allows you to say exactly 5 phrases with 95% accuracy, after repeating each sentence 100 times. In other words, it's totally useless. The sentences are so different that they can be distinguished almost entirely by length. I'm very surprised anyone thinks this is useful. Excerpt: "A brief demonstration was made with five sentences that we had selected for training the algorithm (S1: “Hi Rachel, how you are doing today?”, S2: “Hope your experiments are going well!”, S3: “Merry Christmas!”, S4: “I love you!”, S5: “I don’t trust you.”). Each participant repeated each sentence 100 times for data collection." I never read a press release from a university, it's always exaggerated. Original study: https://www.nature.com/articles/s41467-024-45915-7 https://www.nature.com/articles/s41467-024-45915-7
- light_hue_1 3y ago
  Exactly. They took a neat device and wrapped it in a BS story that's wildly unscientific. Nature and Science don't mind if people outright lie about what their research means as long as it gets hits. The paper is pretty much just as bad. This is how mistrust for science slowly builds up when people publish obvious falsehoods.
joshspankit 3y ago
Is it just me, or does anyone else think it would be amazing to use upcoming voice assistants with something like this letting you “talk” silently?

121 points by geox 3y ago

i'm really excited to see progress being made in this space. subvocal speech recognition seems to be an underfunded area of research. my sense is that it has the potential to make hands free interaction with our devices in public spaces less obnoxious and, consequently, more socially acceptable. however, i notice that the article doesn't mention anything about dictionary size, which is a very important consideration for a tool of this kind.

graphe 3y ago
https://en.m.wikipedia.org/wiki/Basic_english https://en.m.wikipedia.org/wiki/Basic_english this communicates English efficiently with 850 words. I don't think it's basic English is any good but I can see them making simplified English the lingua franca to boost 'literacy rates' in the future.
thorum 3y ago
> The research team demonstrated the system’s accuracy by having the participants pronounce five sentences — both aloud and voicelessly — including “Hi, Rachel, how are you doing today?” and “I love you!” (…) Going forward, the research team plans to continue enlarging the vocabulary of the device through machine learning and to test it in people with speech disorders. It's a proof of concept at this stage but very cool.
dontreact 3y ago
While subvocal is cool and would allow for speech in more places, something that’s earlier on the tech tree and that I would like to see is just robust lipreading. I already am comfortable talking to my phone quietly using my AirPods while looking at my screen, but it seems like in loud public places the accuracy becomes unusable. I imagine it could be easily recovered by the additional signal of lipreading.

anonylizard 3y ago

This seems only useful to people who once had a voice, then lost their voice. Because only this way, would they have a unified mapping of voice cord movements to actual voices. Deaf and mutes can't really use this. It also basically mandates a patch to your throat, because no way of detecting vibrations otherwise. I wonder if there are visual based ways, like sign->text, expression->text, that would benefit from the larger developments in LLMs. Like an LLM that has access to your conversation history, so when you give your smartphone camera a hand sign and a smile, it can guess and output an entire intended speech.

zharknado 3y ago

Very cool! This is an insanely impressive sensor, but the proposed application is still in dream phase. > Going forward, the research team plans to continue enlarging the vocabulary of the device through machine learning and to test it in people with speech disorders. They haven’t tried giving it to a person with a voice disorder. So it just might not work in that application at all. That will likely depend on the degree to which laryngeal muscles are implicated in a given person’s disorder. That’s certainly a valid starting place for research purposes, but it’s very early days. And I imagine you’ll need some very interesting cabling attached to a somewhat beefy device to actually run live inference from this data, plus to drive the speech synthesis.

tbenst 3y ago

This is a super cool device. Note that the decoding is highly limited: they decode into one of five different sentences. This is easier than five words for example as there is more information to distinguish. Unfortunately the media is blowing this way of out proportion as the larynx alone does not contain sufficient information to decode silent speech. If you also sense the lips, tongue articulators, and jaw, then general English decoding becomes possible with high accuracy (eg see our recent work here: https://x.com/tbenst/status/1767952614157848859 https://x.com/tbenst/status/1767952614157848859). It’s not in the preprint but I’ve done experiments with only the larynx recorded and performance is pretty abysmal on even a 10 word vocabulary—-hence why they did a five sentence task.

irviss 3y ago
> If you also sense the lips, tongue articulators, and jaw, then general English decoding becomes possible with high accuracy A bit OT but I see this frequently and I'm curious. Why do you English speakers (or just a US phenomenon?) tend to use the word "English" instead of "language", "linguistic" or one of its related words to refer to a general concept?
- atopal 3y ago
  There are about 6000 spoken languages around the world with an extreme variety in how they produce meaning. How could you make sweeping statements about all of them?
- x1798DE 3y ago
  Not OP, but as a native English speaker and former scientist (though not in this area), I would interpret "x does y on English tasks" to mean "we tested this in English and don't know if the effect generalizes to other languages".
  - thaumasiotes 3y ago
    In this case we do know if the effect generalizes to other languages. It cannot fail to; the larynx, lips, tongue, and jaw are almost all there is. For example, vowels are conventionally defined by jaw position ("height"), tongue position ("frontness"), and lip configuration ("rounded" or not). You might miss some things like creaky voice or ejectives, you'll probably miss aspiration, but all that does is give you a worst-case scenario analogous to a native speaker trying to understand someone with a foreign accent. Extremely high accuracy will be possible.

ImHereToVote 3y ago

I'll take the whole lot.

feverzsj 3y ago

How it compares to electrolarynx, which give you robotic sound.

croemer 3y ago

"Speaking" is a hyperbole. It allows you to say exactly 5 phrases with 95% accuracy, after repeating each sentence 100 times. In other words, it's totally useless. The sentences are so different that they can be distinguished almost entirely by length. I'm very surprised anyone thinks this is useful. Excerpt: "A brief demonstration was made with five sentences that we had selected for training the algorithm (S1: “Hi Rachel, how you are doing today?”, S2: “Hope your experiments are going well!”, S3: “Merry Christmas!”, S4: “I love you!”, S5: “I don’t trust you.”). Each participant repeated each sentence 100 times for data collection." I never read a press release from a university, it's always exaggerated. Original study: https://www.nature.com/articles/s41467-024-45915-7 https://www.nature.com/articles/s41467-024-45915-7

light_hue_1 3y ago
Exactly. They took a neat device and wrapped it in a BS story that's wildly unscientific. Nature and Science don't mind if people outright lie about what their research means as long as it gets hits. The paper is pretty much just as bad. This is how mistrust for science slowly builds up when people publish obvious falsehoods.

joshspankit 3y ago

Is it just me, or does anyone else think it would be amazing to use upcoming voice assistants with something like this letting you “talk” silently?