9 ms·

Deep Learning enables hearing aid wearers to pick out a voice in a crowded room

188 points by WheelsAtLarge 10y ago

csours 10y ago
Tech starts 1/3 the way down the article (ctrl+f "clean speech") > My lab was the first, in 2001, to design such a filter, which labels sound streams as dominated by either speech or noise. With this filter, we would later develop a machine-learning program that separates speech from other sounds based on a few distinguishing features, such as amplitude (loudness), harmonic structure (the particular arrangement of tones), and onset (when a particular sound begins relative to others). > Next, we trained the deep neural network to use these 85 attributes to distinguish speech from noise. > One important refinement along the way was to build a second deep neural network that would be fed by the first one and fine-tune its results. While that first network had focused on labeling attributes within each individual time-frequency unit, the second network would examine the attributes of several units near a particular one > Even people with normal hearing were able to better understand noisy sentences, which means our program could someday help far more people than we originally anticipated > There are, of course, limits to the program’s abilities. For example, in our samples, the type of noise that obscured speech was still quite similar to the type of noise the program had been trained to classify. To function in real life, a program will need to quickly learn to filter out many types of noise, including types different from the ones it has already encountered oh
- jannotti 10y ago
  That last part seems to really put a damper on things, since the problem the author describes is that a person with a hearing aid requires speakers to take turns. Apparently, when people speak together, the multiple voices clash. Even if the hearing aid amplifies voices only, that problem remains. Still, cooler than a lot of things Deep Learning is being applied to these days.
- sdrothrock 10y ago
  As someone with a cochlear implant who lives with the consequences of overly clever programmers who thought they'd "help" by filtering out noise and volume and whatever else... I really wish they wouldn't. This is a technology that makes me so angry some days that I sometimes wonder if it was worth getting implanted, even though I know it was.
  - Kluny 10y ago
    Which one have you got? I know what you mean with the "helpful" bullshit. My conventional hearing aid, on my left ear, has this "smart" mode where it tries to detect speech vs noise, and change the volume or the directionality of the microphone to compensate. You end up with this wildly fluctuating volume all the time where it feels like stationary objects are coming at you. I had them turn that feature off asap. On the other hand, my cochlear implant (right side), has a directional microphone that's actually incredibly useful in noisy situations. Combined with the directional mic on the hearing aid, I can actually hear almost as well as a normal person in a crowded bar, after 20+ years of avoiding them because of how impossible they were to cope with. I strongly recommend it if you can get it - the Nucleus Freedom 6. I'm saving my pennies up to get a second one.
    - sdrothrock 10y ago
      I have the Advanced Bionics Harmony BTE. Since my implant is AB, I wouldn't be able to get the Nucleus Freedom 6. I have an in-ear mic, which does wonders for reducing surrounding noises and also for letting me use a phone normally, but my main issue is with the software itself; I've had issues with it since implantation and they've always been pooh-poohed by audiologists at Hopkins, Tokyo University, and Toranomon. The biggest problem is that it seems to operate on some kind of averaging system -- when there's a noise that's louder than the recent average, everything just cuts out for a few seconds. This is especially noticeable in the morning, where I've just woken up and am trying to get to work, but there are cars and trains etc. making noise and making my hearing cut in and out constantly, which not only drives me up the wall but gives me a terrible headache.
      - 10y ago
tobtoh 10y ago
> The greatest frustration among potential users is that a hearing aid cannot distinguish between, for example, a voice and the sound of a passing car if those sounds occur at the same time. The device cranks up the volume on both, creating an incoherent din. It may be a simplification of the article that I'm misinterpreting, but as someone who got a hearing aid in early 2016, that's not how (modern) hearing aids work. I got my hearing tested which enabled a frequency response of my hearing loss to be plotted (my hearing at low frequencies is fine, at higher freq I have moderate loss). My hearing aid is then tuned to match the inverse of that freq plot (ie boost volume of high frequencies, leave low freq alone). You don't actually want a HA that arbitrarily boosts 'speech' since that won't be matched to your needs and has unintended side effects (like music can sound overly harsh/bright) because un-needed frequencies are being boosted or supressed). -- On a tangent, after I got my new HAs, I complained to the audiologist that they didn't sound very good. Everything sounded far too crisp. She pointed out that having lived with hearing loss for 5-6 years, I actually had almost no idea what something should sound like since my brain had got used to a world with muted high frequency sounds. That blows my mind ... a bit like how do you know the color green is green. Maybe it's purple, but you have been told by someone else that it's green. After a few weeks, my brain re-learnt what sound should sound like and now it sounds 'normal' with HA in. Without HA, everything is a little more muffled (as you would expect) and I really notice how much I used to struggle understanding people (I believe my untreated hearing loss contributed to me losing my job a couple of years ago). Hearing aids have changed my quality of life (at age 40).
- chx 10y ago
  I probably need a hearing aid but I am very reluctant to go for a testing because the last time I had one (very long ago and very far away from Canada, I admit) I thought it was incredibly imprecise: they asked me to press a button when I could hear a sound and I was absolutely unsure whether I heard really something already or just imagined. Is it still so very subjective?
  - tobtoh 10y ago
    Yup - it's still done by that method. Although they do run through multiple frequencies multiple times to try and get consistency. I know what you mean though ... especially if you suffer a bit from tinnitus, it's sometimes really difficult to distinguish if you are hearing a high freq test tone, or just the tinnitus 'noise'.
chas 10y ago
This approach surprised me. Why are they doing feature extraction and then feeding that into a DNN? It seems much more straightforward to have the input of the network be noisy samples and the output be clean samples a la super resolution[0] in images. They probably wouldn't want to use fully-connected layers in that instance, but I don't see any fundamental barriers if they have enough computational power to run a neural network already. Am I missing something? [0] https://arxiv.org/pdf/1603.08155.pdf https://arxiv.org/pdf/1603.08155.pdf
- eb0la 10y ago
  This is the paper they submitted: http://web.cse.ohio-state.edu/~dwang/papers/CWYWH.jasa16.pdf http://web.cse.ohio-state.edu/~dwang/papers/CWYWH.jasa16.pdf I'm not an expert in the hearing field, but makes sense for me: they probably know that some filters work (and are actually used) for hearing, and feeding that preprocessed data would save them training and hyperparameter tuning time.
- Eridrus 10y ago
  The only thing I've seen run on raw audio are WaveNet models, and those are way too expensive to get into an embedded chip, no public real-time implementations exist, though Baidu had a paper which claimed real-time execution speed on some server class Intel CPU last week. They do mention that their CPU implementation could be parallelized too.
- biggieshellz 10y ago
  The filter bands they're talking about are Bark bands (https://en.wikipedia.org/wiki/Bark_scale https://en.wikipedia.org/wiki/Bark_scale) and are actually representative of the way the ear perceives loudness. In a traditional hearing aid, you might have a compressor for each of these Bark bands to counteract the effects of loudness recruitment (http://www.sens.com/helps/helps_d03.htm http://www.sens.com/helps/helps_d03.htm).
- adinisom 10y ago
  That might work, although I think there are two limitations: 1) Hearing aids have a 10ms latency budget. So no matter how much processing they can do, they're limited by how many samples they can look ahead and that limits the design of the filters. The brain can presumably look ahead further to separate sound streams so I think it's pretty impressive that ideal binary masking works. 2) Hearing aids have a power budget. The ones I've looked at achieve low power by running a FIR filter in hardware to shape the sound while a DSP classifies the sound and adjust the filter taps. The DSP doesn't have to run at the same rate as the filter. That seems well matched to the binary filter approach. Likewise features extraction might not run at the same rate as the DNN.
nilsb 10y ago
This sounds like something that could have a potential use for non-hearing-impaired people who have sensory overload issues (e.g. autism).
petra 10y ago
Is deep learning useful for noise-cancellation headphones/earphones? It's extremely hard to design great noise cancelling, only a very few do it, and hence the prices are very high. If deep learning can reduce costs and increase competition here, i think this sector could really grow.
rini17 10y ago
Good read! Is it possible to know which hearing aids brand has this? Pity that the HA manufacturers are so secretive what their buzzwords mean. It's impossible to get good comparison between offered features.
dwighttk 10y ago
I was thinking they were on their way to making a Photoshop for audio until I heard the before and after samples
imron 10y ago
Reinvents is such an awful term. They didn't 'reinvent' anything, they improved upon an existing shortcoming.
nabla9 10y ago
How is this method better compared to independent component analysis?
- jfsantos 10y ago
  For one thing, independent component analysis needs to process signals from as many microphones as there are sources to work properly.
  - nabla9 10y ago
    This is true for vanilla ICA. Independent Subspace ICA models can be applied to more signals than sources problems. It's also possible to use different decomposition methods, or subtracting already detected signals.
meow_mix 10y ago
Really happy to have this guy as our neural nets professor at ohio state :)
phaedrus 10y ago
I can't pick out a voice in a crowded room, or indeed separate speech from any sort of background noise. In ideal listening conditions I miss words making sentences not make sense, and often I don't realize someone has started talking until I've already missed the first sentence. However I don't have any physical hearing problem. Each time I've gotten my hearing tested I've been told my tonal hearing is perfectly normal. Yet the problems I have with picking out and understanding speech are absolutely debilitating, and I can't get anyone to understand that it is a disability and that it is real. At my insistence my audiologist administered a speech processing test, but I was nonplussed to discover this test is completely unrealistic and did not at all match the situations I have trouble with. The way it worked was that it would mix a perfectly clear speech track with white noise, or a repetitive loop of background speech or cafeteria noise. But since the sound streams were mixed together so artificially, my brain could separate the audio streams based on source track, words or no. And since the "interruption" loops were repetitive, my brain could learn the pattern and discount it. So of course I passed that test, too. The speech processing problems I have occur in real environments when executing functions of daily life. In the end the audiologist told me that maybe my problem is that I have ADHD and that my attention isn't able to lock-on or stay with a conversation well. He didn't know of anyone in my area who treated adults with ADHD, but promised to send me a referral. I'm guessing he never found anyone, because that referral never came. However it eventually led to me getting diagnosed and treated for ADHD on my own. (Although it took almost 2 years to even get an appointment.) I've found that getting a diagnosis and medication for ADHD has improved my life immensely. However it has not helped with the original problem; I still can't separate speech from other noises. I resonate with the commenter who says he thinks his undiagnosed (physical) hearing loss once contributed to him losing a job. At work, I find excuses to hide/disconnect my phone because I have so much trouble making out what people are saying over a phone. I use chat and IM, and write everything down or ask for things written down. Still, sometimes I'll miss or not understand some verbal instruction and get in trouble. It also causes relationship problems - so many misunderstandings, misheard words, doing the opposite of what my spouse asked or not realizing she said something to me. I avoid some social activities because I know that background noise there will prevent me from participating, or because mishearing people might lead to a social gaffe or a dangerous misinterpretation of safety instructions. I can't read lips to get by, either - whatever it is in my brain that affects speech processing affects lip reading equally, if not worse, and sometimes when I'm receiving the "all circuits are down" message from my speech centers, I can't even understand someone's sentence no matter how many times they repeat it. But if they write it down on a note I can understand it. In a way, it's like the inverse of dyslexia. Anyway, not that I have much hope of an answer, but anyone know where I can go to talk about it or what kind of doctor would be actually interested and not just brush this off?

188 points by WheelsAtLarge 10y ago

Tech starts 1/3 the way down the article (ctrl+f "clean speech") > My lab was the first, in 2001, to design such a filter, which labels sound streams as dominated by either speech or noise. With this filter, we would later develop a machine-learning program that separates speech from other sounds based on a few distinguishing features, such as amplitude (loudness), harmonic structure (the particular arrangement of tones), and onset (when a particular sound begins relative to others). > Next, we trained the deep neural network to use these 85 attributes to distinguish speech from noise. > One important refinement along the way was to build a second deep neural network that would be fed by the first one and fine-tune its results. While that first network had focused on labeling attributes within each individual time-frequency unit, the second network would examine the attributes of several units near a particular one > Even people with normal hearing were able to better understand noisy sentences, which means our program could someday help far more people than we originally anticipated > There are, of course, limits to the program’s abilities. For example, in our samples, the type of noise that obscured speech was still quite similar to the type of noise the program had been trained to classify. To function in real life, a program will need to quickly learn to filter out many types of noise, including types different from the ones it has already encountered oh

jannotti 10y ago
That last part seems to really put a damper on things, since the problem the author describes is that a person with a hearing aid requires speakers to take turns. Apparently, when people speak together, the multiple voices clash. Even if the hearing aid amplifies voices only, that problem remains. Still, cooler than a lot of things Deep Learning is being applied to these days.
sdrothrock 10y ago
As someone with a cochlear implant who lives with the consequences of overly clever programmers who thought they'd "help" by filtering out noise and volume and whatever else... I really wish they wouldn't. This is a technology that makes me so angry some days that I sometimes wonder if it was worth getting implanted, even though I know it was.
- Kluny 10y ago
  Which one have you got? I know what you mean with the "helpful" bullshit. My conventional hearing aid, on my left ear, has this "smart" mode where it tries to detect speech vs noise, and change the volume or the directionality of the microphone to compensate. You end up with this wildly fluctuating volume all the time where it feels like stationary objects are coming at you. I had them turn that feature off asap. On the other hand, my cochlear implant (right side), has a directional microphone that's actually incredibly useful in noisy situations. Combined with the directional mic on the hearing aid, I can actually hear almost as well as a normal person in a crowded bar, after 20+ years of avoiding them because of how impossible they were to cope with. I strongly recommend it if you can get it - the Nucleus Freedom 6. I'm saving my pennies up to get a second one.
  - sdrothrock 10y ago
    I have the Advanced Bionics Harmony BTE. Since my implant is AB, I wouldn't be able to get the Nucleus Freedom 6. I have an in-ear mic, which does wonders for reducing surrounding noises and also for letting me use a phone normally, but my main issue is with the software itself; I've had issues with it since implantation and they've always been pooh-poohed by audiologists at Hopkins, Tokyo University, and Toranomon. The biggest problem is that it seems to operate on some kind of averaging system -- when there's a noise that's louder than the recent average, everything just cuts out for a few seconds. This is especially noticeable in the morning, where I've just woken up and am trying to get to work, but there are cars and trains etc. making noise and making my hearing cut in and out constantly, which not only drives me up the wall but gives me a terrible headache.
    - 10y ago

tobtoh 10y ago

> The greatest frustration among potential users is that a hearing aid cannot distinguish between, for example, a voice and the sound of a passing car if those sounds occur at the same time. The device cranks up the volume on both, creating an incoherent din. It may be a simplification of the article that I'm misinterpreting, but as someone who got a hearing aid in early 2016, that's not how (modern) hearing aids work. I got my hearing tested which enabled a frequency response of my hearing loss to be plotted (my hearing at low frequencies is fine, at higher freq I have moderate loss). My hearing aid is then tuned to match the inverse of that freq plot (ie boost volume of high frequencies, leave low freq alone). You don't actually want a HA that arbitrarily boosts 'speech' since that won't be matched to your needs and has unintended side effects (like music can sound overly harsh/bright) because un-needed frequencies are being boosted or supressed). -- On a tangent, after I got my new HAs, I complained to the audiologist that they didn't sound very good. Everything sounded far too crisp. She pointed out that having lived with hearing loss for 5-6 years, I actually had almost no idea what something should sound like since my brain had got used to a world with muted high frequency sounds. That blows my mind ... a bit like how do you know the color green is green. Maybe it's purple, but you have been told by someone else that it's green. After a few weeks, my brain re-learnt what sound should sound like and now it sounds 'normal' with HA in. Without HA, everything is a little more muffled (as you would expect) and I really notice how much I used to struggle understanding people (I believe my untreated hearing loss contributed to me losing my job a couple of years ago). Hearing aids have changed my quality of life (at age 40).

chx 10y ago
I probably need a hearing aid but I am very reluctant to go for a testing because the last time I had one (very long ago and very far away from Canada, I admit) I thought it was incredibly imprecise: they asked me to press a button when I could hear a sound and I was absolutely unsure whether I heard really something already or just imagined. Is it still so very subjective?
- tobtoh 10y ago
  Yup - it's still done by that method. Although they do run through multiple frequencies multiple times to try and get consistency. I know what you mean though ... especially if you suffer a bit from tinnitus, it's sometimes really difficult to distinguish if you are hearing a high freq test tone, or just the tinnitus 'noise'.

chas 10y ago

This approach surprised me. Why are they doing feature extraction and then feeding that into a DNN? It seems much more straightforward to have the input of the network be noisy samples and the output be clean samples a la super resolution[0] in images. They probably wouldn't want to use fully-connected layers in that instance, but I don't see any fundamental barriers if they have enough computational power to run a neural network already. Am I missing something? [0] https://arxiv.org/pdf/1603.08155.pdf https://arxiv.org/pdf/1603.08155.pdf

eb0la 10y ago
This is the paper they submitted: http://web.cse.ohio-state.edu/~dwang/papers/CWYWH.jasa16.pdf http://web.cse.ohio-state.edu/~dwang/papers/CWYWH.jasa16.pdf I'm not an expert in the hearing field, but makes sense for me: they probably know that some filters work (and are actually used) for hearing, and feeding that preprocessed data would save them training and hyperparameter tuning time.
Eridrus 10y ago
The only thing I've seen run on raw audio are WaveNet models, and those are way too expensive to get into an embedded chip, no public real-time implementations exist, though Baidu had a paper which claimed real-time execution speed on some server class Intel CPU last week. They do mention that their CPU implementation could be parallelized too.
biggieshellz 10y ago
The filter bands they're talking about are Bark bands (https://en.wikipedia.org/wiki/Bark_scale https://en.wikipedia.org/wiki/Bark_scale) and are actually representative of the way the ear perceives loudness. In a traditional hearing aid, you might have a compressor for each of these Bark bands to counteract the effects of loudness recruitment (http://www.sens.com/helps/helps_d03.htm http://www.sens.com/helps/helps_d03.htm).
adinisom 10y ago
That might work, although I think there are two limitations: 1) Hearing aids have a 10ms latency budget. So no matter how much processing they can do, they're limited by how many samples they can look ahead and that limits the design of the filters. The brain can presumably look ahead further to separate sound streams so I think it's pretty impressive that ideal binary masking works. 2) Hearing aids have a power budget. The ones I've looked at achieve low power by running a FIR filter in hardware to shape the sound while a DSP classifies the sound and adjust the filter taps. The DSP doesn't have to run at the same rate as the filter. That seems well matched to the binary filter approach. Likewise features extraction might not run at the same rate as the DNN.

nilsb 10y ago

This sounds like something that could have a potential use for non-hearing-impaired people who have sensory overload issues (e.g. autism).

petra 10y ago

Is deep learning useful for noise-cancellation headphones/earphones? It's extremely hard to design great noise cancelling, only a very few do it, and hence the prices are very high. If deep learning can reduce costs and increase competition here, i think this sector could really grow.

rini17 10y ago

Good read! Is it possible to know which hearing aids brand has this? Pity that the HA manufacturers are so secretive what their buzzwords mean. It's impossible to get good comparison between offered features.

dwighttk 10y ago

I was thinking they were on their way to making a Photoshop for audio until I heard the before and after samples

imron 10y ago

Reinvents is such an awful term. They didn't 'reinvent' anything, they improved upon an existing shortcoming.

nabla9 10y ago

How is this method better compared to independent component analysis?

jfsantos 10y ago
For one thing, independent component analysis needs to process signals from as many microphones as there are sources to work properly.
- nabla9 10y ago
  This is true for vanilla ICA. Independent Subspace ICA models can be applied to more signals than sources problems. It's also possible to use different decomposition methods, or subtracting already detected signals.

meow_mix 10y ago

Really happy to have this guy as our neural nets professor at ohio state :)

phaedrus 10y ago

I can't pick out a voice in a crowded room, or indeed separate speech from any sort of background noise. In ideal listening conditions I miss words making sentences not make sense, and often I don't realize someone has started talking until I've already missed the first sentence. However I don't have any physical hearing problem. Each time I've gotten my hearing tested I've been told my tonal hearing is perfectly normal. Yet the problems I have with picking out and understanding speech are absolutely debilitating, and I can't get anyone to understand that it is a disability and that it is real. At my insistence my audiologist administered a speech processing test, but I was nonplussed to discover this test is completely unrealistic and did not at all match the situations I have trouble with. The way it worked was that it would mix a perfectly clear speech track with white noise, or a repetitive loop of background speech or cafeteria noise. But since the sound streams were mixed together so artificially, my brain could separate the audio streams based on source track, words or no. And since the "interruption" loops were repetitive, my brain could learn the pattern and discount it. So of course I passed that test, too. The speech processing problems I have occur in real environments when executing functions of daily life. In the end the audiologist told me that maybe my problem is that I have ADHD and that my attention isn't able to lock-on or stay with a conversation well. He didn't know of anyone in my area who treated adults with ADHD, but promised to send me a referral. I'm guessing he never found anyone, because that referral never came. However it eventually led to me getting diagnosed and treated for ADHD on my own. (Although it took almost 2 years to even get an appointment.) I've found that getting a diagnosis and medication for ADHD has improved my life immensely. However it has not helped with the original problem; I still can't separate speech from other noises. I resonate with the commenter who says he thinks his undiagnosed (physical) hearing loss once contributed to him losing a job. At work, I find excuses to hide/disconnect my phone because I have so much trouble making out what people are saying over a phone. I use chat and IM, and write everything down or ask for things written down. Still, sometimes I'll miss or not understand some verbal instruction and get in trouble. It also causes relationship problems - so many misunderstandings, misheard words, doing the opposite of what my spouse asked or not realizing she said something to me. I avoid some social activities because I know that background noise there will prevent me from participating, or because mishearing people might lead to a social gaffe or a dangerous misinterpretation of safety instructions. I can't read lips to get by, either - whatever it is in my brain that affects speech processing affects lip reading equally, if not worse, and sometimes when I'm receiving the "all circuits are down" message from my speech centers, I can't even understand someone's sentence no matter how many times they repeat it. But if they write it down on a note I can understand it. In a way, it's like the inverse of dyslexia. Anyway, not that I have much hope of an answer, but anyone know where I can go to talk about it or what kind of doctor would be actually interested and not just brush this off?