57 ms·

AI headphones let wearer listen to a single person in a crowd by looking at them

956 points by keploy 2y ago

keploy 2y ago
I hope it's not just a prototype press release, will help people with hearing loss..
- dralley 2y ago
  I bet the CIA would love this, too
- ohmyiv 2y ago
  It is just a proof of concept, but they released the source code so others can build on it. Hopefully someone will create something cool, but not charge a ridiculous markup.
- Reptur 2y ago
  Could be a game changer for people with auditory processing disorder too.
- loeg 2y ago
  It's academic research -- very much not productized yet.
toomuchtodo 2y ago
Code: https://github.com/vb000/LookOnceToHear https://github.com/vb000/LookOnceToHear
chabad360 2y ago
This could actually be really helpful to me, as I have trouble hearing someone speaking in a busy room because my mind is trying to pick up everything (I think this is because of my ADHD). Having a way to significantly quiet out other noises aside for the voice of the person I'm speaking with would be amazing.
- misja111 2y ago
  I'm having the same problem, my hearing is fine but talking to people in busy clubs or cafe's is next to impossible for me. This feature would be a blessing for me!
- nsypteras 2y ago
  Ditto. I would pay big money for this if it came in an inconspicuous form factor like airpods. Hopefully it's just a matter of time before Airpods themselves can do this.
- whimsicalism 2y ago
  i don't know much about adhd/autism, but i'm pretty sure i'm somewhat autistic and have this problem really really bad. i score fine on hearing tests where i just have to listen for quiet beeps but have a lot of trouble processing what people are saying especially in a crowded setting. my dad also has this issue
23B1 2y ago
A useful tool for when you need to surveil a shady multinational called Quantum while they discuss their evil plan during a performance of Tosca.
swayvil 2y ago
Now do the same thing with video. Turn anything into a mirror, or something like that.
maxglute 2y ago
A potential feature I didn't know I needed. Have headphones with ANC on around home all the time, would be really useful if it auto passthrough my partners voice.
- __MatrixMan__ 2y ago
  The opposite would be nice too. Silence specifically this source (probably not your partner, though maybe....)
  - pyeri 2y ago
    "Just block or mute their account, eh.." would be carried to a whole new level - in actual life!
    - pests 2y ago
      This is close to a Black Mirror episode. Z-Eyes anyone? Z-Ears?
      - latentsea 2y ago
        Black Mirror was a documentary.
        seoulmetro 2y ago
        Most episodes are just slightly more invasive and unrealistic versions of technology we have had for ages rather than the usual "they saw the future".
        pests 2y ago
        I think you are being too dismissive. We watch ads, not by force, but voluntarily for free services or currencies in mobile games. Politicians can do basically whatever they want. We have cameras and even glasses that record everything we do. V-Tubers are more popular by the minute. People get blackmailed for their online activities, wrong as they might be. Kids walk around with parent-forced app's to track their location and online life. Robot dogs are being sold to the public and being used by the military. People care more about filming something or the documentary than the event itself.
astatine 2y ago
I used to think of building something related to let a mic pick up a single person to handle questions from the audience, during presentations. Will save the hassle of passing around mics. This looks like it could do just that with the headphones feeding directly into the mixer and behaving like a focused mic.
tacocataco 2y ago
You could even make a black list of people you don't want to hear!
- Narishma 2y ago
  Like in that Black Mirror episode.
qup 2y ago
Occasional bartender here. Okay!
CodeCompost 2y ago
As somebody who is hearing impaired, a feature like this would be a Godsend for me! This feature should be integrated into hearing-aids ASAP! Shut up - no, actually - keep talking and take my money!
- gedy 2y ago
  I have sensoneural hearing loss as well and fyi Bose Hearphones do have something a little like this with directional noise cancellation that helps a lot. They are discontinued but you can find them refurbished.
  - gertlex 2y ago
    My phonak HAs have some directional noise cancellation (or biasing at least; I don't have rigorous definitions for these terms)... It helps but isn't great. Has a problem that I think the AI headphones wouldn't solve either: in a (non-quiet) group setting you still need to anticipate who's going to speak when and look at them for best results. The direction bit is just biasing to preferring forward stuff (via two mics on each ear's HA). Sadly, no backwards bias option for overhearing people behind you ;)
    - Angostura 2y ago
      Version 3 will be able to analyse a room for interesting conversation and then control where you are looking via neuralink.
    - richrichardsson 2y ago
      > Sadly, no backwards bias option for overhearing people behind you ;) Put them on "backwards"; left cup on right ear and vice versa: forward facing mics now face backwards ;)
      - gertlex 2y ago
        It's a bit physically trickier than that due to curved tubing and ear molds... but I could totally "try" it with friends, e.g. rotating the BTE hearing aid 180 so it's forward, and they'd have fun too.
anonzzzies 2y ago
This but more advanced would quite nicely help with my tinnitus. I hear fine when one person is speaking (even softly and at a distance), but multiple or with music, I hear nothing.
- snorremd 2y ago
  In the same boat. I have some tinnitus (low frequency, radio static like noise) and struggle with conversation in loud places with lots of background noise and conversation. If I sit in a loud bar it is hopeless hearing what anyone but the closest two persons are saying. Conversation in normal settings are mostly no issue. So something that would enhance the speech of whomever I'm looking at would be super cool. Apple AirPods already have some sound shaping abilities to react to environment and mode. They also support specific voice enhancement if you put your phone down in front of the person speaking. If they ever support directional voice enhancement, like in this research, directly in the AirPods it would help me so much with social interactions in loud places.
- deleted 2y ago
  [deleted]
seydor 2y ago
I think someone made something similar in the 80s by using blind source separation techniques like ICA But this is very useful for people like me who don't hear well in the high frequencies.
29athrowaway 2y ago
https://en.m.wikipedia.org/wiki/Cocktail_party_effect https://en.m.wikipedia.org/wiki/Cocktail_party_effect
- grondilu 2y ago
  This is what this post (or rather just the title, tbh) immediately reminded me of. I remember learning about it in the early 2000s. It was considered a very challenging problem, with very important applications, most notably speech recognition in natural settings. I wonder what is the current status on this. Is this considered solved nowadays?
a-dub 2y ago
cue 2024 gene hackman!
- Findecanor 2y ago
  Is that a reference to the 1978 Superman movie?
  - selimthegrim 2y ago
    https://en.m.wikipedia.org/wiki/The_Conversation https://en.m.wikipedia.org/wiki/The_Conversation
dboreham 2y ago
Presumably the tv ad would feature Gene Hackman. Edit: an AI simulation of Gene Hackman.
surfingdino 2y ago
How does it solve the problem of humans being able to detect that someone's looking at us? We tend to stop talking when we sense someone's staring at us.
- happyopossum 2y ago
  That’s just…. Weird. Conversation couldn’t exist if we were like that, nor would any form of public address. The only behavior close to that I can think of is when someone is looking at someone expectantly, and trying to break in.
- tzs 2y ago
  You don’t need to stare at them. You only have to look at them for a moment to tell the system who you want to listen to. Then you can look away and it keeps listening to them. Actually from the description it sounds (no pun intended!) like you wouldn’t even have to look at them. You just would need to be facing their direction. You could be looking at something else in the direction, like the ground in front of you. When you tell it to start listening to the person you are looking at what it really does is start listening to the person whose sound is coming from that direction, which it figures out from arrival times at both ears.
thrawn0r 2y ago
This could easily hold a library of voices that you interact with (e.g at a bigger table of friends and family) and let you toggle in and out voices that are relevant. Apple please include this feature for your Airpods, thanks! :)
p0w3n3d 2y ago
No need to stand close to the Big Brother's telescreen anymore
risfriend 2y ago
This is stuff for spy movies.
deleted 2y ago
[deleted]
keploy 2y ago
Imagine it helping people with Autism and ADHD! ADHD people have hard time listening to 1 person because part of the brain tries to listen to all other conversations going around.
- genewitch 2y ago
  I have high octane AD[H]D and as my hearing "goes" so does my ability to listen to people with noise around. For at least 12 years i've had to warn my wife and children if they're not facing me when they talk i can't hear them - but i was recently let in on the fact that other people consider them mumblers, so. I have a heard time hearing most people on the phone, especially call centers. Prior to what i would describe as idiotic practices in my 20s and 30s, i could focus rapt attention on as few or as many people as required in nearly any situation, parties, bars, whatever. Music venues obviously not, but looking at someone and talking loudly near their ear (vice versa) worked fine. My hearing issues are similar to my FIL's, who is 50% deaf in one ear and 95%+ in the other. if you're sittin on the wrong side, you're getting a lot of smiles and nods, because "eh?" gets old. real. fast. However, i can hear a raccoon messing around outside, and no one else seems to hear it; also a phone notification going off in the next room, as examples. my hearing "feels" fine, except in the very specific circumstance of human speech recognition. I also have tinnitus sometimes - it comes and goes, and if i concentrate i can focus it to the point of wincing, sometimes. interestingly i have to turn down the master volume of games when i am voice chatting. any amount of noise from the game will interfere with my ability to process people speaking, especially if the game has a lot of talking. Additionally, i cannot watch any tv or film produced after about 2000 or so without subtitles, regardless of how fancy the "5.1 surround" center channel DSP is. sorry for rambling, i suppose i don't think ADHD has anything to do with my hearing loss or issue; I should have worn hearing protection more when i was younger and i'm not regretting it much now, but when i can't hear music i enjoy i'll be pretty sad about that.
  - follower 2y ago
    > suppose i don't think ADHD has anything to do with my hearing loss or issue; I wouldn't be so quick to dismiss that as a potential factor, auditory processing issues are known to something that can be connected to ADHD in some people. (Also mentioned by a few other people in the comments.) Couple of links that go into more details: * https://en.wikipedia.org/wiki/Selective_auditory_attention https://en.wikipedia.org/wiki/Selective_auditory_attention * https://en.wikipedia.org/wiki/Auditory_processing_disorder https://en.wikipedia.org/wiki/Auditory_processing_disorder * https://en.wikipedia.org/wiki/Cocktail_party_effect https://en.wikipedia.org/wiki/Cocktail_party_effect > Additionally, i cannot watch any tv or film produced after about 2000 or so without subtitles, regardless of how fancy the "5.1 surround" center channel DSP is. IMO that's potentially more because audio engineers for home movie releases are... ok, well, let's just say, "have different opinions on how to mix audio with dialogue than me". :) It's a very common compliant. > I should have worn hearing protection more when i was younger Shouldn't we all? :) Still worth trying to protect what you have now--I've used the "Etymotic" brand ear plugs mentioned elsewhere in the comments which are intended to more evenly reduce sound levels without just "muffling" everything. > but when i can't hear music i enjoy i'll be pretty sad about that. Indeed. Understandably.
stubish 2y ago
Curious what sort of processing power or chipsets the 'onboard embedded computer' needs. Could this be an iPhone app? Or is this going to require new, specialized hardware to commoditize?
bernardlunn 2y ago
I am in the market but “ The system is not commercially available”. This is a perfect opportunity for Apple.
deleted 2y ago
[deleted]
cush 2y ago
Their paper is quoting an end-to-end latency of under 20ms… so impressive!
muhammad-saalim 2y ago
Curious if it will also help to find a missing person in the crowd.
amusingimpala75 2y ago
How much is the AI necessary for this? At least for the targeting of sounds in the line of sight, that should be fairly easy to do without AI, but I don’t know about the human voice identification.
- i5heu 2y ago
  I think one could build quite the good system with 2 directional microphones and then do some beamforming or how it is called to isolate the depth one want to perceive. But this is super expensive since you need calibrated mics etc. The biggest advantage of neural nets in this field is that you can use a dirt cheap microphone and postprocess it so good that it is good enough or even very good for humans.
- KaiserPro 2y ago
  Depends on how you do it. If you have good eyetracking, a microphone array and decent object tracking on your AR glasses, then you don't really need much "AI" (ie you have access to https://facebookresearch.github.io/projectaria_tools/docs/ARK/about_ARK https://facebookresearch.github.io/projectaria_tools/docs/AR...) but its not quite possible to do it all on device yet. However its not far off.
- INTPenis 2y ago
  Directional mics were a toy 30 years ago, but an AI that can pick out a single voice and isolate it for you is quite the contemporary achievement.
  - willis936 2y ago
    Yeah I'm not really sure what's going on here. Sonar has been using ML classifiers for decades but afaik stream splitting with 100% confidence is currently considered magic. So what did they apply or what advance did they make? Afaict they threw some audio into a GPT blender without a closer look at what's being done. Edit: I found the link to the paper. It isn't stream splitting so much as it is GPT-assisted beamforming estimation. Good stuff for sure. https://dl.acm.org/doi/10.1145/3613904.3642057 https://dl.acm.org/doi/10.1145/3613904.3642057
gexla 2y ago
They couldn't use this to listen to me. They would just get "I am just a large language model, I can't help you with that." I use a lot of curse words. ;)
- hfjtifkenf 2y ago
  I asked gpt to translate for me the lyrics of a recent popular song containing the word "puta" and it just refused "I'm sorry, I can't help you with that". When I insisted it just ended the conversation.
  - gexla 2y ago
    Right, it wouldn't help you if you wanted to do something like "overhear" tips from attendees of a meth cooking convention.
    - doktrin 2y ago
      It probably should. The amount of ancillary and contextual information required to correctly distinguish between legal and illegal settings and application is invasively high.
      - genewitch 2y ago
        clbuttic AI nonsense
  - vsnf 2y ago
    YouTube's automatic captions do this for any word on an ambiguous blacklist. Not only is this annoying when reading the captions, but I imagine that for the hearing impaired, it would also be condescending to have Google tell you what you can and cannot read, especially since the hearing un-impaired experienced it just fine.
    - mschuster91 2y ago
      And then there's TikTok who will downrank your content if its AI catches you using words like "kill" or "suicide" [1]... and so, as many creators cross-publish on YouTube Shorts as well, it also automatically degrades the content there as the creators use "euphemisms" instead. And then young people snag that up and literally write that way on Reddit. No thanks, I don't want to hear or read "unalived" again. And for fucks sake it's high time the US government steps in on Tiktok - when the CCP literally influences how our children speak it's gone way too far! [1] https://www.wired.com/story/algorithms-suicide-unalive/ https://www.wired.com/story/algorithms-suicide-unalive/
xracy 2y ago
Feels like what AI should be used for... "filtering out the noise" rather than creating it.
eisbaw 2y ago
aka beam-forming. No AI needed, just good mics.
- tzs 2y ago
  How would you track the target with beam-forming? If the target left the room and later returned how would you recognize this and resume tracking them?
- 72deluxe 2y ago
  If you watch the second half of the video, it still picks up the person when they're not facing them (walking around a fountain), so it's only used to target the person for the initial capture.
JSDevOps 2y ago
How is this AI? Not just some form of a parabolic microphone
- voidUpdate 2y ago
  This is using a pair of any old commercial microphones that you can attach to your headphones, without looking like you're from the CIA and pointing a spy microphone at people
- tzs 2y ago
  With a parabolic microphone you have to keep it aimed at the person you want to listen to. If they or you are moving (other than directly toward or away from each other) you will need to keep moving the microphone. With this you look at someone, signal that you want to keep listening to them, and the AI learns their voice. It then lets that voice through the noise cancelling system even as you or they move around the room or you look elsewhere.
- kleiba 2y ago
  Ask the marketing team that, they'll explain.
m463 2y ago
get two of these for e2e communication (peer -> ear)
i5heu 2y ago
This remembers me of NVIDIA RTX Voice [0]. Although not made to isolate single persons, this is quite impressive. I hope that this single person isolation will find it's way to consumer noise-cancelling headphones [0] https://www.youtube.com/watch?v=uWUHkCgslNE https://www.youtube.com/watch?v=uWUHkCgslNE
yosito 2y ago
This will appeal to eavesdroppers.
andrewstuart2 2y ago
This reminds me a lot of https://github.com/xiph/rnnoise https://github.com/xiph/rnnoise and my use of it locally. It zeroes in on voice via RNN which seems to beat most other noise detection filters I've tried. Unfortunately, I mostly disable it these days since it's a bit harder to tune than I'm up for, but it's by far the most promising local noise reduction I've used.
foreigner 2y ago
I'll bet they achieve commercial success with the reverse application. Imagine being able to mute that one obnoxiously loud person with an annoying voice at a party!
- devjab 2y ago
  I think this is the wildest “I guess I’m” old moment I’ve experienced… Do people wear headphones at parties?
  - kortilla 2y ago
    No?
  - SllX 2y ago
    Not unless it’s a silent disco or you’re the DJ.
  - resolutebat 2y ago
    People (well, teenagers) wil.wear headphones to the dinner table if you let them.
    - balfirevic 2y ago
      That makes much more sense than wearing them at the party.
  - aqme28 2y ago
    I guess you’re not old enough. This sounds like a feature for hearing aids.
    - devjab 2y ago
      When you’re old you can just turn them off entirely. Who’s you got talking to you at a party you want to hear anyway?
  - swiftcoder 2y ago
yesbut 2y ago
This is not AI.
- kleiba 2y ago
  Marketing begs to differ.
algasami 2y ago
When ANC headphones came out, my friends thought about something like filtering certain sounds away. I bet many people have also had this kind of idea, but nevertheless, haven't actually built it. This looks intriguing, and with open-source POC code, it seems promising.
- __debugger__ 2y ago
  2nd gen AirPods Pro’s Adaptive Noise Cancellation feature does exactly that.
KaiserPro 2y ago
One thing that the HN crowd should appreciate is just how expensive and shit hearing aids are. go and look the up the price, they are deeply expensive, even for basic "make it louder" type aids. Worse still, because they interfere with your ear, you tend to loose the ability to "steer" your hearing. This means that you can't tune out other conversations/noises or stuff. The one good side effect of facebook spending billions on its (probably) futile search for practical and popular AR is https://www.projectaria.com/glasses/ https://www.projectaria.com/glasses/ Which is a (cheap) platform to do experimentation for AR type actions. However it has eye tracking, microphone array and front facing cameras, so it can be fairly easily modified into being a steerable microphone.
- wazoox 2y ago
  The French youtuber Deus Ex silicium made a through analysis. Basically, hearing aids are slightly worse than ordinary wireless headphones, but 10 times more expensive. https://www.youtube.com/watch?v=QPmwfbLPHG8 https://www.youtube.com/watch?v=QPmwfbLPHG8
  - abdullahkhalids 2y ago
    I doubt that is true. The hearing aids sold as medical devices, have to satisfy regulatory standards. So a lot of incentive to use older reliable tech. Not to mention, it probably takes a couple of years to get the certification for a device. So, any device to market is easily 2-3 years old tech by definition.
    - eequah9L 2y ago
      I think you meant "I don't doubt"? Because none of what you said sounds like a counterargument to what the parent said.
    - datpiff 2y ago
      > Not to mention, it probably takes a couple of years to get the certification for a device. So, any device to market is easily 2-3 years old tech by definition. Approval can be quicker for hearing aids as there are already works-alike devices on the market. 510(k) clearance takes less than 6 months.
deleted 2y ago
[deleted]
tromp 2y ago
> To use the system, a person wearing off-the-shelf headphones fitted with microphones taps a button while directing their head at someone talking. The sound waves from that speaker’s voice then should reach the microphones on both sides of the headset simultaneously; there’s a 16-degree margin of error. Perhaps the accuracy of identifying the correct voice could be vastly increased by adding video input. The AI can then try to match the various voices with the lip movements in the center of the video, basically lip reading.
albert_e 2y ago
I love this. I know this is just the beginning and the tech and UX will mature a lot - but being able to consciously choose what we allow into our sensory world would be a great superpower to have. In the distant future this will all be embedded inside a cochlear (neural?) implant. You can "save" known voices, prioritize them, identify various scenes/modes automatically like meetings/parties/concerts/driving/walking etc, know when to allow external sounds in (alarms, honks, someone calling your attention, etc) And with great power yada yada. I can already imagine a few ways this can be misused / abused / create non-existent challenges and problems too. But I am (cautiously) optimistic that we as human race will collectively figure out how to steer these new technology applications into net positive territory. 2040: iAudio and xSmell blamed for people losing connect with nature's sounds (like bird chirps and flowing streams) and smells (petrichor) - things that inspire us, make us creative, make life worthwile, and make us humans.
fergie 2y ago
This is an actual thing that could work: AI's ability to "stem" voices and instruments is really impressive.
Ibreezy 2y ago
Ok
briansm 2y ago
Bit annoying that they added ambient music over the demo youtube video, spoiling the one thing you want to demonstrate.
- genewitch 2y ago
  this is incessant and annoying, i like watching mechanics complain about cars, but most of them put a music bed or whatever behind their speech and will ask "you hear that?" no, i don't, because of your kenny G knockoff playing at low volume.
Wildgoose 2y ago
My daughter has an auditory impairment which she describes as "brain deaf". Basically, her hearing is perfect but her brain struggles to process sound in a noisy environment; she can't single out what she is listening to. This sounds perfect for her!
- geros 2y ago
  An audiologist described this to me as having an "audio processing disorder". Hope this helps.
- follower 2y ago
  Mentioned elsewhere in the comments but in case it's helpful: * https://en.wikipedia.org/wiki/Selective_auditory_attention https://en.wikipedia.org/wiki/Selective_auditory_attention * https://en.wikipedia.org/wiki/Auditory_processing_disorder https://en.wikipedia.org/wiki/Auditory_processing_disorder * https://en.wikipedia.org/wiki/Cocktail_party_effect https://en.wikipedia.org/wiki/Cocktail_party_effect
bdw5204 2y ago
In my experience, most people don't seem to understand the concept of noise cancelling headphones and will still try to talk to people who clearly can't hear them. I can't imagine it'd be any different for these AI headphones in practical use. Probably worse because the person you're actually trying to talk to might think you can't hear them.
- hkwerf 2y ago
  Maybe it's the opposite? ANC headphones are lacking a means of communicating to people that you're not hearing them. However, if your ANC headphones link you looking at someone with unmuting them, that communication barrier is crossed. This is particularly nice, as looking at someone is commonly a signal for "I'm listening to you".
brap 2y ago
I don't technically have a hearing problem, sometimes when there's a lot of noises occurring at the same time I hear it as one jumble.
- giantg2 2y ago
  I think technically that is considered a hearing problem if you're unable to separate out voices. I forget the name of it though. But this can actually disqualify you from certain types of jobs, such as the police and military.
serial_dev 2y ago
If the size could shrink to the size of a small earplug, I'd love to use this as a person who is not hearing-impaired (at least they couldn't diagnose me with it, so now I'm not sure if their diagnostics sucks, or I'm just a normal person and others pretend better that they hear everything well). In groups and with friends, it's inevitable that you end up in a busy restaurant or a bar, and it always frustrates me that I don't hear something, I ask the person to repeat only to not hear it again, usually because they repeat it at the same low level (considering the circumstances). Missing jokes and throwaway comments is even worse ("hey what are you all laughing about, I didn't hear it, could you repeat it for me like three times until I hear it").
- nolongerthere 2y ago
  > I'm not sure if their diagnostics sucks, or I'm just a normal person and others pretend better that they hear everything well This is one of those frustrating gaslighting things that is half true in that half the time I also pretend to hear what someone else is saying even though I couldn’t just because it’s not really important and making a big deal about it (ie asking them to repeat it at continuously louder decibels) can get awkward.
  - abcdefg_ 2y ago
    So I'm not alone. I'm in my mid-forties and have experienced a significant decline over the last few years. Now I can rarely distinguish one voice in moderate background noise (conversations, music) without leaning towards them, cupping my ears. Sometimes I just have to give up and try to nod or smile at the right time as the conversation goes on around me. I recently had a test at an ENT doctor who told me my hearing is fine and insinuated I was wasting his time. The test was listening to high-pitched beeps over white noise, which isn't representative of the problem. Distinguishing one particular tone over several similar ones would be more like it.
    - glenngillen 2y ago
      Hey, there's dozens of us! :P I wrote about my experience with this last year: https://news.ycombinator.com/item?id=35897515 https://news.ycombinator.com/item?id=35897515 I did exactly the type of diagnosis you're talking about. It was quite good at how it simulated a noisy environment with a bunch of background chatter and then a single voice you were meant to listen to that would repeat various patterns of words with various combinations of lower speaking volume and/or higher background noise. One thing I wish I'd made a point of at the time was the fact that, despite being an apparently soundproof booth with headphones on, I could definitely hear people talking in the waiting room and another audiologist in an adjacent room. Though I'm not sure it would have materially changed their lack of diagnosis (they'd already detected I could hear into negative decibels). I still don't have a diagnosis, but I'm increasingly coming around to the idea that maybe it's not that my hearing is bad but that I actually hear too much. What I'd previously thought was my unability to hear people speaking on the radio in the car when everyone else clearly could wasn't because I couldn't hear the radio, it's that I can't hear it over the top of all the tyre and wind noise I'm also hearing and trying to process out. I don't think the other passengers in the car hear the rest of the noise, they only hear the radio. I bought various types of Loop earplugs and have found them fantastic for live music events. I can now hear my friends when they're talking to me! Unfortunately they greatly amplify my perception of the volume of my own voice when I talk which has the undesirable side-effect of making me talk even quieter so I feel like I'm having to yell when I want to talk to people. I've also not found them as useful as I'd hoped in restaurant-type settings.
jaustin 2y ago
Next can we have them identify ambient noises that need amplification for safety reasons, like the nearly-silent electric car about to run me over, or the bike I'm about to accidentally step in front of? As someone who spends a bit too much of my time walking around on calls, I think selective amplification of ambient sounds for safety would be amazing!
- bartread 2y ago
  Are you vision impaired?
cushychicken 2y ago
I used to work at Sonos, long before their current app update debacle and headphone debut. During the first aborted product effort to develop headphones, we were looking at a conceptual feature similar to this - selectively allowing people’s voices through the ANC chipset. I don’t recall the exact approach the DSP folks were using (I was closer to the hardware for ANC) but they were really only able to figure out how to isolate the wearer’s voice by virtue of that signal having more power than all the others. This is terribly cool. I wonder what other kinds of fun you could have with headphones. ANC chipsets are incredibly powerful and I’d wager their capabilities are not even close to fully tapped.
- fennecfoxy 2y ago
  I wouldn't ever want Sonos hearing aids. Universally Sonos units have basic functionality problems such as not reconnecting to Wifi that has gone down and come back up, especially if the Wifi has changed channel during that time. The "technical" solution is to pull the plug and reboot it (which you can't even do remotely, even if it's connected to Wifi and you want to reboot because Spotify connect on Sonos can be buggy as hell). I can keep a wifi connection up myself and always reconnect using an esp or similar TI etc module...is it so hard for the Sonos firmware devs to do something so basic?
  - richwater 2y ago
    Spotify connect on most devices is buggy as hell.
- ttpphd 2y ago
  If you read the paper, nothing has changed. They still depend on the target talker not having competing co located sounds or voice.
  - bob_theslob646 2y ago
    May you elaborate on what you mean by 'colocated sounds or voice '? Does that mean sound coming in the same direction as the person they are targeting? Does it mean sound near the person /object they are targeting?
nox101 2y ago
Sounds like a great way to spy on all people and extract all conversations. I can't wait for judges to declare that all conversation at your office must be recorded like some of them have for chat. This tech is a step to enable such a thing.
adwi 2y ago
The “cocktail party effect” externalized. Extremely cool. https://en.m.wikipedia.org/wiki/Cocktail_party_effect https://en.m.wikipedia.org/wiki/Cocktail_party_effect
- mateus1 2y ago
  Very interesting, concept. I have a friend with seeming no hearing issues that has to cup his ears when in louder environments such as bars, restaurants, etc. I’m wondering if anyone here has a solution to that.
btbuildem 2y ago
When I was a youngling, I dreamed of having headphones with the opposite power -- muting specific people. For me, it's not the hubbub of a crowd that's distracting, it's usually one or two offending specimens - like in the video example, the inconsiderate vermin using a speakerphone in public. I wonder if the problem maps easily from "select this source" to "select everything but that source"
causi 2y ago
I would like AI headphones that let me pinpoint the source of noises, such as inside a car engine.
classified 2y ago
Is it April 1st already?
nutanc 2y ago
What about the privacy concerns? So basically I can just look at a couple of people talking and eavesdrop?
- anonu 2y ago
  How does this technology change anything? Don't eavesdrop...
anonu 2y ago
By looking at them AND pressing a button. But they might be able to get rid of the button with some sensors and AI.
entico 2y ago
cant wait to see people using earphones at parties
hwbunny 2y ago
[dead]
deleted 2y ago
[deleted]
osjp1988 2y ago
[dead]
VikingCoder 2y ago
Fry_Shut_Up_And_Take_My_Money.gif
oakpond 2y ago
Honestly AI speech recognition still sucks so bad I'm basically convinced it will fall on its face in many daily use cases. I realize this is slightly tangential, but please don't replace customer support with chatbots or whatever you want to call them. It's a freaking horrible experience.
resource_waste 2y ago
>A University of Washington team Oh, so it barely works and its a proof of concept. What is the interesting thing here? We all know how sound waves work. Pretty sure this technology is old. Until there is a product here, it just sounds like you are rehashing noise cancellation. Academia has dug this grave of skepticism. I just have 0 faith this will get to market through University of Washington. Maybe it will be patented and used even less!
m3kw9 2y ago
This is the equivalent of staring but for ears.
helsinkiandrew 2y ago
Presumably this could be used to block out specific voices/sounds. There's an episode of the sci-fi show Black Mirror (White Christmas), where a person is convicted of some hideous crime and permanently blocked/made invisible and inaudible from everyone (the entire population has embedded audio/video processing enhancements by then). You can imagine future headphones where you could block out the guy in your office with the annoying laugh our download 'blocks' from the headphone appstore - no more Rick Astley or the politician you don't like etc.
ck_one 2y ago
Pretty cool what they are working on. However, I wished there would be more funding for restoring hair cells which are the root cause for most people with hearing loss. Researchers are getting closer. Dr. Chen from Harvard was able to regenerate hair cells in mature mice last year. The problem is also becoming more widespread. 30 Mio people in the US and 400 Mio people worldwide have disabling hearing loss. Regenerating hair cells and the synapses around them would also cure Tinnitus. 30 Mio x $5k for a treatment = $60B market (probably way bigger with aging population) I think we probably need more rich tech billionaires to get affected to attract large funding. What billionaires that you know are affected besides: - Brad Jacobs - Ryan from Flexport/Founders Fund
- darig 2y ago
  [dead]
MisterBastahrd 2y ago
While not exactly the same, I came across an app called Tunity the other day. It allows you to use your phone camera to catch the live audio feed of the television that you are attempting to watch, whether the audio is muted or if it's in a loud, crowded location, like a sports bar or airport. I haven't used it, but it's an interesting concept.
andy 2y ago
I opened an issue with this. Maybe someone here knows. I see a Python script I can run on my computer, I haven't tried it yet, but I think I could connect a microphone and process real-time audio and output it in real time, but I don’t know how to detect the user looking at someone. Could you tell me how that works?
- idunnoman1222 2y ago
  To use the system, a person wearing off-the-shelf headphones fitted with microphones taps a button while directing their head at someone talking. The sound waves from that speaker’s voice then should reach the microphones on both sides of the headset simultaneously; I guess by reading the article?
hpen 2y ago
Does this use Computer Vision camera or how does it work?
OkGoDoIt 2y ago
The open source code is at https://github.com/vb000/LookOnceToHear https://github.com/vb000/LookOnceToHear and the research paper is at https://arxiv.org/abs/2405.06289 https://arxiv.org/abs/2405.06289 So perhaps this is not as out of reach as many pop-science articles. I’d love to hear if anyone is able to get this working independently.
glial 2y ago
This is pretty amazing -- and a practical application of a solution to a notoriously tricky problem called the "cocktail party problem."[1] For a small subset of researchers, writing an algorithm to isolate a voice in a crowd is on par with e.g. writing an AI to play Go. [1] https://en.wikipedia.org/wiki/Cocktail_party_effect https://en.wikipedia.org/wiki/Cocktail_party_effect
jmugan 2y ago
I want to filter out all non-nature sounds. I dream of walking through the airport or the park in peace. AI seems the way to go with that since you have to predict the sound to counteract it. Good to see we are finally making progress.
chrisknyfe 2y ago
Before getting all excited that your ML model runs on your brand new 2024 macbook, before you run off to create earbuds / hearing aids with it, please try to run it on-target and see whether your model runs within your runtime budget / power budget / device size budget / battery life budget. And make sure if you're going to do bluetooth + wireless, remember that both bluetooth and wifi transmit on 2.4 GHz, and need to coordinate in order to coexist in the same IoT device. There are interconnects and wire protocols to connect the bluetooth and wifi chips together - or, preferably, you buy a chip that does both.
deleted 2y ago
[deleted]
slaymaker1907 2y ago
Oh my god, these would be absolutely amazing as someone with auditory processing issues.
darkclouds 2y ago
[dead]
wuziyue1214 2y ago
I think this is an awesome invention
masfoobar 2y ago
The concept of headphones in the last 60 years, with the exception of sound quality, comfort, and eventually being cordless.... they have not changed much in terms of style and appearance. However, I think in the next 50 years, headphones will disappear or.. should I say evolve as part of the human anatomy. Same thing for screen monitors, mouse/keyboard, smartphones, etc. Think about it. The way things are going, along with "AI" (sure buzzword in a number of ways but something that will change our way of living) many of things we use will be replaced and, likely, be simple extensions or, dare I say, be implanted. Hard drives will be a thing of the past. Everything will be (as we call today) "cloud-based" and we will be more cybernetic than we think. Of course, someone today will fear such an idea. As we slowly accept the little changes.. in 50 years we will look back and think "how did they cope without it"... a bit like how someone today look back and think "how did they cope without the internet" Many fear what they dont understand. It is the unknown. AI is a fear factor for many. For me, I accept it for what it is and the changes it will impact our lives and our careers. All I will say is -- strap yourselves in.. it will be a bumpy ride. I hope we make it through without destroying ourselves. Once we past it, the world could (finally) be at peace and to quote a famous TV show --- "to bondly go where no man has gone before!"

956 points by keploy 2y ago

keploy 2y ago

I hope it's not just a prototype press release, will help people with hearing loss..

dralley 2y ago
I bet the CIA would love this, too
ohmyiv 2y ago
It is just a proof of concept, but they released the source code so others can build on it. Hopefully someone will create something cool, but not charge a ridiculous markup.
Reptur 2y ago
Could be a game changer for people with auditory processing disorder too.
loeg 2y ago
It's academic research -- very much not productized yet.

toomuchtodo 2y ago

Code: https://github.com/vb000/LookOnceToHear https://github.com/vb000/LookOnceToHear

chabad360 2y ago

This could actually be really helpful to me, as I have trouble hearing someone speaking in a busy room because my mind is trying to pick up everything (I think this is because of my ADHD). Having a way to significantly quiet out other noises aside for the voice of the person I'm speaking with would be amazing.

misja111 2y ago
I'm having the same problem, my hearing is fine but talking to people in busy clubs or cafe's is next to impossible for me. This feature would be a blessing for me!
nsypteras 2y ago
Ditto. I would pay big money for this if it came in an inconspicuous form factor like airpods. Hopefully it's just a matter of time before Airpods themselves can do this.
whimsicalism 2y ago
i don't know much about adhd/autism, but i'm pretty sure i'm somewhat autistic and have this problem really really bad. i score fine on hearing tests where i just have to listen for quiet beeps but have a lot of trouble processing what people are saying especially in a crowded setting. my dad also has this issue

23B1 2y ago

A useful tool for when you need to surveil a shady multinational called Quantum while they discuss their evil plan during a performance of Tosca.

swayvil 2y ago

Now do the same thing with video. Turn anything into a mirror, or something like that.

maxglute 2y ago

A potential feature I didn't know I needed. Have headphones with ANC on around home all the time, would be really useful if it auto passthrough my partners voice.

__MatrixMan__ 2y ago
The opposite would be nice too. Silence specifically this source (probably not your partner, though maybe....)
- pyeri 2y ago
  "Just block or mute their account, eh.." would be carried to a whole new level - in actual life!
  - pests 2y ago
    This is close to a Black Mirror episode. Z-Eyes anyone? Z-Ears?
    - latentsea 2y ago
      Black Mirror was a documentary.
      - seoulmetro 2y ago
        Most episodes are just slightly more invasive and unrealistic versions of technology we have had for ages rather than the usual "they saw the future".
        pests 2y ago
        I think you are being too dismissive. We watch ads, not by force, but voluntarily for free services or currencies in mobile games. Politicians can do basically whatever they want. We have cameras and even glasses that record everything we do. V-Tubers are more popular by the minute. People get blackmailed for their online activities, wrong as they might be. Kids walk around with parent-forced app's to track their location and online life. Robot dogs are being sold to the public and being used by the military. People care more about filming something or the documentary than the event itself.

astatine 2y ago

I used to think of building something related to let a mic pick up a single person to handle questions from the audience, during presentations. Will save the hassle of passing around mics. This looks like it could do just that with the headphones feeding directly into the mixer and behaving like a focused mic.

tacocataco 2y ago

You could even make a black list of people you don't want to hear!

Narishma 2y ago
Like in that Black Mirror episode.

qup 2y ago

Occasional bartender here. Okay!

CodeCompost 2y ago

As somebody who is hearing impaired, a feature like this would be a Godsend for me! This feature should be integrated into hearing-aids ASAP! Shut up - no, actually - keep talking and take my money!

gedy 2y ago
I have sensoneural hearing loss as well and fyi Bose Hearphones do have something a little like this with directional noise cancellation that helps a lot. They are discontinued but you can find them refurbished.
- gertlex 2y ago
  My phonak HAs have some directional noise cancellation (or biasing at least; I don't have rigorous definitions for these terms)... It helps but isn't great. Has a problem that I think the AI headphones wouldn't solve either: in a (non-quiet) group setting you still need to anticipate who's going to speak when and look at them for best results. The direction bit is just biasing to preferring forward stuff (via two mics on each ear's HA). Sadly, no backwards bias option for overhearing people behind you ;)
  - Angostura 2y ago
    Version 3 will be able to analyse a room for interesting conversation and then control where you are looking via neuralink.
  - richrichardsson 2y ago
    > Sadly, no backwards bias option for overhearing people behind you ;) Put them on "backwards"; left cup on right ear and vice versa: forward facing mics now face backwards ;)
    - gertlex 2y ago
      It's a bit physically trickier than that due to curved tubing and ear molds... but I could totally "try" it with friends, e.g. rotating the BTE hearing aid 180 so it's forward, and they'd have fun too.

anonzzzies 2y ago

This but more advanced would quite nicely help with my tinnitus. I hear fine when one person is speaking (even softly and at a distance), but multiple or with music, I hear nothing.

snorremd 2y ago
In the same boat. I have some tinnitus (low frequency, radio static like noise) and struggle with conversation in loud places with lots of background noise and conversation. If I sit in a loud bar it is hopeless hearing what anyone but the closest two persons are saying. Conversation in normal settings are mostly no issue. So something that would enhance the speech of whomever I'm looking at would be super cool. Apple AirPods already have some sound shaping abilities to react to environment and mode. They also support specific voice enhancement if you put your phone down in front of the person speaking. If they ever support directional voice enhancement, like in this research, directly in the AirPods it would help me so much with social interactions in loud places.
deleted 2y ago
[deleted]

seydor 2y ago

I think someone made something similar in the 80s by using blind source separation techniques like ICA But this is very useful for people like me who don't hear well in the high frequencies.

29athrowaway 2y ago

https://en.m.wikipedia.org/wiki/Cocktail_party_effect https://en.m.wikipedia.org/wiki/Cocktail_party_effect

grondilu 2y ago
This is what this post (or rather just the title, tbh) immediately reminded me of. I remember learning about it in the early 2000s. It was considered a very challenging problem, with very important applications, most notably speech recognition in natural settings. I wonder what is the current status on this. Is this considered solved nowadays?

a-dub 2y ago

cue 2024 gene hackman!

Findecanor 2y ago
Is that a reference to the 1978 Superman movie?
- selimthegrim 2y ago
  https://en.m.wikipedia.org/wiki/The_Conversation https://en.m.wikipedia.org/wiki/The_Conversation

dboreham 2y ago

Presumably the tv ad would feature Gene Hackman. Edit: an AI simulation of Gene Hackman.

surfingdino 2y ago

How does it solve the problem of humans being able to detect that someone's looking at us? We tend to stop talking when we sense someone's staring at us.

happyopossum 2y ago
That’s just…. Weird. Conversation couldn’t exist if we were like that, nor would any form of public address. The only behavior close to that I can think of is when someone is looking at someone expectantly, and trying to break in.
tzs 2y ago
You don’t need to stare at them. You only have to look at them for a moment to tell the system who you want to listen to. Then you can look away and it keeps listening to them. Actually from the description it sounds (no pun intended!) like you wouldn’t even have to look at them. You just would need to be facing their direction. You could be looking at something else in the direction, like the ground in front of you. When you tell it to start listening to the person you are looking at what it really does is start listening to the person whose sound is coming from that direction, which it figures out from arrival times at both ears.

thrawn0r 2y ago

This could easily hold a library of voices that you interact with (e.g at a bigger table of friends and family) and let you toggle in and out voices that are relevant. Apple please include this feature for your Airpods, thanks! :)

p0w3n3d 2y ago

No need to stand close to the Big Brother's telescreen anymore

risfriend 2y ago

This is stuff for spy movies.

deleted 2y ago

[deleted]

keploy 2y ago

Imagine it helping people with Autism and ADHD! ADHD people have hard time listening to 1 person because part of the brain tries to listen to all other conversations going around.

genewitch 2y ago
I have high octane AD[H]D and as my hearing "goes" so does my ability to listen to people with noise around. For at least 12 years i've had to warn my wife and children if they're not facing me when they talk i can't hear them - but i was recently let in on the fact that other people consider them mumblers, so. I have a heard time hearing most people on the phone, especially call centers. Prior to what i would describe as idiotic practices in my 20s and 30s, i could focus rapt attention on as few or as many people as required in nearly any situation, parties, bars, whatever. Music venues obviously not, but looking at someone and talking loudly near their ear (vice versa) worked fine. My hearing issues are similar to my FIL's, who is 50% deaf in one ear and 95%+ in the other. if you're sittin on the wrong side, you're getting a lot of smiles and nods, because "eh?" gets old. real. fast. However, i can hear a raccoon messing around outside, and no one else seems to hear it; also a phone notification going off in the next room, as examples. my hearing "feels" fine, except in the very specific circumstance of human speech recognition. I also have tinnitus sometimes - it comes and goes, and if i concentrate i can focus it to the point of wincing, sometimes. interestingly i have to turn down the master volume of games when i am voice chatting. any amount of noise from the game will interfere with my ability to process people speaking, especially if the game has a lot of talking. Additionally, i cannot watch any tv or film produced after about 2000 or so without subtitles, regardless of how fancy the "5.1 surround" center channel DSP is. sorry for rambling, i suppose i don't think ADHD has anything to do with my hearing loss or issue; I should have worn hearing protection more when i was younger and i'm not regretting it much now, but when i can't hear music i enjoy i'll be pretty sad about that.
- follower 2y ago
  > suppose i don't think ADHD has anything to do with my hearing loss or issue; I wouldn't be so quick to dismiss that as a potential factor, auditory processing issues are known to something that can be connected to ADHD in some people. (Also mentioned by a few other people in the comments.) Couple of links that go into more details: * https://en.wikipedia.org/wiki/Selective_auditory_attention https://en.wikipedia.org/wiki/Selective_auditory_attention * https://en.wikipedia.org/wiki/Auditory_processing_disorder https://en.wikipedia.org/wiki/Auditory_processing_disorder * https://en.wikipedia.org/wiki/Cocktail_party_effect https://en.wikipedia.org/wiki/Cocktail_party_effect > Additionally, i cannot watch any tv or film produced after about 2000 or so without subtitles, regardless of how fancy the "5.1 surround" center channel DSP is. IMO that's potentially more because audio engineers for home movie releases are... ok, well, let's just say, "have different opinions on how to mix audio with dialogue than me". :) It's a very common compliant. > I should have worn hearing protection more when i was younger Shouldn't we all? :) Still worth trying to protect what you have now--I've used the "Etymotic" brand ear plugs mentioned elsewhere in the comments which are intended to more evenly reduce sound levels without just "muffling" everything. > but when i can't hear music i enjoy i'll be pretty sad about that. Indeed. Understandably.

stubish 2y ago

Curious what sort of processing power or chipsets the 'onboard embedded computer' needs. Could this be an iPhone app? Or is this going to require new, specialized hardware to commoditize?

bernardlunn 2y ago

I am in the market but “ The system is not commercially available”. This is a perfect opportunity for Apple.

deleted 2y ago

[deleted]

cush 2y ago

Their paper is quoting an end-to-end latency of under 20ms… so impressive!

muhammad-saalim 2y ago

Curious if it will also help to find a missing person in the crowd.

amusingimpala75 2y ago

How much is the AI necessary for this? At least for the targeting of sounds in the line of sight, that should be fairly easy to do without AI, but I don’t know about the human voice identification.

i5heu 2y ago
I think one could build quite the good system with 2 directional microphones and then do some beamforming or how it is called to isolate the depth one want to perceive. But this is super expensive since you need calibrated mics etc. The biggest advantage of neural nets in this field is that you can use a dirt cheap microphone and postprocess it so good that it is good enough or even very good for humans.
KaiserPro 2y ago
Depends on how you do it. If you have good eyetracking, a microphone array and decent object tracking on your AR glasses, then you don't really need much "AI" (ie you have access to https://facebookresearch.github.io/projectaria_tools/docs/ARK/about_ARK https://facebookresearch.github.io/projectaria_tools/docs/AR...) but its not quite possible to do it all on device yet. However its not far off.
INTPenis 2y ago
Directional mics were a toy 30 years ago, but an AI that can pick out a single voice and isolate it for you is quite the contemporary achievement.
- willis936 2y ago
  Yeah I'm not really sure what's going on here. Sonar has been using ML classifiers for decades but afaik stream splitting with 100% confidence is currently considered magic. So what did they apply or what advance did they make? Afaict they threw some audio into a GPT blender without a closer look at what's being done. Edit: I found the link to the paper. It isn't stream splitting so much as it is GPT-assisted beamforming estimation. Good stuff for sure. https://dl.acm.org/doi/10.1145/3613904.3642057 https://dl.acm.org/doi/10.1145/3613904.3642057

gexla 2y ago

They couldn't use this to listen to me. They would just get "I am just a large language model, I can't help you with that." I use a lot of curse words. ;)

hfjtifkenf 2y ago
I asked gpt to translate for me the lyrics of a recent popular song containing the word "puta" and it just refused "I'm sorry, I can't help you with that". When I insisted it just ended the conversation.
- gexla 2y ago
  Right, it wouldn't help you if you wanted to do something like "overhear" tips from attendees of a meth cooking convention.
  - doktrin 2y ago
    It probably should. The amount of ancillary and contextual information required to correctly distinguish between legal and illegal settings and application is invasively high.
    - genewitch 2y ago
      clbuttic AI nonsense
- vsnf 2y ago
  YouTube's automatic captions do this for any word on an ambiguous blacklist. Not only is this annoying when reading the captions, but I imagine that for the hearing impaired, it would also be condescending to have Google tell you what you can and cannot read, especially since the hearing un-impaired experienced it just fine.
  - mschuster91 2y ago
    And then there's TikTok who will downrank your content if its AI catches you using words like "kill" or "suicide" [1]... and so, as many creators cross-publish on YouTube Shorts as well, it also automatically degrades the content there as the creators use "euphemisms" instead. And then young people snag that up and literally write that way on Reddit. No thanks, I don't want to hear or read "unalived" again. And for fucks sake it's high time the US government steps in on Tiktok - when the CCP literally influences how our children speak it's gone way too far! [1] https://www.wired.com/story/algorithms-suicide-unalive/ https://www.wired.com/story/algorithms-suicide-unalive/

xracy 2y ago

Feels like what AI should be used for... "filtering out the noise" rather than creating it.

eisbaw 2y ago

aka beam-forming. No AI needed, just good mics.

tzs 2y ago
How would you track the target with beam-forming? If the target left the room and later returned how would you recognize this and resume tracking them?
72deluxe 2y ago
If you watch the second half of the video, it still picks up the person when they're not facing them (walking around a fountain), so it's only used to target the person for the initial capture.

JSDevOps 2y ago

How is this AI? Not just some form of a parabolic microphone

voidUpdate 2y ago
This is using a pair of any old commercial microphones that you can attach to your headphones, without looking like you're from the CIA and pointing a spy microphone at people
tzs 2y ago
With a parabolic microphone you have to keep it aimed at the person you want to listen to. If they or you are moving (other than directly toward or away from each other) you will need to keep moving the microphone. With this you look at someone, signal that you want to keep listening to them, and the AI learns their voice. It then lets that voice through the noise cancelling system even as you or they move around the room or you look elsewhere.
kleiba 2y ago
Ask the marketing team that, they'll explain.

m463 2y ago

get two of these for e2e communication (peer -> ear)

i5heu 2y ago

This remembers me of NVIDIA RTX Voice [0]. Although not made to isolate single persons, this is quite impressive. I hope that this single person isolation will find it's way to consumer noise-cancelling headphones [0] https://www.youtube.com/watch?v=uWUHkCgslNE https://www.youtube.com/watch?v=uWUHkCgslNE

yosito 2y ago

This will appeal to eavesdroppers.

andrewstuart2 2y ago

This reminds me a lot of https://github.com/xiph/rnnoise https://github.com/xiph/rnnoise and my use of it locally. It zeroes in on voice via RNN which seems to beat most other noise detection filters I've tried. Unfortunately, I mostly disable it these days since it's a bit harder to tune than I'm up for, but it's by far the most promising local noise reduction I've used.

foreigner 2y ago

I'll bet they achieve commercial success with the reverse application. Imagine being able to mute that one obnoxiously loud person with an annoying voice at a party!

devjab 2y ago
I think this is the wildest “I guess I’m” old moment I’ve experienced… Do people wear headphones at parties?
- kortilla 2y ago
  No?
- SllX 2y ago
  Not unless it’s a silent disco or you’re the DJ.
- resolutebat 2y ago
  People (well, teenagers) wil.wear headphones to the dinner table if you let them.
  - balfirevic 2y ago
    That makes much more sense than wearing them at the party.
- aqme28 2y ago
  I guess you’re not old enough. This sounds like a feature for hearing aids.
  - devjab 2y ago
    When you’re old you can just turn them off entirely. Who’s you got talking to you at a party you want to hear anyway?
- swiftcoder 2y ago

yesbut 2y ago

This is not AI.

kleiba 2y ago
Marketing begs to differ.

algasami 2y ago

When ANC headphones came out, my friends thought about something like filtering certain sounds away. I bet many people have also had this kind of idea, but nevertheless, haven't actually built it. This looks intriguing, and with open-source POC code, it seems promising.

__debugger__ 2y ago
2nd gen AirPods Pro’s Adaptive Noise Cancellation feature does exactly that.

KaiserPro 2y ago

One thing that the HN crowd should appreciate is just how expensive and shit hearing aids are. go and look the up the price, they are deeply expensive, even for basic "make it louder" type aids. Worse still, because they interfere with your ear, you tend to loose the ability to "steer" your hearing. This means that you can't tune out other conversations/noises or stuff. The one good side effect of facebook spending billions on its (probably) futile search for practical and popular AR is https://www.projectaria.com/glasses/ https://www.projectaria.com/glasses/ Which is a (cheap) platform to do experimentation for AR type actions. However it has eye tracking, microphone array and front facing cameras, so it can be fairly easily modified into being a steerable microphone.

wazoox 2y ago
The French youtuber Deus Ex silicium made a through analysis. Basically, hearing aids are slightly worse than ordinary wireless headphones, but 10 times more expensive. https://www.youtube.com/watch?v=QPmwfbLPHG8 https://www.youtube.com/watch?v=QPmwfbLPHG8
- abdullahkhalids 2y ago
  I doubt that is true. The hearing aids sold as medical devices, have to satisfy regulatory standards. So a lot of incentive to use older reliable tech. Not to mention, it probably takes a couple of years to get the certification for a device. So, any device to market is easily 2-3 years old tech by definition.
  - eequah9L 2y ago
    I think you meant "I don't doubt"? Because none of what you said sounds like a counterargument to what the parent said.
  - datpiff 2y ago
    > Not to mention, it probably takes a couple of years to get the certification for a device. So, any device to market is easily 2-3 years old tech by definition. Approval can be quicker for hearing aids as there are already works-alike devices on the market. 510(k) clearance takes less than 6 months.

deleted 2y ago

[deleted]

tromp 2y ago

> To use the system, a person wearing off-the-shelf headphones fitted with microphones taps a button while directing their head at someone talking. The sound waves from that speaker’s voice then should reach the microphones on both sides of the headset simultaneously; there’s a 16-degree margin of error. Perhaps the accuracy of identifying the correct voice could be vastly increased by adding video input. The AI can then try to match the various voices with the lip movements in the center of the video, basically lip reading.

albert_e 2y ago

I love this. I know this is just the beginning and the tech and UX will mature a lot - but being able to consciously choose what we allow into our sensory world would be a great superpower to have. In the distant future this will all be embedded inside a cochlear (neural?) implant. You can "save" known voices, prioritize them, identify various scenes/modes automatically like meetings/parties/concerts/driving/walking etc, know when to allow external sounds in (alarms, honks, someone calling your attention, etc) And with great power yada yada. I can already imagine a few ways this can be misused / abused / create non-existent challenges and problems too. But I am (cautiously) optimistic that we as human race will collectively figure out how to steer these new technology applications into net positive territory. 2040: iAudio and xSmell blamed for people losing connect with nature's sounds (like bird chirps and flowing streams) and smells (petrichor) - things that inspire us, make us creative, make life worthwile, and make us humans.

fergie 2y ago

This is an actual thing that could work: AI's ability to "stem" voices and instruments is really impressive.

Ibreezy 2y ago

briansm 2y ago

Bit annoying that they added ambient music over the demo youtube video, spoiling the one thing you want to demonstrate.

genewitch 2y ago
this is incessant and annoying, i like watching mechanics complain about cars, but most of them put a music bed or whatever behind their speech and will ask "you hear that?" no, i don't, because of your kenny G knockoff playing at low volume.

Wildgoose 2y ago

My daughter has an auditory impairment which she describes as "brain deaf". Basically, her hearing is perfect but her brain struggles to process sound in a noisy environment; she can't single out what she is listening to. This sounds perfect for her!

geros 2y ago
An audiologist described this to me as having an "audio processing disorder". Hope this helps.
follower 2y ago
Mentioned elsewhere in the comments but in case it's helpful: * https://en.wikipedia.org/wiki/Selective_auditory_attention https://en.wikipedia.org/wiki/Selective_auditory_attention * https://en.wikipedia.org/wiki/Auditory_processing_disorder https://en.wikipedia.org/wiki/Auditory_processing_disorder * https://en.wikipedia.org/wiki/Cocktail_party_effect https://en.wikipedia.org/wiki/Cocktail_party_effect

bdw5204 2y ago

In my experience, most people don't seem to understand the concept of noise cancelling headphones and will still try to talk to people who clearly can't hear them. I can't imagine it'd be any different for these AI headphones in practical use. Probably worse because the person you're actually trying to talk to might think you can't hear them.

hkwerf 2y ago
Maybe it's the opposite? ANC headphones are lacking a means of communicating to people that you're not hearing them. However, if your ANC headphones link you looking at someone with unmuting them, that communication barrier is crossed. This is particularly nice, as looking at someone is commonly a signal for "I'm listening to you".

brap 2y ago

I don't technically have a hearing problem, sometimes when there's a lot of noises occurring at the same time I hear it as one jumble.

giantg2 2y ago
I think technically that is considered a hearing problem if you're unable to separate out voices. I forget the name of it though. But this can actually disqualify you from certain types of jobs, such as the police and military.

serial_dev 2y ago

If the size could shrink to the size of a small earplug, I'd love to use this as a person who is not hearing-impaired (at least they couldn't diagnose me with it, so now I'm not sure if their diagnostics sucks, or I'm just a normal person and others pretend better that they hear everything well). In groups and with friends, it's inevitable that you end up in a busy restaurant or a bar, and it always frustrates me that I don't hear something, I ask the person to repeat only to not hear it again, usually because they repeat it at the same low level (considering the circumstances). Missing jokes and throwaway comments is even worse ("hey what are you all laughing about, I didn't hear it, could you repeat it for me like three times until I hear it").

nolongerthere 2y ago
> I'm not sure if their diagnostics sucks, or I'm just a normal person and others pretend better that they hear everything well This is one of those frustrating gaslighting things that is half true in that half the time I also pretend to hear what someone else is saying even though I couldn’t just because it’s not really important and making a big deal about it (ie asking them to repeat it at continuously louder decibels) can get awkward.
- abcdefg_ 2y ago
  So I'm not alone. I'm in my mid-forties and have experienced a significant decline over the last few years. Now I can rarely distinguish one voice in moderate background noise (conversations, music) without leaning towards them, cupping my ears. Sometimes I just have to give up and try to nod or smile at the right time as the conversation goes on around me. I recently had a test at an ENT doctor who told me my hearing is fine and insinuated I was wasting his time. The test was listening to high-pitched beeps over white noise, which isn't representative of the problem. Distinguishing one particular tone over several similar ones would be more like it.
  - glenngillen 2y ago
    Hey, there's dozens of us! :P I wrote about my experience with this last year: https://news.ycombinator.com/item?id=35897515 https://news.ycombinator.com/item?id=35897515 I did exactly the type of diagnosis you're talking about. It was quite good at how it simulated a noisy environment with a bunch of background chatter and then a single voice you were meant to listen to that would repeat various patterns of words with various combinations of lower speaking volume and/or higher background noise. One thing I wish I'd made a point of at the time was the fact that, despite being an apparently soundproof booth with headphones on, I could definitely hear people talking in the waiting room and another audiologist in an adjacent room. Though I'm not sure it would have materially changed their lack of diagnosis (they'd already detected I could hear into negative decibels). I still don't have a diagnosis, but I'm increasingly coming around to the idea that maybe it's not that my hearing is bad but that I actually hear too much. What I'd previously thought was my unability to hear people speaking on the radio in the car when everyone else clearly could wasn't because I couldn't hear the radio, it's that I can't hear it over the top of all the tyre and wind noise I'm also hearing and trying to process out. I don't think the other passengers in the car hear the rest of the noise, they only hear the radio. I bought various types of Loop earplugs and have found them fantastic for live music events. I can now hear my friends when they're talking to me! Unfortunately they greatly amplify my perception of the volume of my own voice when I talk which has the undesirable side-effect of making me talk even quieter so I feel like I'm having to yell when I want to talk to people. I've also not found them as useful as I'd hoped in restaurant-type settings.

jaustin 2y ago

Next can we have them identify ambient noises that need amplification for safety reasons, like the nearly-silent electric car about to run me over, or the bike I'm about to accidentally step in front of? As someone who spends a bit too much of my time walking around on calls, I think selective amplification of ambient sounds for safety would be amazing!

bartread 2y ago
Are you vision impaired?

cushychicken 2y ago

I used to work at Sonos, long before their current app update debacle and headphone debut. During the first aborted product effort to develop headphones, we were looking at a conceptual feature similar to this - selectively allowing people’s voices through the ANC chipset. I don’t recall the exact approach the DSP folks were using (I was closer to the hardware for ANC) but they were really only able to figure out how to isolate the wearer’s voice by virtue of that signal having more power than all the others. This is terribly cool. I wonder what other kinds of fun you could have with headphones. ANC chipsets are incredibly powerful and I’d wager their capabilities are not even close to fully tapped.

fennecfoxy 2y ago
I wouldn't ever want Sonos hearing aids. Universally Sonos units have basic functionality problems such as not reconnecting to Wifi that has gone down and come back up, especially if the Wifi has changed channel during that time. The "technical" solution is to pull the plug and reboot it (which you can't even do remotely, even if it's connected to Wifi and you want to reboot because Spotify connect on Sonos can be buggy as hell). I can keep a wifi connection up myself and always reconnect using an esp or similar TI etc module...is it so hard for the Sonos firmware devs to do something so basic?
- richwater 2y ago
  Spotify connect on most devices is buggy as hell.
ttpphd 2y ago
If you read the paper, nothing has changed. They still depend on the target talker not having competing co located sounds or voice.
- bob_theslob646 2y ago
  May you elaborate on what you mean by 'colocated sounds or voice '? Does that mean sound coming in the same direction as the person they are targeting? Does it mean sound near the person /object they are targeting?

nox101 2y ago

Sounds like a great way to spy on all people and extract all conversations. I can't wait for judges to declare that all conversation at your office must be recorded like some of them have for chat. This tech is a step to enable such a thing.

adwi 2y ago

The “cocktail party effect” externalized. Extremely cool. https://en.m.wikipedia.org/wiki/Cocktail_party_effect https://en.m.wikipedia.org/wiki/Cocktail_party_effect

mateus1 2y ago
Very interesting, concept. I have a friend with seeming no hearing issues that has to cup his ears when in louder environments such as bars, restaurants, etc. I’m wondering if anyone here has a solution to that.

btbuildem 2y ago

When I was a youngling, I dreamed of having headphones with the opposite power -- muting specific people. For me, it's not the hubbub of a crowd that's distracting, it's usually one or two offending specimens - like in the video example, the inconsiderate vermin using a speakerphone in public. I wonder if the problem maps easily from "select this source" to "select everything but that source"

causi 2y ago

I would like AI headphones that let me pinpoint the source of noises, such as inside a car engine.

classified 2y ago

Is it April 1st already?

nutanc 2y ago

What about the privacy concerns? So basically I can just look at a couple of people talking and eavesdrop?

anonu 2y ago
How does this technology change anything? Don't eavesdrop...

anonu 2y ago

By looking at them AND pressing a button. But they might be able to get rid of the button with some sensors and AI.

entico 2y ago

cant wait to see people using earphones at parties

hwbunny 2y ago

[dead]

deleted 2y ago

[deleted]

osjp1988 2y ago

[dead]

VikingCoder 2y ago

Fry_Shut_Up_And_Take_My_Money.gif

oakpond 2y ago

Honestly AI speech recognition still sucks so bad I'm basically convinced it will fall on its face in many daily use cases. I realize this is slightly tangential, but please don't replace customer support with chatbots or whatever you want to call them. It's a freaking horrible experience.

resource_waste 2y ago

>A University of Washington team Oh, so it barely works and its a proof of concept. What is the interesting thing here? We all know how sound waves work. Pretty sure this technology is old. Until there is a product here, it just sounds like you are rehashing noise cancellation. Academia has dug this grave of skepticism. I just have 0 faith this will get to market through University of Washington. Maybe it will be patented and used even less!

m3kw9 2y ago

This is the equivalent of staring but for ears.

helsinkiandrew 2y ago

Presumably this could be used to block out specific voices/sounds. There's an episode of the sci-fi show Black Mirror (White Christmas), where a person is convicted of some hideous crime and permanently blocked/made invisible and inaudible from everyone (the entire population has embedded audio/video processing enhancements by then). You can imagine future headphones where you could block out the guy in your office with the annoying laugh our download 'blocks' from the headphone appstore - no more Rick Astley or the politician you don't like etc.

ck_one 2y ago

Pretty cool what they are working on. However, I wished there would be more funding for restoring hair cells which are the root cause for most people with hearing loss. Researchers are getting closer. Dr. Chen from Harvard was able to regenerate hair cells in mature mice last year. The problem is also becoming more widespread. 30 Mio people in the US and 400 Mio people worldwide have disabling hearing loss. Regenerating hair cells and the synapses around them would also cure Tinnitus. 30 Mio x $5k for a treatment = $60B market (probably way bigger with aging population) I think we probably need more rich tech billionaires to get affected to attract large funding. What billionaires that you know are affected besides: - Brad Jacobs - Ryan from Flexport/Founders Fund

darig 2y ago
[dead]

MisterBastahrd 2y ago

While not exactly the same, I came across an app called Tunity the other day. It allows you to use your phone camera to catch the live audio feed of the television that you are attempting to watch, whether the audio is muted or if it's in a loud, crowded location, like a sports bar or airport. I haven't used it, but it's an interesting concept.

andy 2y ago

I opened an issue with this. Maybe someone here knows. I see a Python script I can run on my computer, I haven't tried it yet, but I think I could connect a microphone and process real-time audio and output it in real time, but I don’t know how to detect the user looking at someone. Could you tell me how that works?

idunnoman1222 2y ago
To use the system, a person wearing off-the-shelf headphones fitted with microphones taps a button while directing their head at someone talking. The sound waves from that speaker’s voice then should reach the microphones on both sides of the headset simultaneously; I guess by reading the article?

hpen 2y ago

Does this use Computer Vision camera or how does it work?

OkGoDoIt 2y ago

The open source code is at https://github.com/vb000/LookOnceToHear https://github.com/vb000/LookOnceToHear and the research paper is at https://arxiv.org/abs/2405.06289 https://arxiv.org/abs/2405.06289 So perhaps this is not as out of reach as many pop-science articles. I’d love to hear if anyone is able to get this working independently.

glial 2y ago

This is pretty amazing -- and a practical application of a solution to a notoriously tricky problem called the "cocktail party problem."[1] For a small subset of researchers, writing an algorithm to isolate a voice in a crowd is on par with e.g. writing an AI to play Go. [1] https://en.wikipedia.org/wiki/Cocktail_party_effect https://en.wikipedia.org/wiki/Cocktail_party_effect

jmugan 2y ago

I want to filter out all non-nature sounds. I dream of walking through the airport or the park in peace. AI seems the way to go with that since you have to predict the sound to counteract it. Good to see we are finally making progress.

chrisknyfe 2y ago

Before getting all excited that your ML model runs on your brand new 2024 macbook, before you run off to create earbuds / hearing aids with it, please try to run it on-target and see whether your model runs within your runtime budget / power budget / device size budget / battery life budget. And make sure if you're going to do bluetooth + wireless, remember that both bluetooth and wifi transmit on 2.4 GHz, and need to coordinate in order to coexist in the same IoT device. There are interconnects and wire protocols to connect the bluetooth and wifi chips together - or, preferably, you buy a chip that does both.

deleted 2y ago

[deleted]

slaymaker1907 2y ago

Oh my god, these would be absolutely amazing as someone with auditory processing issues.

darkclouds 2y ago

[dead]

wuziyue1214 2y ago

I think this is an awesome invention

masfoobar 2y ago

The concept of headphones in the last 60 years, with the exception of sound quality, comfort, and eventually being cordless.... they have not changed much in terms of style and appearance. However, I think in the next 50 years, headphones will disappear or.. should I say evolve as part of the human anatomy. Same thing for screen monitors, mouse/keyboard, smartphones, etc. Think about it. The way things are going, along with "AI" (sure buzzword in a number of ways but something that will change our way of living) many of things we use will be replaced and, likely, be simple extensions or, dare I say, be implanted. Hard drives will be a thing of the past. Everything will be (as we call today) "cloud-based" and we will be more cybernetic than we think. Of course, someone today will fear such an idea. As we slowly accept the little changes.. in 50 years we will look back and think "how did they cope without it"... a bit like how someone today look back and think "how did they cope without the internet" Many fear what they dont understand. It is the unknown. AI is a fear factor for many. For me, I accept it for what it is and the changes it will impact our lives and our careers. All I will say is -- strap yourselves in.. it will be a bumpy ride. I hope we make it through without destroying ourselves. Once we past it, the world could (finally) be at peace and to quote a famous TV show --- "to bondly go where no man has gone before!"