19 ms·

The way this works (and I'm obviously taking a high level view here) is by comparing what is being played to what is being captured. There is an inherent latenc

The way this works (and I'm obviously taking a high level view here) is by comparing what is being played to what is being captured. There is an inherent latency in between what is called the capture stream (the mic) and the reverse stream (what is being output to the speakers, be it people taking or music or whatever), and by finding this latency and comparing, one can cancel the music from the speech captured. Within a single process, or tree of processes that can cooperate, this is straightforward (modulo the actual audio signal processing which isn't) to do: keep what you're playing for a few hundreds milliseconds around, compare to what you're getting in the microphone, find correlations, cancel. If the process aren't related there are multiple ways to do this. Either the OS provides a capture API that does the cancellation, this is what happens e.g. on macOS for Firefox and Safari, you can use this. The OS knows what is being output. This is often available on mobile as well. Sometimes (Linux desktop, Windows) the OS provides a loopback stream: a way to capture the audio that is being played back, and that can similarly be used for cancellation. If none of this is available, you mix the audio output and perform cancellation yourself, and the behaviour your observe happens. Source: I do that, but at Mozilla and we unsurprisingly have the same problems and solutions.

gpvos 2y ago
It just seems more logical for the OS to do that, rather than the application. Basically every application that uses microphone input will want to do this, and will want to compensate for all audio output of the device, not just its own. Why does the OS not provide a way to do this?
- hpen 2y ago
  How sure are you that Basically every application wants this? So should there be a flag at the os level for enabling the cancellation? How do you control that flag?
  - gregschlom 2y ago
    It would be trivial to pass that flag in whatever API the application calls to request access to the microphone stream.
  - kalleboo 2y ago
    > How do you control that flag? https://developer.apple.com/documentation/avfaudio/avaudiosession/mode https://developer.apple.com/documentation/avfaudio/avaudiose...
- duped 2y ago
  > Basically every application that uses microphone input will want to do this The OS doesn't have more information about this than applications and it's not that obvious whether an application wants the OS to fuck around with the audio input it sees. Even in the applications where this might be the obvious default behavior, you're wrong - since most listeners don't use loudspeakers at all, and this is not a problem when they wear headphones. And detecting that (also, is the input a microphone at all?) is not straightforward. Not all audio applications are phone calls.
  - dheera 2y ago
    The OS can have multiple sound input devices for the application to choose from, "raw" and "fuckarounded with"
    - duped 2y ago
      That doesn't make sense in the context of default devices. MacOS's AVKit (or is it CoreAudio?) APIs that configure the streams created on the device makes way more sense, since it's a property of the audio i/o stream and not the devices.
  - deleted 2y ago
    [deleted]
  - fsckboy 2y ago
    >The OS doesn't have more information about this than applications the OP pointed out that this only works if he uses a browser monoculture the OS does have more information than that, it can know what is being played by any/all apps, and what is being picked up by the mic
    - duped 2y ago
      The "OS" isn't special here, apps can listen to system audio. fwiw, you only need to know anything about outputs if you are doing AEC. Blind source separation doesn't have that problem and can just process the input stream.
- bryanrasmussen 2y ago
  I suppose the OS probably makes something like this available, when using Voiceover on Mac and presenting in teams by default only the mic comes into teams, you need to do something to share the other processes audio. That's mac of course but in my experience Windows is much more trusting of what it gives applications access to so I suppose the same thing is available there.
- dilap 2y ago
  On mac/iOS, you get this using the AVAudioEngine API if you set voiceProcessingEnabled to true on the input node. It corrects for audio being played from all applications on the device.
  - vundercind 2y ago
    My first thought in reading the question was “if your browser is doing that, your platform architecture has… some room for improvement”.
    - naniwaduni 2y ago
      Having room for nontrivial improvement is, to be fair, a normal state of affairs for platforms.
  - xp84 2y ago
    This has certainly made conference calls significantly more usable. I feel like it must have come around during 2020, because I feel like pre-covid I would go around BEGGING everyone I did calls with to get a headset, because otherwise everyone else's voice would echo back through their microphone 0.75s later. Today I recently realized I could just literally do calls out loud on my laptop mic and speaker and somehow it works. Nice to know why!
- IvyMike 2y ago
  Did you just invent yet another linux audio stack?
- asveikau 2y ago
  The OS doesn't know that the application doesn't want feedback from the speaker, and not 100% of applications will want such filtering. I think a best practice from the OS side would be to provide it as an optional flag. (Default could be on or off, with reasonable possibility for debate in either direction, but an app that really knows what it wants should be able to ask for it.)
- swatcoder 2y ago
  > Why does the OS not provide a way to do this? Some do. But you need to have a strong-handed OS team that's willing to push everybody towards their most modern and highly integrated interfaces and sunset their older interfaces. Not everybody wants that in their OS. Some want operating systems that can be pieced together from myriad components maintained by radically different teams, some want to see their API's/interfaces preserved for decades of backwards compatibility, some want minimal features from their OS and maximum raw flexibility in user space, etc
  - dfc 2y ago
    > Some do Which Operating systems do this?
    - swatcoder 2y ago
      As others have noted, this is trivial for most macOS and iOS apps to opt in to. Frankly, I imagine its also available at the system level on Windows (and maybe Android and Linux) but probably only among applications that happen to be using certain audio frameworks/engines.
      - beepbooptheory 2y ago
        It doesn't seem to me that module-echo-cancel in Pulseaudio completely meets the requirements here (only one source), but it looks close, and seems in general like where you would implement something like this. 1. https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Modules/#module-echo-cancel https://www.freedesktop.org/wiki/Software/PulseAudio/Documen...
        hayley-patton 2y ago
        I think module-null-sink and module-loopback could be used to create a virtual source which combines multiple sources, though the source/sink thing makes my head spin. Or, more simply, I suppose using the loopback of whatever audio output device does the combination (and the same mixing) for you, if you play all audio through one output device (which is most likely)?
- kbolino 2y ago
  This assumes there is an OS-managed software mixer sitting in the middle of all audio streams between programs and devices. Historically, that wasn't the case, because it would introduce a lot of latency and jitter in the audio. I believe it is still possible for a program to get exclusive access to an audio output device on Windows (WASAPI) and Linux (ALSA).
  - dahauns 2y ago
    Historically, true, but nowadays it's pretty much standard for all the big OS. Being able to get exclusive access/bypass the system via certain means (ASIO would be another) doesn't make it go away.
- rlpb 2y ago
  There is a third place: a common library that all the apps use. If it is in the OS then it becomes brittle. If there's an improvement in the technology which requires an API change, that becomes difficult without keeping backwards compatibility or the previous implementation forever. Instead, there would be a newer generation common library which might eventually replace the first but only if the entire ecosystem chooses to leave the old one behind. Meanwhile there'd be a place for both. Apps that share use of a library would simply dynamically link to it. This is the way things usually work in the Free Software world. For example: need JPEG support? You'll probably end up linking to libjpeg or an equivalent. Most languages have a binding to the same library. Is that part of the OS? I guess the answer depends on how you define OS. On a Free Software platform it's difficult to say when a given library is part of the OS and when it is not.
  - kalleboo 2y ago
    > If it is in the OS then it becomes brittle My experience is the opposite. When it's part of the OS, it's stable and you just say "you need OS version X or better" and it will just work. When it's a library, you eventually end up in dependency hell of deprecated libraries and differing versions (or worst case, the JavaScript ecosystem when the platform provides almost nothing and you get npm).
    - rlpb 2y ago
      Depends on the OS I guess. When it's established enough, all distributions carry a high enough version that it's not an issue. If it's not established enough, I'd argue that it isn't ready to be part of an "OS" anyway (regardless of the definition of that word).
generalizations 2y ago
Is there any way to apply this outside the browser? Like, is there a version of this that can be used with Pulseaudio?
- correct-horse 2y ago
  To spare others from googling: https://docs.pipewire.org/page_module_echo_cancel.html https://docs.pipewire.org/page_module_echo_cancel.html https://wiki.archlinux.org/title/PipeWire/Examples#Echo_cancellation https://wiki.archlinux.org/title/PipeWire/Examples#Echo_canc... If you're still on pulseaudio for some reason, it ships with a similar module named "module-echo-cancel": https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Modules/ https://www.freedesktop.org/wiki/Software/PulseAudio/Documen...
  - generalizations 2y ago
    Huh, thanks. I was interested in this probably 6-8 years ago, and when I went digging the stackoverflow answer mentioned elsewhere in this thread [0] was as far as I got. I guess the tech has progressed since then. [0] https://stackoverflow.com/questions/21795944/remove-known-audio-output-from-microphone-input https://stackoverflow.com/questions/21795944/remove-known-au...
    - varjag 2y ago
      It was there 8 years ago.
Johnie 2y ago
This reminds me of: >The missile knows where it is at all times. It knows this because it knows where it isn't. By subtracting where it is from where it isn't, or where it isn't from where it is (whichever is greater), it obtains a difference, or deviation https://knowyourmeme.com/memes/the-missile-knows-where-it-is https://knowyourmeme.com/memes/the-missile-knows-where-it-is
- enva2712 2y ago
  https://www.youtube.com/watch?v=c8jWx2qmBWI https://www.youtube.com/watch?v=c8jWx2qmBWI
  - TaylorAlexander 2y ago
    the missile is eepy https://youtu.be/Csp_OABIsBM https://youtu.be/Csp_OABIsBM
- Sakos 2y ago
  This is almost weirdly philosophical. I've been thinking about this all morning.
- CasperH2O 2y ago
  Up to a point that text makes a lot of sense for describing a PID controller, which is a form of control that only really looks at error and tries to get it to zero.
  - constantcrying 2y ago
    >a PID controller, which is a form of control that only really looks at error As the name implies the PID controller relies on proportional, integral and derivative information about the error. What you mean is a purely P controller, which just relies on the error. Missiles are also not guided by a PID controller, that would be silly. They (or the guidance computer in the airplane) has to take into account the trajectory of the target and guide the missile in a way to intercept that target, which is not something you can accomplish with just a PID controller.
wormius 2y ago
For a little more context on negative feedback to those who want to know more (I believe this is what you're referring to?) Here's a short historical interview with Harold Black from AT&T on his discovery/invention of the negative feedback technique for noise reduction. It's not super explanatory but a nice historical context: https://youtu.be/iFrxyJAtJ7U?si=8ONC8N2KZwq3Jfsq https://youtu.be/iFrxyJAtJ7U?si=8ONC8N2KZwq3Jfsq Here's a more indepth circuit explanation: https://youtu.be/iFrxyJAtJ7U?si=8ONC8N2KZwq3Jfsq https://youtu.be/iFrxyJAtJ7U?si=8ONC8N2KZwq3Jfsq IIRC the issue was AT&T was trying to get cross-country calling, but to make the signal carry further you needed a louder signal. Amplifying the signal also the distortion. So Harold came up with this method that ultimately allowed enough signal reduction to allow calls to cross the country within the power constraints available. For some reason I recall something about transmission about Denver being a cut off point before the signal was too degraded... But I'm too old and forgetful so I could be misremembering something I read a while ago. If anyone has more specific info/context/citations that'd be great. Since this is just "hearsay" from memory, but I think it's something like this.
Log_out_ 2y ago
At the lowest level its a fouriertransform over a systems (your room the echochambers response is know from some testsound )and the expected output going through that transform on its way to the mic is subtracted. Most socks and machines have dedicated systems for that. The very same chip produces the echo of the surroundings.

gpvos 2y ago

It just seems more logical for the OS to do that, rather than the application. Basically every application that uses microphone input will want to do this, and will want to compensate for all audio output of the device, not just its own. Why does the OS not provide a way to do this?

hpen 2y ago
How sure are you that Basically every application wants this? So should there be a flag at the os level for enabling the cancellation? How do you control that flag?
- gregschlom 2y ago
  It would be trivial to pass that flag in whatever API the application calls to request access to the microphone stream.
- kalleboo 2y ago
  > How do you control that flag? https://developer.apple.com/documentation/avfaudio/avaudiosession/mode https://developer.apple.com/documentation/avfaudio/avaudiose...
duped 2y ago
> Basically every application that uses microphone input will want to do this The OS doesn't have more information about this than applications and it's not that obvious whether an application wants the OS to fuck around with the audio input it sees. Even in the applications where this might be the obvious default behavior, you're wrong - since most listeners don't use loudspeakers at all, and this is not a problem when they wear headphones. And detecting that (also, is the input a microphone at all?) is not straightforward. Not all audio applications are phone calls.
- dheera 2y ago
  The OS can have multiple sound input devices for the application to choose from, "raw" and "fuckarounded with"
  - duped 2y ago
    That doesn't make sense in the context of default devices. MacOS's AVKit (or is it CoreAudio?) APIs that configure the streams created on the device makes way more sense, since it's a property of the audio i/o stream and not the devices.
- deleted 2y ago
  [deleted]
- fsckboy 2y ago
  >The OS doesn't have more information about this than applications the OP pointed out that this only works if he uses a browser monoculture the OS does have more information than that, it can know what is being played by any/all apps, and what is being picked up by the mic
  - duped 2y ago
    The "OS" isn't special here, apps can listen to system audio. fwiw, you only need to know anything about outputs if you are doing AEC. Blind source separation doesn't have that problem and can just process the input stream.
bryanrasmussen 2y ago
I suppose the OS probably makes something like this available, when using Voiceover on Mac and presenting in teams by default only the mic comes into teams, you need to do something to share the other processes audio. That's mac of course but in my experience Windows is much more trusting of what it gives applications access to so I suppose the same thing is available there.
dilap 2y ago
On mac/iOS, you get this using the AVAudioEngine API if you set voiceProcessingEnabled to true on the input node. It corrects for audio being played from all applications on the device.
- vundercind 2y ago
  My first thought in reading the question was “if your browser is doing that, your platform architecture has… some room for improvement”.
  - naniwaduni 2y ago
    Having room for nontrivial improvement is, to be fair, a normal state of affairs for platforms.
- xp84 2y ago
  This has certainly made conference calls significantly more usable. I feel like it must have come around during 2020, because I feel like pre-covid I would go around BEGGING everyone I did calls with to get a headset, because otherwise everyone else's voice would echo back through their microphone 0.75s later. Today I recently realized I could just literally do calls out loud on my laptop mic and speaker and somehow it works. Nice to know why!
IvyMike 2y ago
Did you just invent yet another linux audio stack?
asveikau 2y ago
The OS doesn't know that the application doesn't want feedback from the speaker, and not 100% of applications will want such filtering. I think a best practice from the OS side would be to provide it as an optional flag. (Default could be on or off, with reasonable possibility for debate in either direction, but an app that really knows what it wants should be able to ask for it.)
swatcoder 2y ago
> Why does the OS not provide a way to do this? Some do. But you need to have a strong-handed OS team that's willing to push everybody towards their most modern and highly integrated interfaces and sunset their older interfaces. Not everybody wants that in their OS. Some want operating systems that can be pieced together from myriad components maintained by radically different teams, some want to see their API's/interfaces preserved for decades of backwards compatibility, some want minimal features from their OS and maximum raw flexibility in user space, etc
- dfc 2y ago
  > Some do Which Operating systems do this?
  - swatcoder 2y ago
    As others have noted, this is trivial for most macOS and iOS apps to opt in to. Frankly, I imagine its also available at the system level on Windows (and maybe Android and Linux) but probably only among applications that happen to be using certain audio frameworks/engines.
    - beepbooptheory 2y ago
      It doesn't seem to me that module-echo-cancel in Pulseaudio completely meets the requirements here (only one source), but it looks close, and seems in general like where you would implement something like this. 1. https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Modules/#module-echo-cancel https://www.freedesktop.org/wiki/Software/PulseAudio/Documen...
      - hayley-patton 2y ago
        I think module-null-sink and module-loopback could be used to create a virtual source which combines multiple sources, though the source/sink thing makes my head spin. Or, more simply, I suppose using the loopback of whatever audio output device does the combination (and the same mixing) for you, if you play all audio through one output device (which is most likely)?
kbolino 2y ago
This assumes there is an OS-managed software mixer sitting in the middle of all audio streams between programs and devices. Historically, that wasn't the case, because it would introduce a lot of latency and jitter in the audio. I believe it is still possible for a program to get exclusive access to an audio output device on Windows (WASAPI) and Linux (ALSA).
- dahauns 2y ago
  Historically, true, but nowadays it's pretty much standard for all the big OS. Being able to get exclusive access/bypass the system via certain means (ASIO would be another) doesn't make it go away.
rlpb 2y ago
There is a third place: a common library that all the apps use. If it is in the OS then it becomes brittle. If there's an improvement in the technology which requires an API change, that becomes difficult without keeping backwards compatibility or the previous implementation forever. Instead, there would be a newer generation common library which might eventually replace the first but only if the entire ecosystem chooses to leave the old one behind. Meanwhile there'd be a place for both. Apps that share use of a library would simply dynamically link to it. This is the way things usually work in the Free Software world. For example: need JPEG support? You'll probably end up linking to libjpeg or an equivalent. Most languages have a binding to the same library. Is that part of the OS? I guess the answer depends on how you define OS. On a Free Software platform it's difficult to say when a given library is part of the OS and when it is not.
- kalleboo 2y ago
  > If it is in the OS then it becomes brittle My experience is the opposite. When it's part of the OS, it's stable and you just say "you need OS version X or better" and it will just work. When it's a library, you eventually end up in dependency hell of deprecated libraries and differing versions (or worst case, the JavaScript ecosystem when the platform provides almost nothing and you get npm).
  - rlpb 2y ago
    Depends on the OS I guess. When it's established enough, all distributions carry a high enough version that it's not an issue. If it's not established enough, I'd argue that it isn't ready to be part of an "OS" anyway (regardless of the definition of that word).

generalizations 2y ago

Is there any way to apply this outside the browser? Like, is there a version of this that can be used with Pulseaudio?

correct-horse 2y ago
To spare others from googling: https://docs.pipewire.org/page_module_echo_cancel.html https://docs.pipewire.org/page_module_echo_cancel.html https://wiki.archlinux.org/title/PipeWire/Examples#Echo_cancellation https://wiki.archlinux.org/title/PipeWire/Examples#Echo_canc... If you're still on pulseaudio for some reason, it ships with a similar module named "module-echo-cancel": https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Modules/ https://www.freedesktop.org/wiki/Software/PulseAudio/Documen...
- generalizations 2y ago
  Huh, thanks. I was interested in this probably 6-8 years ago, and when I went digging the stackoverflow answer mentioned elsewhere in this thread [0] was as far as I got. I guess the tech has progressed since then. [0] https://stackoverflow.com/questions/21795944/remove-known-audio-output-from-microphone-input https://stackoverflow.com/questions/21795944/remove-known-au...
  - varjag 2y ago
    It was there 8 years ago.

Johnie 2y ago

This reminds me of: >The missile knows where it is at all times. It knows this because it knows where it isn't. By subtracting where it is from where it isn't, or where it isn't from where it is (whichever is greater), it obtains a difference, or deviation https://knowyourmeme.com/memes/the-missile-knows-where-it-is https://knowyourmeme.com/memes/the-missile-knows-where-it-is

enva2712 2y ago
https://www.youtube.com/watch?v=c8jWx2qmBWI https://www.youtube.com/watch?v=c8jWx2qmBWI
- TaylorAlexander 2y ago
  the missile is eepy https://youtu.be/Csp_OABIsBM https://youtu.be/Csp_OABIsBM
Sakos 2y ago
This is almost weirdly philosophical. I've been thinking about this all morning.
CasperH2O 2y ago
Up to a point that text makes a lot of sense for describing a PID controller, which is a form of control that only really looks at error and tries to get it to zero.
- constantcrying 2y ago
  >a PID controller, which is a form of control that only really looks at error As the name implies the PID controller relies on proportional, integral and derivative information about the error. What you mean is a purely P controller, which just relies on the error. Missiles are also not guided by a PID controller, that would be silly. They (or the guidance computer in the airplane) has to take into account the trajectory of the target and guide the missile in a way to intercept that target, which is not something you can accomplish with just a PID controller.

wormius 2y ago

For a little more context on negative feedback to those who want to know more (I believe this is what you're referring to?) Here's a short historical interview with Harold Black from AT&T on his discovery/invention of the negative feedback technique for noise reduction. It's not super explanatory but a nice historical context: https://youtu.be/iFrxyJAtJ7U?si=8ONC8N2KZwq3Jfsq https://youtu.be/iFrxyJAtJ7U?si=8ONC8N2KZwq3Jfsq Here's a more indepth circuit explanation: https://youtu.be/iFrxyJAtJ7U?si=8ONC8N2KZwq3Jfsq https://youtu.be/iFrxyJAtJ7U?si=8ONC8N2KZwq3Jfsq IIRC the issue was AT&T was trying to get cross-country calling, but to make the signal carry further you needed a louder signal. Amplifying the signal also the distortion. So Harold came up with this method that ultimately allowed enough signal reduction to allow calls to cross the country within the power constraints available. For some reason I recall something about transmission about Denver being a cut off point before the signal was too degraded... But I'm too old and forgetful so I could be misremembering something I read a while ago. If anyone has more specific info/context/citations that'd be great. Since this is just "hearsay" from memory, but I think it's something like this.

Log_out_ 2y ago

At the lowest level its a fouriertransform over a systems (your room the echochambers response is know from some testsound )and the expected output going through that transform on its way to the mic is subtracted. Most socks and machines have dedicated systems for that. The very same chip produces the echo of the surroundings.