4 ms·

The competitive gap with Discord in terms of media quality is probably something like: * Need a low-latency SFU. This should be very doable; not only are ther

The competitive gap with Discord in terms of media quality is probably something like: * Need a low-latency SFU. This should be very doable; not only are there a lot of good FOSS SFUs to build on top of these days, the history of the Matrix team is actually that we built VoIP stacks fulltime before we shifted focus to Matrix, and we've built MCUs and media servers of all flavours in the past. MSC3401 should also give us a competitive edge given latency will be automagically minimised by using the physically closest decentralised SFU, thus letting anyone bring their SFU to the party. * Needs a SFU with good rate control (and/or FEC). This is probably the single most important thing to get right in terms of quality. Signal wrote up a good overview of why: https://signal.org/blog/how-to-build-encrypted-group-calls/ https://signal.org/blog/how-to-build-encrypted-group-calls/ * Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc). Ideally you need something like https://krisp.ai/ https://krisp.ai/ or https://workspaceupdates.googleblog.com/2021/06/background-noise-reduction-cancellation-update.html https://workspaceupdates.googleblog.com/2021/06/background-n... in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard. However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (https://github.com/matrix-org/seshat https://github.com/matrix-org/seshat), we'll have a go at doing something similar for this problem too. * Excellent automatic gain control. The importance of normalising/compressing everyone's audio so they're equivalent loudness is really important. We're also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds. We can probably also be more bullish about supporting new audio codecs like Lyra. EDIT: oh, the other obvious thing is echo cancellation - but we're currently at the mercy of the browser's WebRTC stack for that. However, we could ship a tweaked WebRTC in Element Desktop (or other native Matrix clients) in future to do better than plain old WebRTC.

spockz 5y ago
Nice elaborate answer. Discord has noise cancellation by krisp. I believe it is offline only. That would be good enough I suppose. But I’m not sure whether it can be loaded in webasm so it works in the browser. (To meet the cross platform needs of Matrix/Element.)
jfkimmes 5y ago
Very insightful! I'll have a look at the links. Thanks for answering.
anticensor 5y ago
Discord voice is actually a server muxing MCU emulating an SFU.
- 7sidedmarble 5y ago
  Do you have any details on that? From what I've seen they just say it's a normal SFU architecture.
dopa42365 5y ago
Something wrong with simply using RNNoise?
- Arathorn 5y ago
  mmm, https://github.com/jitsi/rnnoise-wasm https://github.com/jitsi/rnnoise-wasm looks like it might help. wonder if jitsi actually uses it.
  - Cu3PO42 5y ago
    The repository is under the Jitsi organization, so I'd say chances of them using it or at least planning to use it are quite high.
  - saghul 5y ago
    We are currently using it for the “your microphone is noisy” notification feature, not for actual noise detection. With Audio Worklets or insertable streams, however, it should be possible to apply it to the stream im real time, though we haven’t tried it yet.
  - namibj 5y ago
    You can try in Mumble.
throwaway29879 5y ago
Can you at least let us know what "stacks" anyone involved has produced? last I checked nobody involved has ever been involved in real voice platforms (as evidenced by 8 years of "voice first" but actually producing very little actual voice integration)
- Arathorn 5y ago
  Sure. The entirety of the original Matrix team used to be a team called “next gen telephony” inside a startup called MX Telecom, which then got acquired by Amdocs and turned into their Unified Communications division, which I ran. The product was a horizontally scalable SIP softswitch built on resiprocate (B2BUA SBC and stateless router components), and then with media processing done using a C++ media graph framework we created called mxmedia. This was similar to gstreamer, but with more flexible threading and using C++ typing extensively to model media formats and propagate them through the media graph. It ended up with roughly the same featureset as today’s gstreamer, but about 10 years earlier - so with muxers, demuxers, packetizers, codecs, resamplers, colourspace transforms, AEC, AGC, rate control, SRTP, ICE, etc. We used it serverside for media gateways for transcoding calls for ISDN, H.323 and IAX; we then ported it clientside when smartphones came around to use it as a webrtc-style softphone SDK. Unfortunately the IP was all proprietary and owned by the company, and when WebRTC stole our lunch we shifted focus to messaging and building Matrix, and have slowly returned focus to VoIP - hence Element Call.
  - throwaway29879 5y ago
    That's pretty much what I thought, which makes me wonder why at least SIP integration wasn't early on the list, there is the voip bridge but it relied on freeswitch and obviously is unmaintained so it doesn't work anymore, is there anything on the road map to make this happen?
ryukafalz 5y ago
> We're also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds. Thank you! This is something I've wanted in a videoconferencing app since I discovered it was a thing, and I'm glad it's Element doing it.
pthatcherg 5y ago
Hey, I'm the author of the Signal blog post about SFUs. I have a few questions/comments: 1. I don't think there are many good open source SFUs to choose from. I know of 2, maybe 3 (including our new one). There may be many, but few have good rate control. But maybe I just don't know about them? I'd be happy to learn of more good ones. 2. Echo cancellation is certainly a hard problem, but it doesn't conflict with E2EE unless you do it on the server, which isn't necessary. So perhaps it may be a somewhat harder problem because you close off one possible approach (doing it server-side), but many (most?) echo cancellation solutions are done client-side. 3. You may not be completely dependent on WebRTC's echo cancellation any more because of the new MediaStreamTrackProcessor and MediaStreamTrackGenerator APIs. I don't know if it will work for echo cancellation, but it might.
- Arathorn 5y ago
  1. So the SFUs we're currently looking at are yours, ion-sfu (and/or galene) and mediasoup. Honestly we haven't finished looking at how they compare for rate control, but the Pion team seems very interested in ensuring they have good rate control. 2. From context I think you're talking about noise cancellation here? I assumed that some of the more exotic ML-based ones ran serverside, which obviously is incompatible with E2EE. It sounds like there are a bunch of options for running WASM-based intelligent noise cancellation clientside though, especially with MediaStreamTrackProcessor and friends. rnnoiseless as a pure Rust->WASM port of rnnoise looks fun, for instance: https://github.com/jneem/nnnoiseless https://github.com/jneem/nnnoiseless 3. True, although given Google are highly motivated to make AEC work properly in WebRTC, I guess I'm hoping that they'll continue improving it, much as they have been. I certainly never want to have to write or integrate one ever again :D

spockz 5y ago

Nice elaborate answer. Discord has noise cancellation by krisp. I believe it is offline only. That would be good enough I suppose. But I’m not sure whether it can be loaded in webasm so it works in the browser. (To meet the cross platform needs of Matrix/Element.)

jfkimmes 5y ago

Very insightful! I'll have a look at the links. Thanks for answering.

anticensor 5y ago

Discord voice is actually a server muxing MCU emulating an SFU.

7sidedmarble 5y ago
Do you have any details on that? From what I've seen they just say it's a normal SFU architecture.

dopa42365 5y ago

Something wrong with simply using RNNoise?

Arathorn 5y ago
mmm, https://github.com/jitsi/rnnoise-wasm https://github.com/jitsi/rnnoise-wasm looks like it might help. wonder if jitsi actually uses it.
- Cu3PO42 5y ago
  The repository is under the Jitsi organization, so I'd say chances of them using it or at least planning to use it are quite high.
- saghul 5y ago
  We are currently using it for the “your microphone is noisy” notification feature, not for actual noise detection. With Audio Worklets or insertable streams, however, it should be possible to apply it to the stream im real time, though we haven’t tried it yet.
- namibj 5y ago
  You can try in Mumble.

throwaway29879 5y ago

Can you at least let us know what "stacks" anyone involved has produced? last I checked nobody involved has ever been involved in real voice platforms (as evidenced by 8 years of "voice first" but actually producing very little actual voice integration)

Arathorn 5y ago
Sure. The entirety of the original Matrix team used to be a team called “next gen telephony” inside a startup called MX Telecom, which then got acquired by Amdocs and turned into their Unified Communications division, which I ran. The product was a horizontally scalable SIP softswitch built on resiprocate (B2BUA SBC and stateless router components), and then with media processing done using a C++ media graph framework we created called mxmedia. This was similar to gstreamer, but with more flexible threading and using C++ typing extensively to model media formats and propagate them through the media graph. It ended up with roughly the same featureset as today’s gstreamer, but about 10 years earlier - so with muxers, demuxers, packetizers, codecs, resamplers, colourspace transforms, AEC, AGC, rate control, SRTP, ICE, etc. We used it serverside for media gateways for transcoding calls for ISDN, H.323 and IAX; we then ported it clientside when smartphones came around to use it as a webrtc-style softphone SDK. Unfortunately the IP was all proprietary and owned by the company, and when WebRTC stole our lunch we shifted focus to messaging and building Matrix, and have slowly returned focus to VoIP - hence Element Call.
- throwaway29879 5y ago
  That's pretty much what I thought, which makes me wonder why at least SIP integration wasn't early on the list, there is the voip bridge but it relied on freeswitch and obviously is unmaintained so it doesn't work anymore, is there anything on the road map to make this happen?

ryukafalz 5y ago

> We're also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds. Thank you! This is something I've wanted in a videoconferencing app since I discovered it was a thing, and I'm glad it's Element doing it.

pthatcherg 5y ago

Hey, I'm the author of the Signal blog post about SFUs. I have a few questions/comments: 1. I don't think there are many good open source SFUs to choose from. I know of 2, maybe 3 (including our new one). There may be many, but few have good rate control. But maybe I just don't know about them? I'd be happy to learn of more good ones. 2. Echo cancellation is certainly a hard problem, but it doesn't conflict with E2EE unless you do it on the server, which isn't necessary. So perhaps it may be a somewhat harder problem because you close off one possible approach (doing it server-side), but many (most?) echo cancellation solutions are done client-side. 3. You may not be completely dependent on WebRTC's echo cancellation any more because of the new MediaStreamTrackProcessor and MediaStreamTrackGenerator APIs. I don't know if it will work for echo cancellation, but it might.

Arathorn 5y ago
1. So the SFUs we're currently looking at are yours, ion-sfu (and/or galene) and mediasoup. Honestly we haven't finished looking at how they compare for rate control, but the Pion team seems very interested in ensuring they have good rate control. 2. From context I think you're talking about noise cancellation here? I assumed that some of the more exotic ML-based ones ran serverside, which obviously is incompatible with E2EE. It sounds like there are a bunch of options for running WASM-based intelligent noise cancellation clientside though, especially with MediaStreamTrackProcessor and friends. rnnoiseless as a pure Rust->WASM port of rnnoise looks fun, for instance: https://github.com/jneem/nnnoiseless https://github.com/jneem/nnnoiseless 3. True, although given Google are highly motivated to make AEC work properly in WebRTC, I guess I'm hoping that they'll continue improving it, much as they have been. I certainly never want to have to write or integrate one ever again :D