6 ms·

Could you give some insights on how you estimate the amount of effort of making Element Call competitive, performance-wise, with say Discord. I heard that Disco

Could you give some insights on how you estimate the amount of effort of making Element Call competitive, performance-wise, with say Discord. I heard that Discord threw a lot of time and money at optimizing voice in their product. Can you just jump in and realistically compete? There are a lot of performance/latency/sound quality comparisons online of Mumble vs TeamSpeak vs Discord and recently Jitsi vs MS Teams vs Zoom, etc. I feel like this is a problem-space that can be optimized to an arbitrarily deep extent. Just two examples that come to mind are SFU performance/efficiency and noise reduction, two things where e.g Jitsi notoriously lags behind Zoom.

Arathorn 5y ago
The competitive gap with Discord in terms of media quality is probably something like: * Need a low-latency SFU. This should be very doable; not only are there a lot of good FOSS SFUs to build on top of these days, the history of the Matrix team is actually that we built VoIP stacks fulltime before we shifted focus to Matrix, and we've built MCUs and media servers of all flavours in the past. MSC3401 should also give us a competitive edge given latency will be automagically minimised by using the physically closest decentralised SFU, thus letting anyone bring their SFU to the party. * Needs a SFU with good rate control (and/or FEC). This is probably the single most important thing to get right in terms of quality. Signal wrote up a good overview of why: https://signal.org/blog/how-to-build-encrypted-group-calls/ https://signal.org/blog/how-to-build-encrypted-group-calls/ * Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc). Ideally you need something like https://krisp.ai/ https://krisp.ai/ or https://workspaceupdates.googleblog.com/2021/06/background-noise-reduction-cancellation-update.html https://workspaceupdates.googleblog.com/2021/06/background-n... in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard. However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (https://github.com/matrix-org/seshat https://github.com/matrix-org/seshat), we'll have a go at doing something similar for this problem too. * Excellent automatic gain control. The importance of normalising/compressing everyone's audio so they're equivalent loudness is really important. We're also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds. We can probably also be more bullish about supporting new audio codecs like Lyra. EDIT: oh, the other obvious thing is echo cancellation - but we're currently at the mercy of the browser's WebRTC stack for that. However, we could ship a tweaked WebRTC in Element Desktop (or other native Matrix clients) in future to do better than plain old WebRTC.
- spockz 5y ago
  Nice elaborate answer. Discord has noise cancellation by krisp. I believe it is offline only. That would be good enough I suppose. But I’m not sure whether it can be loaded in webasm so it works in the browser. (To meet the cross platform needs of Matrix/Element.)
- jfkimmes 5y ago
  Very insightful! I'll have a look at the links. Thanks for answering.
- anticensor 5y ago
  Discord voice is actually a server muxing MCU emulating an SFU.
  - 7sidedmarble 5y ago
    Do you have any details on that? From what I've seen they just say it's a normal SFU architecture.
- dopa42365 5y ago
  Something wrong with simply using RNNoise?
  - Arathorn 5y ago
    mmm, https://github.com/jitsi/rnnoise-wasm https://github.com/jitsi/rnnoise-wasm looks like it might help. wonder if jitsi actually uses it.
    - Cu3PO42 5y ago
      The repository is under the Jitsi organization, so I'd say chances of them using it or at least planning to use it are quite high.
    - saghul 5y ago
      We are currently using it for the “your microphone is noisy” notification feature, not for actual noise detection. With Audio Worklets or insertable streams, however, it should be possible to apply it to the stream im real time, though we haven’t tried it yet.

Arathorn 5y ago

The competitive gap with Discord in terms of media quality is probably something like: * Need a low-latency SFU. This should be very doable; not only are there a lot of good FOSS SFUs to build on top of these days, the history of the Matrix team is actually that we built VoIP stacks fulltime before we shifted focus to Matrix, and we've built MCUs and media servers of all flavours in the past. MSC3401 should also give us a competitive edge given latency will be automagically minimised by using the physically closest decentralised SFU, thus letting anyone bring their SFU to the party. * Needs a SFU with good rate control (and/or FEC). This is probably the single most important thing to get right in terms of quality. Signal wrote up a good overview of why: https://signal.org/blog/how-to-build-encrypted-group-calls/ https://signal.org/blog/how-to-build-encrypted-group-calls/ * Excellent noise cancellation (and background noise elimination, microphone scratch noise elimination etc). Ideally you need something like https://krisp.ai/ https://krisp.ai/ or https://workspaceupdates.googleblog.com/2021/06/background-noise-reduction-cancellation-update.html https://workspaceupdates.googleblog.com/2021/06/background-n... in the mix - but doing this in an E2EE-friendly and privacy preserving manner is Hard. However, just like we solved E2EE full text search by doing it clientside and making the indexes gossipable between your clients (https://github.com/matrix-org/seshat https://github.com/matrix-org/seshat), we'll have a go at doing something similar for this problem too. * Excellent automatic gain control. The importance of normalising/compressing everyone's audio so they're equivalent loudness is really important. We're also in the process of adding in spatial audio (unsure if Discord has that) which should help a tonne with distinguishing the different audio feeds. We can probably also be more bullish about supporting new audio codecs like Lyra. EDIT: oh, the other obvious thing is echo cancellation - but we're currently at the mercy of the browser's WebRTC stack for that. However, we could ship a tweaked WebRTC in Element Desktop (or other native Matrix clients) in future to do better than plain old WebRTC.

spockz 5y ago
Nice elaborate answer. Discord has noise cancellation by krisp. I believe it is offline only. That would be good enough I suppose. But I’m not sure whether it can be loaded in webasm so it works in the browser. (To meet the cross platform needs of Matrix/Element.)
jfkimmes 5y ago
Very insightful! I'll have a look at the links. Thanks for answering.
anticensor 5y ago
Discord voice is actually a server muxing MCU emulating an SFU.
- 7sidedmarble 5y ago
  Do you have any details on that? From what I've seen they just say it's a normal SFU architecture.
dopa42365 5y ago
Something wrong with simply using RNNoise?
- Arathorn 5y ago
  mmm, https://github.com/jitsi/rnnoise-wasm https://github.com/jitsi/rnnoise-wasm looks like it might help. wonder if jitsi actually uses it.
  - Cu3PO42 5y ago
    The repository is under the Jitsi organization, so I'd say chances of them using it or at least planning to use it are quite high.
  - saghul 5y ago
    We are currently using it for the “your microphone is noisy” notification feature, not for actual noise detection. With Audio Worklets or insertable streams, however, it should be possible to apply it to the stream im real time, though we haven’t tried it yet.