17 ms·

Replacing WebRTC: real-time latency with WebTransport and WebCodecs

301 points by kixelated 3y ago

tamimio 3y ago
I got excited for a second that a new something will replace webrtc for media/videos.. it is not. Around 2 years ago in a project I wanted to transfer a raw 4K stream with low latency (sub 50ms) over cellular network, WebRTC performed poorly it was a no go, I ended up making my own algorithm that enables FEC (Forward Error Correction) based on IETF payload scheme 20, and piped it through UDP with Gstreamer and managed to make it work, but obviously wasn’t in the browser.
- englishm 3y ago
  That sounds like it was an interesting project! Jean-Baptiste Kempf also presented something at Demuxed last week using FEC and QUIC datagrams to get similarly low latency delivery over WebTransport into a browser. There's also a draft[1] for adding FEC to QUIC itself so it's quite possible Media over QUIC (MoQ) could benefit from this approach as well. I'm not sure why you say "it is not." We have a working MoQ demo running already on https://quic.video https://quic.video that includes a Rust-based relay server and a TypeScript player. The BBB demo video is being ingested using a Rust-based CLI tool I wrote called moq-pub which takes fragmented MP4 input from ffmpeg and sends it to a MoQ relay. You can also "go live" from your own browser if you'd like to test the latency and quality that way, too. [1]: https://www.ietf.org/archive/id/draft-michel-quic-fec-01.html https://www.ietf.org/archive/id/draft-michel-quic-fec-01.htm...
  - tamimio 3y ago
    Thanks, it was a big project and streaming 4k in realtime from a flying drone was one of the challenging parts! I have some write up about it although nothing too technical but some videos there demonstrating some differences(1) > I'm not sure why you say "it is not." Pardon my ignorance it looked like it isn’t replacing WebRTC entirely yet, but glad I was wrong, I never tried anything QUIC for media related, would love to try the MoQ tool you did, and like the fact it’s rust based too as the one I did was written in rust. I will give it a test for sure, it’s been two years and I wasn’t following any updates so hopefully there’s an improvement compared what it was back then. > takes fragmented MP4 input from ffmpeg and sends it to a MoQ relay. Just a quick question, is ffmpeg a “requirement” per se for that CLI tool? As I remember I had to ditch ffmpeg in favor of gstreamer since the former one was eating up a lot of resources compared to Gstreamer, and it was crucial issue since the server was basically an SBC on a flying drone. (1) https://tamim.io/professional_projects/nerds-heavy-lift-drone-with-low-latency-command-and-control-over-cellular-5g-and-4k-video-stream-in-realtime/#4k-realtime-video-streaming https://tamim.io/professional_projects/nerds-heavy-lift-dron...
    - englishm 3y ago
      The current moq-pub implementation only requires valid fMP4 (a la CMAF) to be provided over stdin. I haven't tested, but I imagine you can do the same with gstreamer. Separately, I've been working on a wrapper library for moq-rs that I've been calling 'libmoq'. The intent there is to provide a C FFI that non-Rust code can link against. The first integration target for libmoq is ffmpeg. (I have a few bugs to work out before I clean up and shout about that code, but it does mostly work already.) I gave a presentation about some of this work last week at Demuxed, but the VoDs probably won't be available on YouTube until Decemberish. Also, I understand the gstreamer project has better support for Rust so I'll be looking at that soon, too.
      - oplav 3y ago
        I believe gstreamer has support for fMP4: https://gstreamer.freedesktop.org/documentation/fmp4/index.html https://gstreamer.freedesktop.org/documentation/fmp4/index.h... I stumbled upon a MoQ video by kixelated a couple months ago and have been meaning to give the above a try, but haven't gotten around to it yet so not sure if it will do the trick.
      - tamimio 3y ago
        Appreciated! Where do I find these presentations, if you don’t me asking?
        englishm 3y ago
        https://2023.demuxed.com/ https://2023.demuxed.com/ has the day-long uncut videos available behind a $20 "VoD-only" post-event "ticket" paywall. Eventually https://www.youtube.com/@Demuxed https://www.youtube.com/@Demuxed will have the edited VoDs for Demuxed 2023 available for free. Usually those go up around December-ish. I also posted my slides on LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:7124766662339268608/ https://www.linkedin.com/feed/update/urn:li:activity:7124766...
- Sean-Der 3y ago
  That's a cool problem! I could see how WebRTC out of the box would perform poorly. It wants to NACK + delay to give a good 'conferencing experience'. I bet with FlexFEC [0] and playout-delay [1] you would get the behavior you were looking for. The sender would have to be custom, but the receivers (browsers) should just work! If you are interested in giving it a shot again would love to help :) [0] https://datatracker.ietf.org/doc/html/rfc8627 https://datatracker.ietf.org/doc/html/rfc8627 [1] https://webrtc.googlesource.com/src/+/main/docs/native-code/rtp-hdrext/playout-delay/README.md https://webrtc.googlesource.com/src/+/main/docs/native-code/...
  - tamimio 3y ago
    It was an interesting problem indeed! I had some write up about it (and the whole project) in a link I have in the comment above, might gives more context. > I bet with FlexFEC [0] The previous draft (1) of this was my basis when I did the FEC sender. I didn’t manage to have it streamed 4K into the browser though, my client was OBS with gstreamer as it was far more performant than a browser in my tests, have you/any demo you did that I can try to stream it into the browser? That would be really major improvement! And appreciate the help, O would definitely give it another shot! (1) https://datatracker.ietf.org/doc/html/draft-ietf-payload-flexible-fec-scheme-20 https://datatracker.ietf.org/doc/html/draft-ietf-payload-fle...
    - Sean-Der 3y ago
      I would love too! Join https://pion.ly/slack https://pion.ly/slack and I am Sean-Der If you prefer discord there also is a dedicated space for ‘real-time broadcasting’ https://discord.gg/DV4ufzvJ4T https://discord.gg/DV4ufzvJ4T
      - tamimio 3y ago
        Thank you!
- imtringued 3y ago
  Well, a raw 4k stream has a bit rate of 11.9 gbps. I would be surprised if that worked over a cellular network at all.
  - vlovich123 3y ago
    At 60fps but yeah, I think they mean they still passed the raw 4k stream through a lossy codec before putting it over cellular.
  - tamimio 3y ago
    I think it was around 3gbps or even less if I remember correctly, it wasn’t 60fps and I can’t remember the color depth either, and the cellular network was a SA (Stand Alone) mmwave private network, not a commercial one, so it did work in our project eventually.
- modeless 3y ago
  Yeah. From what I can see WebRTC is basically a ready made implementation of a video calling app, bolted on the side of the browser, that you can slap some UI on top of. If you want to do anything that isn't a straight Zoom style video calling app (or Stadia, RIP), you'll immediately run into a wall of 5 year old known but unfixed bugs, or worse. To be fair, video calling is an incredibly important use case and I'm glad it can be done in the browser. I am grateful to the WebRTC team every time I can join a Zoom call without installing a bunch of extra crap on my machine. I just hope that WebRTC really can someday be replaced by composing multiple APIs that are much smaller in scope and more flexible, to allow for use cases other than essentially Zoom and Stadia. I guess I'm just repeating what the article said, but it's so right that it's worth repeating.
  - est 3y ago
    > WebRTC is basically a ready made implementation of a video calling app, bolted on the side of the browser Yeah, Google bought GlobalIPSound, dumps the source code to w3c as a standard.
DaleCurtis 3y ago
Thanks for the nice write up! I work on the WebCodecs team at Chrome. I'm glad to hear it's mostly working for you. If you (or anyone else) has specific requests for new knobs regarding "We may need more encoding options, like non-reference frames or SVC", please file issues at https://github.com/w3c/webcodecs/issues https://github.com/w3c/webcodecs/issues
- krebby 3y ago
  Encoding alpha, please! https://github.com/w3c/webcodecs/issues/672 https://github.com/w3c/webcodecs/issues/672 Thanks for the great work on WebCodecs!
- vlovich123 3y ago
  There’s a few that would be neat: * maybe possible already, but it’s not immediately clear how to change the bitrate of the encoder dynamically when doing VBR/CBR (seems like you can only do it with per-frame quantization params which isn’t very friendly) * being able to specify the reference frame to use for encoding p frames * being able to generate slices efficiently / display them easily. For example, Oculus Link encodes 1/n of the video in parallel encoders and decodes similarly. This way your encoding time only contributes 1/n frame encode/decode worth of latency because the rest is amortized with tx+decode of other slices. I suspect the biggest requirement here is to be able to cheaply and easily get N VideoFrames OR be able to cheaply split a VideoFrame into horizontal or vertical slices.
  - DaleCurtis 3y ago
    * Hmm, what kind of scheme are you thinking beyond per frame QP? Does an abstraction on top of QP work for the case you have in mind? * Reference frame control seems to be https://github.com/w3c/webcodecs/issues/285 https://github.com/w3c/webcodecs/issues/285, there's some interest in this for 2024, so I'd expect progress here. * Does splitting frames in WebGPU/WebGL work for the use case here? I'm not sure we could do anything internally (we're at the mercy of hardware decode implementations) without implementing such a shader.
fenesiistvan 3y ago
I don't get this. After the initial ICE negotiation you can just send raw RTP packets (encrypted with SRTP/DTLS). There is no need for any ACK packets. FEC can be done at codec level. What I am missing?
- kixelated 3y ago
  Are you referencing this line? > 2x the packets, because libsctp immediately ACKs every “datagram”. The section is about data channels, which uses SCTP and is ACK-based. Yes, you can use RTP with NACK and/or FEC with the media stack, but not with the data stack.
  - the8472 3y ago
    TCP can coalesce acks for multiple packets, can't SCTP do the same?
    - Sean-Der 3y ago
      SCTP can (and does) a SACK[0] isn't needed for each DATA chunk. [0] https://datatracker.ietf.org/doc/html/rfc4960#section-3.3.4 https://datatracker.ietf.org/doc/html/rfc4960#section-3.3.4
    - kixelated 3y ago
      The protocol can do it, but libsctp (used by browsers) was not coalescing ACKs. I'm not sure if it has been fixed yet.
- pthatcherg 3y ago
  From a server or a native client, you can send whatever RTP packets you want, but you cannot send whatever RTP packets you want from a web client, and you cannot have access to the RTP packets from a web client and do whatever you want with them, at least not very easily. We are working on an extension to WebRTC called RtpTransport that would allow for just that, but it's in early stages of design and standardization.
gvkhna 3y ago
Unfortunately Safari is still a major hold out on WebTransport with no clear update, considering all other browsers have now supported it in GA for 3+ years.
- englishm 3y ago
  I don't know the timeline, but Apple has committed [1] to adding WebTransport support to WebKit. [1]: https://github.com/WebKit/standards-positions/issues/18#issuecomment-1495890122 https://github.com/WebKit/standards-positions/issues/18#issu...
  - kixelated 3y ago
    Eric Kinnear (linked post; Apple) is the author of the HTTP/2 fallback for WebTransport, so it's safe to say that WebTransport will be available in WebKit at some point.
  - gvkhna 3y ago
    Great news! It looks like Google GRPC team is probably waiting for Safari Webtransport support to implement grpc over the web in WT. Even though Chrome has had it for 2 years they have kept saying they’ll wait until it becomes GA.
- gs17 3y ago
  Yeah, I have a project where WebTransport would be a huge improvement, but not being able to support Safari is a dealbreaker.
- doctorpangloss 3y ago
  Google Chrome never implemented trailers, so no gRPC. Every browser is guilty. > I spent almost two years building/optimizing a partial WebRTC stack @ Twitch using pion. Our use-case was quite custom and we ultimately scrapped it, but your millage may vary. So many words about protocols. Protocols aren't hard or interesting. QA is hard. libwebrtc has an insurmountable lead on QA. None of these ad-hoc things, nor WebRTC implementations like Pion, will ever catch up, let alone be deployed in Mobile Safari.
sansseriff 3y ago
What's a good place to learn about all the networking jargon in this article? Also, a nitpicky point: As a community it would be nice to stop thinking of 'links to wikipedia pages, mdn docs, or github issues' as useful or pertinent forms of citations or footnotes. If I'm immersed in an article and I come across a term I don't know like 'SCTP', do you think a link to some dense proposal standard webpage is appropriate in this context for providing the background info I need? Of course academia is guilty of this too. Canonical citations don't tell the reader much else beyond 'I know what I'm talking about' and 'if you want to know more spend 30 minutes reading this other article' But since we're web based here I think we can do better. Tooltips that expand into a helpful paragraph about e.g. SCTP would be a start.
- Sean-Der 3y ago
  For the WebRTC jargon check out https://webrtcforthecurious.com/ https://webrtcforthecurious.com/ If that still doesn’t cover enough I would love to hear! Always trying to make it better.
- englishm 3y ago
  Here are a couple resources which may be helpful: - https://www.mux.com/video-glossary https://www.mux.com/video-glossary - https://howvideo.works/ https://howvideo.works/ Also, if you do want to take the time to read a longer and denser article, but come away with an understanding of much of the breadth of modern streaming tech, there's a really great survey paper available in pre-print here: https://doi.org/10.48550/arXiv.2310.03256 https://doi.org/10.48550/arXiv.2310.03256
- IKantRead 3y ago
  I really like High Performance Browser Networking by Ilya Grigorik. It's published by O'Reilly but is also free online [0]. What's particularly great about it is, unlike most other networking books, it focuses on the issue from the browser/web developer perspective which is particularly helpful for WebRTC and generally applicable to daily web dev work. 0. https://hpbn.co/ https://hpbn.co/
seydor 3y ago
> Back then, the web was a very different place. Flash was the only way to do live media and it was a mess. Not sure why people are saying that. WebRTC is far harder to make it work. Peer-to-peer is a cpu black-hole-level hog
- kixelated 3y ago
  I maintained the Flash video player at Twitch until I couldn't take it any longer and created an HTML5 player. Flash was a mess. :)
  - seydor 3y ago
    is html5 video part of webrtc?
    - yalok 3y ago
      no, they are not related
mehagar 3y ago
An additional benefit of WebTransport over WebRTC DataChannels is that the WebTransport API is supported in Web Workers, meaning you can send and receive data off the main thread for better performance.
- kixelated 3y ago
  Absolutely! I'm doing that in my implementation: the main thread immediately transfers each incoming QUIC stream to a WebWorker, which then reads/decodes the container/codec and renders via OffscreenCanvas. I didn't realize that DataChannels were main thread only. That's good to know!
- deleted 3y ago
  [deleted]
znpy 3y ago
> The best and worst part about WebRTC is that it supports peer-to-peer. I hope P2P stays around in browsers.
- englishm 3y ago
  > I hope P2P stays around in browsers. I do, too. There was a W3C draft [1] ~pthatcherg was working on for P2P support for QUIC in the browser that could have maybe become a path to WebRTC using QUIC as a transport, but I think it may have been dropped somewhere along the line to the current WebTransport spec and implementations. (If Peter sees this, I'd love to learn more about how that transpired and what the current status of those ideas might be.) A more recent IETF individual draft [2] defines a way to do ICE/STUN-like address discovery using an extension to QUIC itself, so maybe that discussion indicates some revived interest in P2P use cases. [1]: https://w3c.github.io/p2p-webtransport/ https://w3c.github.io/p2p-webtransport/ [2]: https://datatracker.ietf.org/doc/html/draft-seemann-quic-address-discovery-00 https://datatracker.ietf.org/doc/html/draft-seemann-quic-add...
Fischgericht 3y ago
A wonderful write-up, thank you very much. However, it needs to be said that with currently available technologies, there is no need to have a 5 seconds buffer. Have a look at the amazing work of the OvenMedia team (not affiliated). Using their stack, and using Low-Latency HLS (LLHLS) I have been able to easily reach an end-to-end latency between the camera on site to the end-user viewing it in the browser of <500ms. At 20 MBit/s, and using either SRT or RTMP for stream upload. I understand that most likely your huge buffers come from you expecting your streamers to be using RTMP over crappy links, and therefore you already need to buffer THEIR data. Twitch really should invest in supporting SRT. It's supported in OBS. Anyway: Once you have the stream in your backend, the technology to have sub-second latency live streaming using existing web standards is there. https://airensoft.gitbook.io/ovenmediaengine/ https://airensoft.gitbook.io/ovenmediaengine/ But all of this being said: What you are doing there is looking amazing, so keep up the good work!
- kixelated 3y ago
  Glad you liked it! It's really difficult to compare the latency of different protocols because it depends on the network conditions. If you assume flawless connectivity, then real-time latency is trivial to achieve. Pipe frames over TCP like RTMP and bam, you've done it. It's almost meaningless to compare the best-case latency. The important part is determining how a protocol behaves during congestion. LL-HLS doesn't do great in that regard; frankly it will perform worse than RTMP if that's our yardstick because of head-of-line blocking, large fragments, and the playlist in the hot path. Twitch uses a fork of HLS called LHLS which should have lower latency, but we were still seeing 3-5s in some parts of the world. But yeah, P90 matters more than P10 when it comes to latency. One late frame ruins the broth. A real-time protocol needs a plan to avoid queues at all costs and that's just difficult with TCP.
moffkalast 3y ago
> The core issue is that WebRTC is not a protocol; it’s a monolith. > The WebRTC media stack is designed for conferencing and does an amazing job at it. The problems start when you try to use it for anything else. The main problem with WebRTC is that it's designed to be overcomplicated and garbage on purpose. A way to leverage UDP for video streaming without any possibility for anyone to ever send an actual UDP packet... so that it's not possible to turn every mildly popular website into a god tier DDOS botnet. WebRTC is what we get when we can't have nice things.
- englishm 3y ago
  I very much disagree with the characterization of WebRTC being "designed to be overcomplicated and garbage on purpose" but I do think your point about needing to not open the door to DDoS botnet capabilities is something worth highlighting. There are a number of very challenging constraints placed on the design of browser APIs and this is one of them that is often grappled with when trying to expose more powerful networking capabilities. Another that's particularly challenging to deal with in the media space is the need to avoid adding too much additional fingerprinting surface area. For each, the appropriate balance can be very difficult to strike and often requires a fair bit of creativity that can look like "complication" without the context of the constraints being solved for.
- jallmann 3y ago
  When WebRTC was being developed, a lot of the underlying protocols were derived from existing standards (RTP, SDP, etc), ostensibly with the intent of making it easier to bridge to legacy video-conference systems that may have been running SIP or similar. Beyond that, there is a ton of stuff to accommodate different device capabilities, network conditions, realities of the Internet, etc. WebRTC has tried to be a "nice thing" for developers from an API perspective for the use-case of streaming realtime video to the browser, but of course all those knobs under the hood make it seem like a hodgepodge, rightly or not. Would a clean-slate design with more tightly scoped goals fare better? Probably. But the underlying complexity needs to be handled somewhere and there are always trade-offs between control and ease-of-use.
- rizky05 3y ago
mosfets 3y ago
This is not going to replace the p2p functionality of webrtc right?
doubloon 3y ago
i made a small robot with realtime video, spent many hours on all these acronyms and webRTC stuff, wound up using simple jmuxer and python websockify. everything else was so complicated i could never figure it out. there is just so much jargon and layers on layers on layers.
formula1 3y ago
Personally, im interested in seeing flyweb developed. Devices easily locally connected sounds good to me https://flyweb.github.io/ https://flyweb.github.io/
keepamovin 3y ago
Twitch doesn’t need the same aggressive latency as Google Meet, but WebRTC is hard-coded to compromise on quality. In general, it’s quite difficult to customize WebRTC outside of a few configurable modes. It’s a black box that you turn on, and if it works it works. And if it doesn’t work, then you have to deal with the pain that is forking libwebrtc… or just hope Google fixes it for you. Solving this was indeed quite a challenge, but what we did was marry WebRTC data channels with compressed video frames that were unencoded. This meant we could drop frames all day long with zero artefacts and seamlessly adapt to the changing bandwidth conditions you find in the real world. The reason glitch-avoidance was so crucial was the same reason we wanted fine-grained control over the quality: the text and interface heavy use case would not permit frequent low quality frames. Our ack-based protocol is simple and effective. In fact, many companies use WebRTC data channels to avoid the WebRTC media stack (ex. Zoom). So, in a sense we use an approach similar to Zoom. The post author says this didn't work for them because their datagrams were not efficiently chunked. Somehow, we don't worry about this, likely because we don't need to retransmit frames avoiding head-of-line blocking, and our frames are compressed with JPEG, making them smaller by default. Our congestion control involves: - per N frames ack (or ACK is worth N frames, more efficient than per-frame-ack) - measuring frame-ack RTT over a short window and decreasing quality in 20% increments when alerts are tripped. As soon as RTT returns to normal we jump back to max quality. - if quality hits minimum, we increase Q, where we normally send every Qth-frame. Additionally we also maintain two channels: WebSocket and WebRTC, and we actually switch frames to the fastest one. It's usually WebRTC but sometimes it's WebSocket. This switching further eases congestion. Overall, it's a pretty smoothly working machine, developed from scratch through thousands of iterations and experiments. This system allows us to essentially side step most congestion. We trade frame rate for latency, as staying in "sync" with real time is the most important metric for application usability and responsiveness. This is because in resource constrained scenarios it's more important for people to feel that their actions are still having immediate effects, than it is to receive high frequency visual updates about those effects.

301 points by kixelated 3y ago

I got excited for a second that a new something will replace webrtc for media/videos.. it is not. Around 2 years ago in a project I wanted to transfer a raw 4K stream with low latency (sub 50ms) over cellular network, WebRTC performed poorly it was a no go, I ended up making my own algorithm that enables FEC (Forward Error Correction) based on IETF payload scheme 20, and piped it through UDP with Gstreamer and managed to make it work, but obviously wasn’t in the browser.

englishm 3y ago
That sounds like it was an interesting project! Jean-Baptiste Kempf also presented something at Demuxed last week using FEC and QUIC datagrams to get similarly low latency delivery over WebTransport into a browser. There's also a draft[1] for adding FEC to QUIC itself so it's quite possible Media over QUIC (MoQ) could benefit from this approach as well. I'm not sure why you say "it is not." We have a working MoQ demo running already on https://quic.video https://quic.video that includes a Rust-based relay server and a TypeScript player. The BBB demo video is being ingested using a Rust-based CLI tool I wrote called moq-pub which takes fragmented MP4 input from ffmpeg and sends it to a MoQ relay. You can also "go live" from your own browser if you'd like to test the latency and quality that way, too. [1]: https://www.ietf.org/archive/id/draft-michel-quic-fec-01.html https://www.ietf.org/archive/id/draft-michel-quic-fec-01.htm...
- tamimio 3y ago
  Thanks, it was a big project and streaming 4k in realtime from a flying drone was one of the challenging parts! I have some write up about it although nothing too technical but some videos there demonstrating some differences(1) > I'm not sure why you say "it is not." Pardon my ignorance it looked like it isn’t replacing WebRTC entirely yet, but glad I was wrong, I never tried anything QUIC for media related, would love to try the MoQ tool you did, and like the fact it’s rust based too as the one I did was written in rust. I will give it a test for sure, it’s been two years and I wasn’t following any updates so hopefully there’s an improvement compared what it was back then. > takes fragmented MP4 input from ffmpeg and sends it to a MoQ relay. Just a quick question, is ffmpeg a “requirement” per se for that CLI tool? As I remember I had to ditch ffmpeg in favor of gstreamer since the former one was eating up a lot of resources compared to Gstreamer, and it was crucial issue since the server was basically an SBC on a flying drone. (1) https://tamim.io/professional_projects/nerds-heavy-lift-drone-with-low-latency-command-and-control-over-cellular-5g-and-4k-video-stream-in-realtime/#4k-realtime-video-streaming https://tamim.io/professional_projects/nerds-heavy-lift-dron...
  - englishm 3y ago
    The current moq-pub implementation only requires valid fMP4 (a la CMAF) to be provided over stdin. I haven't tested, but I imagine you can do the same with gstreamer. Separately, I've been working on a wrapper library for moq-rs that I've been calling 'libmoq'. The intent there is to provide a C FFI that non-Rust code can link against. The first integration target for libmoq is ffmpeg. (I have a few bugs to work out before I clean up and shout about that code, but it does mostly work already.) I gave a presentation about some of this work last week at Demuxed, but the VoDs probably won't be available on YouTube until Decemberish. Also, I understand the gstreamer project has better support for Rust so I'll be looking at that soon, too.
    - oplav 3y ago
      I believe gstreamer has support for fMP4: https://gstreamer.freedesktop.org/documentation/fmp4/index.html https://gstreamer.freedesktop.org/documentation/fmp4/index.h... I stumbled upon a MoQ video by kixelated a couple months ago and have been meaning to give the above a try, but haven't gotten around to it yet so not sure if it will do the trick.
    - tamimio 3y ago
      Appreciated! Where do I find these presentations, if you don’t me asking?
      - englishm 3y ago
        https://2023.demuxed.com/ https://2023.demuxed.com/ has the day-long uncut videos available behind a $20 "VoD-only" post-event "ticket" paywall. Eventually https://www.youtube.com/@Demuxed https://www.youtube.com/@Demuxed will have the edited VoDs for Demuxed 2023 available for free. Usually those go up around December-ish. I also posted my slides on LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:7124766662339268608/ https://www.linkedin.com/feed/update/urn:li:activity:7124766...
Sean-Der 3y ago
That's a cool problem! I could see how WebRTC out of the box would perform poorly. It wants to NACK + delay to give a good 'conferencing experience'. I bet with FlexFEC [0] and playout-delay [1] you would get the behavior you were looking for. The sender would have to be custom, but the receivers (browsers) should just work! If you are interested in giving it a shot again would love to help :) [0] https://datatracker.ietf.org/doc/html/rfc8627 https://datatracker.ietf.org/doc/html/rfc8627 [1] https://webrtc.googlesource.com/src/+/main/docs/native-code/rtp-hdrext/playout-delay/README.md https://webrtc.googlesource.com/src/+/main/docs/native-code/...
- tamimio 3y ago
  It was an interesting problem indeed! I had some write up about it (and the whole project) in a link I have in the comment above, might gives more context. > I bet with FlexFEC [0] The previous draft (1) of this was my basis when I did the FEC sender. I didn’t manage to have it streamed 4K into the browser though, my client was OBS with gstreamer as it was far more performant than a browser in my tests, have you/any demo you did that I can try to stream it into the browser? That would be really major improvement! And appreciate the help, O would definitely give it another shot! (1) https://datatracker.ietf.org/doc/html/draft-ietf-payload-flexible-fec-scheme-20 https://datatracker.ietf.org/doc/html/draft-ietf-payload-fle...
  - Sean-Der 3y ago
    I would love too! Join https://pion.ly/slack https://pion.ly/slack and I am Sean-Der If you prefer discord there also is a dedicated space for ‘real-time broadcasting’ https://discord.gg/DV4ufzvJ4T https://discord.gg/DV4ufzvJ4T
    - tamimio 3y ago
      Thank you!
imtringued 3y ago
Well, a raw 4k stream has a bit rate of 11.9 gbps. I would be surprised if that worked over a cellular network at all.
- vlovich123 3y ago
  At 60fps but yeah, I think they mean they still passed the raw 4k stream through a lossy codec before putting it over cellular.
- tamimio 3y ago
  I think it was around 3gbps or even less if I remember correctly, it wasn’t 60fps and I can’t remember the color depth either, and the cellular network was a SA (Stand Alone) mmwave private network, not a commercial one, so it did work in our project eventually.
modeless 3y ago
Yeah. From what I can see WebRTC is basically a ready made implementation of a video calling app, bolted on the side of the browser, that you can slap some UI on top of. If you want to do anything that isn't a straight Zoom style video calling app (or Stadia, RIP), you'll immediately run into a wall of 5 year old known but unfixed bugs, or worse. To be fair, video calling is an incredibly important use case and I'm glad it can be done in the browser. I am grateful to the WebRTC team every time I can join a Zoom call without installing a bunch of extra crap on my machine. I just hope that WebRTC really can someday be replaced by composing multiple APIs that are much smaller in scope and more flexible, to allow for use cases other than essentially Zoom and Stadia. I guess I'm just repeating what the article said, but it's so right that it's worth repeating.
- est 3y ago
  > WebRTC is basically a ready made implementation of a video calling app, bolted on the side of the browser Yeah, Google bought GlobalIPSound, dumps the source code to w3c as a standard.

DaleCurtis 3y ago

Thanks for the nice write up! I work on the WebCodecs team at Chrome. I'm glad to hear it's mostly working for you. If you (or anyone else) has specific requests for new knobs regarding "We may need more encoding options, like non-reference frames or SVC", please file issues at https://github.com/w3c/webcodecs/issues https://github.com/w3c/webcodecs/issues

krebby 3y ago
Encoding alpha, please! https://github.com/w3c/webcodecs/issues/672 https://github.com/w3c/webcodecs/issues/672 Thanks for the great work on WebCodecs!
vlovich123 3y ago
There’s a few that would be neat: * maybe possible already, but it’s not immediately clear how to change the bitrate of the encoder dynamically when doing VBR/CBR (seems like you can only do it with per-frame quantization params which isn’t very friendly) * being able to specify the reference frame to use for encoding p frames * being able to generate slices efficiently / display them easily. For example, Oculus Link encodes 1/n of the video in parallel encoders and decodes similarly. This way your encoding time only contributes 1/n frame encode/decode worth of latency because the rest is amortized with tx+decode of other slices. I suspect the biggest requirement here is to be able to cheaply and easily get N VideoFrames OR be able to cheaply split a VideoFrame into horizontal or vertical slices.
- DaleCurtis 3y ago
  * Hmm, what kind of scheme are you thinking beyond per frame QP? Does an abstraction on top of QP work for the case you have in mind? * Reference frame control seems to be https://github.com/w3c/webcodecs/issues/285 https://github.com/w3c/webcodecs/issues/285, there's some interest in this for 2024, so I'd expect progress here. * Does splitting frames in WebGPU/WebGL work for the use case here? I'm not sure we could do anything internally (we're at the mercy of hardware decode implementations) without implementing such a shader.

fenesiistvan 3y ago

I don't get this. After the initial ICE negotiation you can just send raw RTP packets (encrypted with SRTP/DTLS). There is no need for any ACK packets. FEC can be done at codec level. What I am missing?

kixelated 3y ago
Are you referencing this line? > 2x the packets, because libsctp immediately ACKs every “datagram”. The section is about data channels, which uses SCTP and is ACK-based. Yes, you can use RTP with NACK and/or FEC with the media stack, but not with the data stack.
- the8472 3y ago
  TCP can coalesce acks for multiple packets, can't SCTP do the same?
  - Sean-Der 3y ago
    SCTP can (and does) a SACK[0] isn't needed for each DATA chunk. [0] https://datatracker.ietf.org/doc/html/rfc4960#section-3.3.4 https://datatracker.ietf.org/doc/html/rfc4960#section-3.3.4
  - kixelated 3y ago
    The protocol can do it, but libsctp (used by browsers) was not coalescing ACKs. I'm not sure if it has been fixed yet.
pthatcherg 3y ago
From a server or a native client, you can send whatever RTP packets you want, but you cannot send whatever RTP packets you want from a web client, and you cannot have access to the RTP packets from a web client and do whatever you want with them, at least not very easily. We are working on an extension to WebRTC called RtpTransport that would allow for just that, but it's in early stages of design and standardization.

gvkhna 3y ago

Unfortunately Safari is still a major hold out on WebTransport with no clear update, considering all other browsers have now supported it in GA for 3+ years.

englishm 3y ago
I don't know the timeline, but Apple has committed [1] to adding WebTransport support to WebKit. [1]: https://github.com/WebKit/standards-positions/issues/18#issuecomment-1495890122 https://github.com/WebKit/standards-positions/issues/18#issu...
- kixelated 3y ago
  Eric Kinnear (linked post; Apple) is the author of the HTTP/2 fallback for WebTransport, so it's safe to say that WebTransport will be available in WebKit at some point.
- gvkhna 3y ago
  Great news! It looks like Google GRPC team is probably waiting for Safari Webtransport support to implement grpc over the web in WT. Even though Chrome has had it for 2 years they have kept saying they’ll wait until it becomes GA.
gs17 3y ago
Yeah, I have a project where WebTransport would be a huge improvement, but not being able to support Safari is a dealbreaker.
doctorpangloss 3y ago
Google Chrome never implemented trailers, so no gRPC. Every browser is guilty. > I spent almost two years building/optimizing a partial WebRTC stack @ Twitch using pion. Our use-case was quite custom and we ultimately scrapped it, but your millage may vary. So many words about protocols. Protocols aren't hard or interesting. QA is hard. libwebrtc has an insurmountable lead on QA. None of these ad-hoc things, nor WebRTC implementations like Pion, will ever catch up, let alone be deployed in Mobile Safari.

sansseriff 3y ago

What's a good place to learn about all the networking jargon in this article? Also, a nitpicky point: As a community it would be nice to stop thinking of 'links to wikipedia pages, mdn docs, or github issues' as useful or pertinent forms of citations or footnotes. If I'm immersed in an article and I come across a term I don't know like 'SCTP', do you think a link to some dense proposal standard webpage is appropriate in this context for providing the background info I need? Of course academia is guilty of this too. Canonical citations don't tell the reader much else beyond 'I know what I'm talking about' and 'if you want to know more spend 30 minutes reading this other article' But since we're web based here I think we can do better. Tooltips that expand into a helpful paragraph about e.g. SCTP would be a start.

Sean-Der 3y ago
For the WebRTC jargon check out https://webrtcforthecurious.com/ https://webrtcforthecurious.com/ If that still doesn’t cover enough I would love to hear! Always trying to make it better.
englishm 3y ago
Here are a couple resources which may be helpful: - https://www.mux.com/video-glossary https://www.mux.com/video-glossary - https://howvideo.works/ https://howvideo.works/ Also, if you do want to take the time to read a longer and denser article, but come away with an understanding of much of the breadth of modern streaming tech, there's a really great survey paper available in pre-print here: https://doi.org/10.48550/arXiv.2310.03256 https://doi.org/10.48550/arXiv.2310.03256
IKantRead 3y ago
I really like High Performance Browser Networking by Ilya Grigorik. It's published by O'Reilly but is also free online [0]. What's particularly great about it is, unlike most other networking books, it focuses on the issue from the browser/web developer perspective which is particularly helpful for WebRTC and generally applicable to daily web dev work. 0. https://hpbn.co/ https://hpbn.co/

seydor 3y ago

> Back then, the web was a very different place. Flash was the only way to do live media and it was a mess. Not sure why people are saying that. WebRTC is far harder to make it work. Peer-to-peer is a cpu black-hole-level hog

kixelated 3y ago
I maintained the Flash video player at Twitch until I couldn't take it any longer and created an HTML5 player. Flash was a mess. :)
- seydor 3y ago
  is html5 video part of webrtc?
  - yalok 3y ago
    no, they are not related

mehagar 3y ago

An additional benefit of WebTransport over WebRTC DataChannels is that the WebTransport API is supported in Web Workers, meaning you can send and receive data off the main thread for better performance.

kixelated 3y ago
Absolutely! I'm doing that in my implementation: the main thread immediately transfers each incoming QUIC stream to a WebWorker, which then reads/decodes the container/codec and renders via OffscreenCanvas. I didn't realize that DataChannels were main thread only. That's good to know!
deleted 3y ago
[deleted]

znpy 3y ago

> The best and worst part about WebRTC is that it supports peer-to-peer. I hope P2P stays around in browsers.

englishm 3y ago
> I hope P2P stays around in browsers. I do, too. There was a W3C draft [1] ~pthatcherg was working on for P2P support for QUIC in the browser that could have maybe become a path to WebRTC using QUIC as a transport, but I think it may have been dropped somewhere along the line to the current WebTransport spec and implementations. (If Peter sees this, I'd love to learn more about how that transpired and what the current status of those ideas might be.) A more recent IETF individual draft [2] defines a way to do ICE/STUN-like address discovery using an extension to QUIC itself, so maybe that discussion indicates some revived interest in P2P use cases. [1]: https://w3c.github.io/p2p-webtransport/ https://w3c.github.io/p2p-webtransport/ [2]: https://datatracker.ietf.org/doc/html/draft-seemann-quic-address-discovery-00 https://datatracker.ietf.org/doc/html/draft-seemann-quic-add...

Fischgericht 3y ago

A wonderful write-up, thank you very much. However, it needs to be said that with currently available technologies, there is no need to have a 5 seconds buffer. Have a look at the amazing work of the OvenMedia team (not affiliated). Using their stack, and using Low-Latency HLS (LLHLS) I have been able to easily reach an end-to-end latency between the camera on site to the end-user viewing it in the browser of <500ms. At 20 MBit/s, and using either SRT or RTMP for stream upload. I understand that most likely your huge buffers come from you expecting your streamers to be using RTMP over crappy links, and therefore you already need to buffer THEIR data. Twitch really should invest in supporting SRT. It's supported in OBS. Anyway: Once you have the stream in your backend, the technology to have sub-second latency live streaming using existing web standards is there. https://airensoft.gitbook.io/ovenmediaengine/ https://airensoft.gitbook.io/ovenmediaengine/ But all of this being said: What you are doing there is looking amazing, so keep up the good work!

kixelated 3y ago
Glad you liked it! It's really difficult to compare the latency of different protocols because it depends on the network conditions. If you assume flawless connectivity, then real-time latency is trivial to achieve. Pipe frames over TCP like RTMP and bam, you've done it. It's almost meaningless to compare the best-case latency. The important part is determining how a protocol behaves during congestion. LL-HLS doesn't do great in that regard; frankly it will perform worse than RTMP if that's our yardstick because of head-of-line blocking, large fragments, and the playlist in the hot path. Twitch uses a fork of HLS called LHLS which should have lower latency, but we were still seeing 3-5s in some parts of the world. But yeah, P90 matters more than P10 when it comes to latency. One late frame ruins the broth. A real-time protocol needs a plan to avoid queues at all costs and that's just difficult with TCP.

moffkalast 3y ago

> The core issue is that WebRTC is not a protocol; it’s a monolith. > The WebRTC media stack is designed for conferencing and does an amazing job at it. The problems start when you try to use it for anything else. The main problem with WebRTC is that it's designed to be overcomplicated and garbage on purpose. A way to leverage UDP for video streaming without any possibility for anyone to ever send an actual UDP packet... so that it's not possible to turn every mildly popular website into a god tier DDOS botnet. WebRTC is what we get when we can't have nice things.

englishm 3y ago
I very much disagree with the characterization of WebRTC being "designed to be overcomplicated and garbage on purpose" but I do think your point about needing to not open the door to DDoS botnet capabilities is something worth highlighting. There are a number of very challenging constraints placed on the design of browser APIs and this is one of them that is often grappled with when trying to expose more powerful networking capabilities. Another that's particularly challenging to deal with in the media space is the need to avoid adding too much additional fingerprinting surface area. For each, the appropriate balance can be very difficult to strike and often requires a fair bit of creativity that can look like "complication" without the context of the constraints being solved for.
jallmann 3y ago
When WebRTC was being developed, a lot of the underlying protocols were derived from existing standards (RTP, SDP, etc), ostensibly with the intent of making it easier to bridge to legacy video-conference systems that may have been running SIP or similar. Beyond that, there is a ton of stuff to accommodate different device capabilities, network conditions, realities of the Internet, etc. WebRTC has tried to be a "nice thing" for developers from an API perspective for the use-case of streaming realtime video to the browser, but of course all those knobs under the hood make it seem like a hodgepodge, rightly or not. Would a clean-slate design with more tightly scoped goals fare better? Probably. But the underlying complexity needs to be handled somewhere and there are always trade-offs between control and ease-of-use.
rizky05 3y ago

mosfets 3y ago

This is not going to replace the p2p functionality of webrtc right?

doubloon 3y ago

i made a small robot with realtime video, spent many hours on all these acronyms and webRTC stuff, wound up using simple jmuxer and python websockify. everything else was so complicated i could never figure it out. there is just so much jargon and layers on layers on layers.

formula1 3y ago

Personally, im interested in seeing flyweb developed. Devices easily locally connected sounds good to me https://flyweb.github.io/ https://flyweb.github.io/

keepamovin 3y ago

Twitch doesn’t need the same aggressive latency as Google Meet, but WebRTC is hard-coded to compromise on quality. In general, it’s quite difficult to customize WebRTC outside of a few configurable modes. It’s a black box that you turn on, and if it works it works. And if it doesn’t work, then you have to deal with the pain that is forking libwebrtc… or just hope Google fixes it for you. Solving this was indeed quite a challenge, but what we did was marry WebRTC data channels with compressed video frames that were unencoded. This meant we could drop frames all day long with zero artefacts and seamlessly adapt to the changing bandwidth conditions you find in the real world. The reason glitch-avoidance was so crucial was the same reason we wanted fine-grained control over the quality: the text and interface heavy use case would not permit frequent low quality frames. Our ack-based protocol is simple and effective. In fact, many companies use WebRTC data channels to avoid the WebRTC media stack (ex. Zoom). So, in a sense we use an approach similar to Zoom. The post author says this didn't work for them because their datagrams were not efficiently chunked. Somehow, we don't worry about this, likely because we don't need to retransmit frames avoiding head-of-line blocking, and our frames are compressed with JPEG, making them smaller by default. Our congestion control involves: - per N frames ack (or ACK is worth N frames, more efficient than per-frame-ack) - measuring frame-ack RTT over a short window and decreasing quality in 20% increments when alerts are tripped. As soon as RTT returns to normal we jump back to max quality. - if quality hits minimum, we increase Q, where we normally send every Qth-frame. Additionally we also maintain two channels: WebSocket and WebRTC, and we actually switch frames to the fastest one. It's usually WebRTC but sometimes it's WebSocket. This switching further eases congestion. Overall, it's a pretty smoothly working machine, developed from scratch through thousands of iterations and experiments. This system allows us to essentially side step most congestion. We trade frame rate for latency, as staying in "sync" with real time is the most important metric for application usability and responsiveness. This is because in resource constrained scenarios it's more important for people to feel that their actions are still having immediate effects, than it is to receive high frequency visual updates about those effects.