3 ms·

I got excited for a second that a new something will replace webrtc for media/videos.. it is not. Around 2 years ago in a project I wanted to transfer a raw 4K

I got excited for a second that a new something will replace webrtc for media/videos.. it is not. Around 2 years ago in a project I wanted to transfer a raw 4K stream with low latency (sub 50ms) over cellular network, WebRTC performed poorly it was a no go, I ended up making my own algorithm that enables FEC (Forward Error Correction) based on IETF payload scheme 20, and piped it through UDP with Gstreamer and managed to make it work, but obviously wasn’t in the browser.

englishm 3y ago
That sounds like it was an interesting project! Jean-Baptiste Kempf also presented something at Demuxed last week using FEC and QUIC datagrams to get similarly low latency delivery over WebTransport into a browser. There's also a draft[1] for adding FEC to QUIC itself so it's quite possible Media over QUIC (MoQ) could benefit from this approach as well. I'm not sure why you say "it is not." We have a working MoQ demo running already on https://quic.video https://quic.video that includes a Rust-based relay server and a TypeScript player. The BBB demo video is being ingested using a Rust-based CLI tool I wrote called moq-pub which takes fragmented MP4 input from ffmpeg and sends it to a MoQ relay. You can also "go live" from your own browser if you'd like to test the latency and quality that way, too. [1]: https://www.ietf.org/archive/id/draft-michel-quic-fec-01.html https://www.ietf.org/archive/id/draft-michel-quic-fec-01.htm...
- tamimio 3y ago
  Thanks, it was a big project and streaming 4k in realtime from a flying drone was one of the challenging parts! I have some write up about it although nothing too technical but some videos there demonstrating some differences(1) > I'm not sure why you say "it is not." Pardon my ignorance it looked like it isn’t replacing WebRTC entirely yet, but glad I was wrong, I never tried anything QUIC for media related, would love to try the MoQ tool you did, and like the fact it’s rust based too as the one I did was written in rust. I will give it a test for sure, it’s been two years and I wasn’t following any updates so hopefully there’s an improvement compared what it was back then. > takes fragmented MP4 input from ffmpeg and sends it to a MoQ relay. Just a quick question, is ffmpeg a “requirement” per se for that CLI tool? As I remember I had to ditch ffmpeg in favor of gstreamer since the former one was eating up a lot of resources compared to Gstreamer, and it was crucial issue since the server was basically an SBC on a flying drone. (1) https://tamim.io/professional_projects/nerds-heavy-lift-drone-with-low-latency-command-and-control-over-cellular-5g-and-4k-video-stream-in-realtime/#4k-realtime-video-streaming https://tamim.io/professional_projects/nerds-heavy-lift-dron...
  - englishm 3y ago
    The current moq-pub implementation only requires valid fMP4 (a la CMAF) to be provided over stdin. I haven't tested, but I imagine you can do the same with gstreamer. Separately, I've been working on a wrapper library for moq-rs that I've been calling 'libmoq'. The intent there is to provide a C FFI that non-Rust code can link against. The first integration target for libmoq is ffmpeg. (I have a few bugs to work out before I clean up and shout about that code, but it does mostly work already.) I gave a presentation about some of this work last week at Demuxed, but the VoDs probably won't be available on YouTube until Decemberish. Also, I understand the gstreamer project has better support for Rust so I'll be looking at that soon, too.
    - oplav 3y ago
      I believe gstreamer has support for fMP4: https://gstreamer.freedesktop.org/documentation/fmp4/index.html https://gstreamer.freedesktop.org/documentation/fmp4/index.h... I stumbled upon a MoQ video by kixelated a couple months ago and have been meaning to give the above a try, but haven't gotten around to it yet so not sure if it will do the trick.
    - tamimio 3y ago
      Appreciated! Where do I find these presentations, if you don’t me asking?
      - englishm 3y ago
        https://2023.demuxed.com/ https://2023.demuxed.com/ has the day-long uncut videos available behind a $20 "VoD-only" post-event "ticket" paywall. Eventually https://www.youtube.com/@Demuxed https://www.youtube.com/@Demuxed will have the edited VoDs for Demuxed 2023 available for free. Usually those go up around December-ish. I also posted my slides on LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:7124766662339268608/ https://www.linkedin.com/feed/update/urn:li:activity:7124766...
Sean-Der 3y ago
That's a cool problem! I could see how WebRTC out of the box would perform poorly. It wants to NACK + delay to give a good 'conferencing experience'. I bet with FlexFEC [0] and playout-delay [1] you would get the behavior you were looking for. The sender would have to be custom, but the receivers (browsers) should just work! If you are interested in giving it a shot again would love to help :) [0] https://datatracker.ietf.org/doc/html/rfc8627 https://datatracker.ietf.org/doc/html/rfc8627 [1] https://webrtc.googlesource.com/src/+/main/docs/native-code/rtp-hdrext/playout-delay/README.md https://webrtc.googlesource.com/src/+/main/docs/native-code/...
- tamimio 3y ago
  It was an interesting problem indeed! I had some write up about it (and the whole project) in a link I have in the comment above, might gives more context. > I bet with FlexFEC [0] The previous draft (1) of this was my basis when I did the FEC sender. I didn’t manage to have it streamed 4K into the browser though, my client was OBS with gstreamer as it was far more performant than a browser in my tests, have you/any demo you did that I can try to stream it into the browser? That would be really major improvement! And appreciate the help, O would definitely give it another shot! (1) https://datatracker.ietf.org/doc/html/draft-ietf-payload-flexible-fec-scheme-20 https://datatracker.ietf.org/doc/html/draft-ietf-payload-fle...
  - Sean-Der 3y ago
    I would love too! Join https://pion.ly/slack https://pion.ly/slack and I am Sean-Der If you prefer discord there also is a dedicated space for ‘real-time broadcasting’ https://discord.gg/DV4ufzvJ4T https://discord.gg/DV4ufzvJ4T
    - tamimio 3y ago
      Thank you!
imtringued 3y ago
Well, a raw 4k stream has a bit rate of 11.9 gbps. I would be surprised if that worked over a cellular network at all.
- vlovich123 3y ago
  At 60fps but yeah, I think they mean they still passed the raw 4k stream through a lossy codec before putting it over cellular.
- tamimio 3y ago
  I think it was around 3gbps or even less if I remember correctly, it wasn’t 60fps and I can’t remember the color depth either, and the cellular network was a SA (Stand Alone) mmwave private network, not a commercial one, so it did work in our project eventually.
modeless 3y ago
Yeah. From what I can see WebRTC is basically a ready made implementation of a video calling app, bolted on the side of the browser, that you can slap some UI on top of. If you want to do anything that isn't a straight Zoom style video calling app (or Stadia, RIP), you'll immediately run into a wall of 5 year old known but unfixed bugs, or worse. To be fair, video calling is an incredibly important use case and I'm glad it can be done in the browser. I am grateful to the WebRTC team every time I can join a Zoom call without installing a bunch of extra crap on my machine. I just hope that WebRTC really can someday be replaced by composing multiple APIs that are much smaller in scope and more flexible, to allow for use cases other than essentially Zoom and Stadia. I guess I'm just repeating what the article said, but it's so right that it's worth repeating.
- est 3y ago
  > WebRTC is basically a ready made implementation of a video calling app, bolted on the side of the browser Yeah, Google bought GlobalIPSound, dumps the source code to w3c as a standard.

englishm 3y ago

That sounds like it was an interesting project! Jean-Baptiste Kempf also presented something at Demuxed last week using FEC and QUIC datagrams to get similarly low latency delivery over WebTransport into a browser. There's also a draft[1] for adding FEC to QUIC itself so it's quite possible Media over QUIC (MoQ) could benefit from this approach as well. I'm not sure why you say "it is not." We have a working MoQ demo running already on https://quic.video https://quic.video that includes a Rust-based relay server and a TypeScript player. The BBB demo video is being ingested using a Rust-based CLI tool I wrote called moq-pub which takes fragmented MP4 input from ffmpeg and sends it to a MoQ relay. You can also "go live" from your own browser if you'd like to test the latency and quality that way, too. [1]: https://www.ietf.org/archive/id/draft-michel-quic-fec-01.html https://www.ietf.org/archive/id/draft-michel-quic-fec-01.htm...

tamimio 3y ago
Thanks, it was a big project and streaming 4k in realtime from a flying drone was one of the challenging parts! I have some write up about it although nothing too technical but some videos there demonstrating some differences(1) > I'm not sure why you say "it is not." Pardon my ignorance it looked like it isn’t replacing WebRTC entirely yet, but glad I was wrong, I never tried anything QUIC for media related, would love to try the MoQ tool you did, and like the fact it’s rust based too as the one I did was written in rust. I will give it a test for sure, it’s been two years and I wasn’t following any updates so hopefully there’s an improvement compared what it was back then. > takes fragmented MP4 input from ffmpeg and sends it to a MoQ relay. Just a quick question, is ffmpeg a “requirement” per se for that CLI tool? As I remember I had to ditch ffmpeg in favor of gstreamer since the former one was eating up a lot of resources compared to Gstreamer, and it was crucial issue since the server was basically an SBC on a flying drone. (1) https://tamim.io/professional_projects/nerds-heavy-lift-drone-with-low-latency-command-and-control-over-cellular-5g-and-4k-video-stream-in-realtime/#4k-realtime-video-streaming https://tamim.io/professional_projects/nerds-heavy-lift-dron...
- englishm 3y ago
  The current moq-pub implementation only requires valid fMP4 (a la CMAF) to be provided over stdin. I haven't tested, but I imagine you can do the same with gstreamer. Separately, I've been working on a wrapper library for moq-rs that I've been calling 'libmoq'. The intent there is to provide a C FFI that non-Rust code can link against. The first integration target for libmoq is ffmpeg. (I have a few bugs to work out before I clean up and shout about that code, but it does mostly work already.) I gave a presentation about some of this work last week at Demuxed, but the VoDs probably won't be available on YouTube until Decemberish. Also, I understand the gstreamer project has better support for Rust so I'll be looking at that soon, too.
  - oplav 3y ago
    I believe gstreamer has support for fMP4: https://gstreamer.freedesktop.org/documentation/fmp4/index.html https://gstreamer.freedesktop.org/documentation/fmp4/index.h... I stumbled upon a MoQ video by kixelated a couple months ago and have been meaning to give the above a try, but haven't gotten around to it yet so not sure if it will do the trick.
  - tamimio 3y ago
    Appreciated! Where do I find these presentations, if you don’t me asking?
    - englishm 3y ago
      https://2023.demuxed.com/ https://2023.demuxed.com/ has the day-long uncut videos available behind a $20 "VoD-only" post-event "ticket" paywall. Eventually https://www.youtube.com/@Demuxed https://www.youtube.com/@Demuxed will have the edited VoDs for Demuxed 2023 available for free. Usually those go up around December-ish. I also posted my slides on LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:7124766662339268608/ https://www.linkedin.com/feed/update/urn:li:activity:7124766...

Sean-Der 3y ago

That's a cool problem! I could see how WebRTC out of the box would perform poorly. It wants to NACK + delay to give a good 'conferencing experience'. I bet with FlexFEC [0] and playout-delay [1] you would get the behavior you were looking for. The sender would have to be custom, but the receivers (browsers) should just work! If you are interested in giving it a shot again would love to help :) [0] https://datatracker.ietf.org/doc/html/rfc8627 https://datatracker.ietf.org/doc/html/rfc8627 [1] https://webrtc.googlesource.com/src/+/main/docs/native-code/rtp-hdrext/playout-delay/README.md https://webrtc.googlesource.com/src/+/main/docs/native-code/...

tamimio 3y ago
It was an interesting problem indeed! I had some write up about it (and the whole project) in a link I have in the comment above, might gives more context. > I bet with FlexFEC [0] The previous draft (1) of this was my basis when I did the FEC sender. I didn’t manage to have it streamed 4K into the browser though, my client was OBS with gstreamer as it was far more performant than a browser in my tests, have you/any demo you did that I can try to stream it into the browser? That would be really major improvement! And appreciate the help, O would definitely give it another shot! (1) https://datatracker.ietf.org/doc/html/draft-ietf-payload-flexible-fec-scheme-20 https://datatracker.ietf.org/doc/html/draft-ietf-payload-fle...
- Sean-Der 3y ago
  I would love too! Join https://pion.ly/slack https://pion.ly/slack and I am Sean-Der If you prefer discord there also is a dedicated space for ‘real-time broadcasting’ https://discord.gg/DV4ufzvJ4T https://discord.gg/DV4ufzvJ4T
  - tamimio 3y ago
    Thank you!

imtringued 3y ago

Well, a raw 4k stream has a bit rate of 11.9 gbps. I would be surprised if that worked over a cellular network at all.

vlovich123 3y ago
At 60fps but yeah, I think they mean they still passed the raw 4k stream through a lossy codec before putting it over cellular.
tamimio 3y ago
I think it was around 3gbps or even less if I remember correctly, it wasn’t 60fps and I can’t remember the color depth either, and the cellular network was a SA (Stand Alone) mmwave private network, not a commercial one, so it did work in our project eventually.

modeless 3y ago

Yeah. From what I can see WebRTC is basically a ready made implementation of a video calling app, bolted on the side of the browser, that you can slap some UI on top of. If you want to do anything that isn't a straight Zoom style video calling app (or Stadia, RIP), you'll immediately run into a wall of 5 year old known but unfixed bugs, or worse. To be fair, video calling is an incredibly important use case and I'm glad it can be done in the browser. I am grateful to the WebRTC team every time I can join a Zoom call without installing a bunch of extra crap on my machine. I just hope that WebRTC really can someday be replaced by composing multiple APIs that are much smaller in scope and more flexible, to allow for use cases other than essentially Zoom and Stadia. I guess I'm just repeating what the article said, but it's so right that it's worth repeating.

est 3y ago
> WebRTC is basically a ready made implementation of a video calling app, bolted on the side of the browser Yeah, Google bought GlobalIPSound, dumps the source code to w3c as a standard.