8 ms·

Streaming Video and Audio with Low Delay

117 points by mortenvp 9y ago

thefourthchime 9y ago
The title is misleading, when they say low-delay, really they mean over UDP instead of TCP. Their answer to video over UDP is a marginal efficiency gain in a forward error correction algorithm called RLNC. I haven't looked at RLNC, but it does seem to have some gains over more traditional FEC schemes. https://arxiv.org/abs/1802.04873 https://arxiv.org/abs/1802.04873
- mortenvp 9y ago
  As stated in the article, low delay is very hard to achieve over a reliable transport based on re-transmissions (such as TCP) - you have to wait for the loss to be detected and then the retransmitted packet to arrive. This is at least 3 x the latency of the link. Therefore if you really care about bounded low latency you need to use some form of erasure correcting algorithm (to provide upfront redundancy). In the article we simply show that substituting one "old" algorithm for a more modern one can give you much better efficiency (protection against packet loss) for the same bandwidth and latency/delay budget.
  - tokyodude 9y ago
    I'm not disputing anything you mentioned but I just thought I'd point out that, at least in my experience, Soundcloud has close to instant response streaming audio. Pick a new song it starts streaming immediately. Que a different time and it jumps to that part immediately. Looking in the devtools it's pretty clear they are sending lots of small packets of audio over websockets rather than using the browser's built in audio streaming features as most music sites do. Of course it probably still has the TCP issue if there is an error but in general it's way more responsive than other music sites I've used.
    - thefourthchime 9y ago
      Audio is much easier to stream though. You need nearly no data to start streaming audio, and it's likely you're internet will keep up. The video is very different, the overhead is a lot more, 128kbit vs 2Mbps for example. On top of that, a video player will try to buffer ahead enough video to make sure you won't get and buffering later on, which means it needs to measure your bandwidth and try to guess how much. UDP can be nearly instant except in for b-frames, however if it's over UDP is probably over transport stream which has it's own buffering levels built in to make sure the decoder has a large enough buffer.
      - Selfcommit 9y ago
        So the overhead to begin a video stream is due to the size of video data? Would video behave generally more like audio data (could stream immidiately) if the typical user bandwidth was larger?
        IshKebab 9y ago
        Yes. Although it still might not be as good due to TCP's slow start.
        thefourthchime 9y ago
        In reality it's the size yes, but there is an inherent delay because of the way B-frames work. You'll need to buffer 2-3 frames worth 33ms each (for NTSC), so 66-99ms delay regardless of throughput. Audio doesn't have such a concept, you get 33ms worth of audio, you can play it right away. Of course, you can opt to not have b-frames and negate that issue.
        mgamache 9y ago
        most (all?) of the video codecs designed for low latency don't use b-frames for this reason.
        9y ago
  - thefourthchime 9y ago
    Please help me understand then. I see RLNC compared to 2022 Single mode has the same overhead of 25%. When I compare the two using your tool at the end of the article, the only change I see is an improvement from 5% random loss to 25%. The burst loose stays the same. Correct?
    - mortenvp 9y ago
      Yes, you get a much better random loss protection with RLNC. So if you know that all your losses are bursts you may be able to live with 2022. Essentially the difference between the two algorithms are that with 2022 only a subset of your packets are protected by a redundancy packet (and you have to choose how to protect them), whereas with RLNC you can protect all available packets. If we leave the premise of this post (namely that we want to generate traffic in the same pattern as 2022) RLNC can even protect against longer bursts compared to 2022. Did that make sense?
fenesiistvan 9y ago
Instead of a separate protocol you can already use a codec with built-in FEC such as OPUS.
- nh2 9y ago
  Which video codecs have that inbuilt?
  - TD-Linux 9y ago
    None, so a separate FEC can still be useful. You can use FEC like described in the article with WebRTC: https://tools.ietf.org/html/draft-ietf-rtcweb-fec-08 https://tools.ietf.org/html/draft-ietf-rtcweb-fec-08
jakobegger 9y ago
Anybody have an idea how to put this into practice? I recently tried streaming video over wifi from a Raspberry (for a robotics hobby project) , and everything I tried was either unusable or very delayed. Is there an open source low latency video streaming solution for hobbyists?
- Jakob 9y ago
  Have you tried ffmpeg? Its streaming capabilities always worked really well for me. It can stream over UDP as well: https://trac.ffmpeg.org/wiki/StreamingGuide https://trac.ffmpeg.org/wiki/StreamingGuide. Also see the paragraph on latency (https://trac.ffmpeg.org/wiki/StreamingGuide#Latency https://trac.ffmpeg.org/wiki/StreamingGuide#Latency) on how to avoid common pitfalls.
- mgamache 9y ago
  you should easily get < 1 second latency from an RPi => Desktop over WiFi. Strangely it's usually the client's buffering that is the culprit.
  - harlanji 9y ago
    Confirmed. Best results can be had by starting with the Ubuntu Mate distro which has the rpi omx and mmal optimized ffmpeg out of the box, else steps to compile exactly right were time consuming. Here’s a gif of RPi streaming through a CDN which itself is fast, showing timestamps, using janky ffmpeg I built (aka. Worst-case): https://twitter.com/iSpoogeDaily/status/986681611600084992?s=20 https://twitter.com/iSpoogeDaily/status/986681611600084992?s... Tuning is ongoing, learning as I go. Thanks for client buffering pointer.
- mbrumlow 9y ago
  I use ffmpeg and some custom services for http://robot247.io http://robot247.io All the code is on GitHub. I can get very low latency video. Most of the delay comes in the form of the speed of light being so slow. On the web site in use jsmpeg.
  - 9y ago
kazinator 9y ago
Low delay is much, much more important for calls than for streaming. One second of buffering delay may be acceptable in streaming playback (users often contend with longer delays). That much delay will severely degrade a video call, especially if the audio stays synced with the delayed video.
- tzahola 9y ago
  It depends. Maybe people have live news streams fed into a neural network controlling a high frequency trading engine.
  - mgamache 9y ago
    like robo-watching CNBC and trading based on that?
- MiniMax42 9y ago
  Also, for real-life auctions where online bidders can participate
  - kazinator 9y ago
    There is justification in regarding that as a form of video call rather than media playback, even though the video may be only in one direction and the reverse communication (flow of bids) isn't AV.
  - bcryptd 9y ago
    We are testing WebRTC for this very purpose and seeing under 500ms latency to our clients.
- laumars 9y ago
  That really depends on the service. For on-demand services like Netflix or YouTube your argument is a fair one. But for live services it matters a lot more - particularly in broadcasting. Even more so if your clients are watching the same feed and using that to determine when to inject ads (eg in a live sporting event). Then it quickly becomes very important to keep latency as low as possible.
coldsauce 9y ago
Anyone know what the best streaming solution for browser <-> browser video calling is? It probably has to be built on top of WebRTC but I'm wondering if there are codecs and forward error correction algorithms out there already in Javascript to use.
- stanlarroque 9y ago
  Kurento is an excellent open source solution for this use-case. Check out the project here: https://github.com/Kurento/kurento-media-server https://github.com/Kurento/kurento-media-server
- kwindla 9y ago
  For small numbers of people in a call (say, n < 5), the "best" -- meaning lowest latency -- solution is direct browser to browser WebRTC connections. Both Chrome's and Firefox's WebRTC implementations have quite good FEC built in. And sending UDP packets directly between peers will have much lower latency than routing through a media server. Of course, sometimes peer-to-peer won't work for you. Maybe you have requirements that push you towards routing media through a server. (Content filtering, or compositing video or mixing audio, for example.) Or maybe you have more than a few people in a call. If so, upstream bandwidth and encoding become bottlenecks for mesh/peer-to-peer. Finally, some firewalls won't allow UDP traffic from/to computers behind them, so you'll need to route UDP through a central server, or (much worse) tunnel over TCP. Back on the subject of latency and error correction in WebRTC, here are some fun links: Draft spec for FEC in WebRTC: https://tools.ietf.org/html/draft-ietf-rtcweb-fec-08 https://tools.ietf.org/html/draft-ietf-rtcweb-fec-08 Mozilla article from when they first turned on Opus FEC. Includes sample audio for calls with 19% packet loss. (19% packet loss is very, very bad. My startup makes a browser-to-browser video calling tool, and we try hard to deal well with packet loss that high, but it's a losing battle.) https://blog.mozilla.org/webrtc/audio-fec-experiments/ https://blog.mozilla.org/webrtc/audio-fec-experiments/ Notes from the very knowledgeable folks at Callstats.io about WebRTC FEC. Covers some of the same material as this thread's original post: https://www.callstats.io/2016/11/09/how-to-recover-lost-media-packets-in-webrtc-with-fec/ https://www.callstats.io/2016/11/09/how-to-recover-lost-medi... Tsahi Levent-Levi's benchmarks showing how a few different media servers perform in the context of 10% packet loss: https://testrtc.com/webrtc-media-server-packet-loss/ https://testrtc.com/webrtc-media-server-packet-loss/
urlgrey 9y ago
Periscope developed a Low-Latency HTTP Live Streaming (LHLS) technique that relies on HTTP chunked transfer-encoding to stream video bytes as they are encoded at the origin. This is still subject to TCP packet retransmission overhead, but the time-to-first-byte is reduced significantly and leads to less buffering on the client. Here's a Periscope post about LHLS: https://medium.com/@periscopecode/introducing-lhls-media-streaming-eb6212948bef https://medium.com/@periscopecode/introducing-lhls-media-str... Most systems that serve HLS media use fixed content-length segments, which requires knowledge of the length of a segment before the first byte can be sent over the wire. So, for a 5 second segment you would need to encode the entire 5 seconds before the first byte can be sent; this does not apply when streaming the segments with chunked transfer encoding. Incidentally, at Mux we also use chunked transfer-encoding to stream video that is encoded on-the-fly with great performance.
- reggieband 9y ago
  I've heard from colleagues that this won't be possible with DASH due to the switch to fMP4 format. One of my co-workers tells me that fMP4 requires the entire segment to be loaded before playback can begin while TS segments don't require this. We've been looking into very small segments (e.g. 1s duration) to reduce latency but I've been interested in the LHLS approach since I first heard of it.
  - TD-Linux 9y ago
    fMP4 has an index for each chunk, so you have to buffer the whole thing to create the index on the writing side. However, with DASH you do have the option for WebM as well, which does not need and index and can be streamed. Or really small fMP4 segments work too.
  - urlgrey 9y ago
    Very short segment durations are effective only when latency is more important than quality. Each TS segment must start with a key-frame, and the GOP size can't exceed the duration of a segment (e.g. one second). Lowering the segment duration increases the frequency of key-frames, which has the effect of lowering the quality you can achieve at a given bitrate.
rjeli 9y ago
Anyone know what PSNow/ cloud gaming services use? I’m using it to play some ps3 games and the delay is immediately noticeable when you start playing, but your brain adjusts and it’s not noticeable. Has to be around 300-400 ms round trip

117 points by mortenvp 9y ago

The title is misleading, when they say low-delay, really they mean over UDP instead of TCP. Their answer to video over UDP is a marginal efficiency gain in a forward error correction algorithm called RLNC. I haven't looked at RLNC, but it does seem to have some gains over more traditional FEC schemes. https://arxiv.org/abs/1802.04873 https://arxiv.org/abs/1802.04873

mortenvp 9y ago
As stated in the article, low delay is very hard to achieve over a reliable transport based on re-transmissions (such as TCP) - you have to wait for the loss to be detected and then the retransmitted packet to arrive. This is at least 3 x the latency of the link. Therefore if you really care about bounded low latency you need to use some form of erasure correcting algorithm (to provide upfront redundancy). In the article we simply show that substituting one "old" algorithm for a more modern one can give you much better efficiency (protection against packet loss) for the same bandwidth and latency/delay budget.
- tokyodude 9y ago
  I'm not disputing anything you mentioned but I just thought I'd point out that, at least in my experience, Soundcloud has close to instant response streaming audio. Pick a new song it starts streaming immediately. Que a different time and it jumps to that part immediately. Looking in the devtools it's pretty clear they are sending lots of small packets of audio over websockets rather than using the browser's built in audio streaming features as most music sites do. Of course it probably still has the TCP issue if there is an error but in general it's way more responsive than other music sites I've used.
  - thefourthchime 9y ago
    Audio is much easier to stream though. You need nearly no data to start streaming audio, and it's likely you're internet will keep up. The video is very different, the overhead is a lot more, 128kbit vs 2Mbps for example. On top of that, a video player will try to buffer ahead enough video to make sure you won't get and buffering later on, which means it needs to measure your bandwidth and try to guess how much. UDP can be nearly instant except in for b-frames, however if it's over UDP is probably over transport stream which has it's own buffering levels built in to make sure the decoder has a large enough buffer.
    - Selfcommit 9y ago
      So the overhead to begin a video stream is due to the size of video data? Would video behave generally more like audio data (could stream immidiately) if the typical user bandwidth was larger?
      - IshKebab 9y ago
        Yes. Although it still might not be as good due to TCP's slow start.
      - thefourthchime 9y ago
        In reality it's the size yes, but there is an inherent delay because of the way B-frames work. You'll need to buffer 2-3 frames worth 33ms each (for NTSC), so 66-99ms delay regardless of throughput. Audio doesn't have such a concept, you get 33ms worth of audio, you can play it right away. Of course, you can opt to not have b-frames and negate that issue.
        mgamache 9y ago
        most (all?) of the video codecs designed for low latency don't use b-frames for this reason.
        9y ago
- thefourthchime 9y ago
  Please help me understand then. I see RLNC compared to 2022 Single mode has the same overhead of 25%. When I compare the two using your tool at the end of the article, the only change I see is an improvement from 5% random loss to 25%. The burst loose stays the same. Correct?
  - mortenvp 9y ago
    Yes, you get a much better random loss protection with RLNC. So if you know that all your losses are bursts you may be able to live with 2022. Essentially the difference between the two algorithms are that with 2022 only a subset of your packets are protected by a redundancy packet (and you have to choose how to protect them), whereas with RLNC you can protect all available packets. If we leave the premise of this post (namely that we want to generate traffic in the same pattern as 2022) RLNC can even protect against longer bursts compared to 2022. Did that make sense?

fenesiistvan 9y ago

Instead of a separate protocol you can already use a codec with built-in FEC such as OPUS.

nh2 9y ago
Which video codecs have that inbuilt?
- TD-Linux 9y ago
  None, so a separate FEC can still be useful. You can use FEC like described in the article with WebRTC: https://tools.ietf.org/html/draft-ietf-rtcweb-fec-08 https://tools.ietf.org/html/draft-ietf-rtcweb-fec-08

jakobegger 9y ago

Anybody have an idea how to put this into practice? I recently tried streaming video over wifi from a Raspberry (for a robotics hobby project) , and everything I tried was either unusable or very delayed. Is there an open source low latency video streaming solution for hobbyists?

Jakob 9y ago
Have you tried ffmpeg? Its streaming capabilities always worked really well for me. It can stream over UDP as well: https://trac.ffmpeg.org/wiki/StreamingGuide https://trac.ffmpeg.org/wiki/StreamingGuide. Also see the paragraph on latency (https://trac.ffmpeg.org/wiki/StreamingGuide#Latency https://trac.ffmpeg.org/wiki/StreamingGuide#Latency) on how to avoid common pitfalls.
mgamache 9y ago
you should easily get < 1 second latency from an RPi => Desktop over WiFi. Strangely it's usually the client's buffering that is the culprit.
- harlanji 9y ago
  Confirmed. Best results can be had by starting with the Ubuntu Mate distro which has the rpi omx and mmal optimized ffmpeg out of the box, else steps to compile exactly right were time consuming. Here’s a gif of RPi streaming through a CDN which itself is fast, showing timestamps, using janky ffmpeg I built (aka. Worst-case): https://twitter.com/iSpoogeDaily/status/986681611600084992?s=20 https://twitter.com/iSpoogeDaily/status/986681611600084992?s... Tuning is ongoing, learning as I go. Thanks for client buffering pointer.
mbrumlow 9y ago
I use ffmpeg and some custom services for http://robot247.io http://robot247.io All the code is on GitHub. I can get very low latency video. Most of the delay comes in the form of the speed of light being so slow. On the web site in use jsmpeg.
- 9y ago

kazinator 9y ago

Low delay is much, much more important for calls than for streaming. One second of buffering delay may be acceptable in streaming playback (users often contend with longer delays). That much delay will severely degrade a video call, especially if the audio stays synced with the delayed video.

tzahola 9y ago
It depends. Maybe people have live news streams fed into a neural network controlling a high frequency trading engine.
- mgamache 9y ago
  like robo-watching CNBC and trading based on that?
MiniMax42 9y ago
Also, for real-life auctions where online bidders can participate
- kazinator 9y ago
  There is justification in regarding that as a form of video call rather than media playback, even though the video may be only in one direction and the reverse communication (flow of bids) isn't AV.
- bcryptd 9y ago
  We are testing WebRTC for this very purpose and seeing under 500ms latency to our clients.
laumars 9y ago
That really depends on the service. For on-demand services like Netflix or YouTube your argument is a fair one. But for live services it matters a lot more - particularly in broadcasting. Even more so if your clients are watching the same feed and using that to determine when to inject ads (eg in a live sporting event). Then it quickly becomes very important to keep latency as low as possible.

coldsauce 9y ago

Anyone know what the best streaming solution for browser <-> browser video calling is? It probably has to be built on top of WebRTC but I'm wondering if there are codecs and forward error correction algorithms out there already in Javascript to use.

stanlarroque 9y ago
Kurento is an excellent open source solution for this use-case. Check out the project here: https://github.com/Kurento/kurento-media-server https://github.com/Kurento/kurento-media-server
kwindla 9y ago
For small numbers of people in a call (say, n < 5), the "best" -- meaning lowest latency -- solution is direct browser to browser WebRTC connections. Both Chrome's and Firefox's WebRTC implementations have quite good FEC built in. And sending UDP packets directly between peers will have much lower latency than routing through a media server. Of course, sometimes peer-to-peer won't work for you. Maybe you have requirements that push you towards routing media through a server. (Content filtering, or compositing video or mixing audio, for example.) Or maybe you have more than a few people in a call. If so, upstream bandwidth and encoding become bottlenecks for mesh/peer-to-peer. Finally, some firewalls won't allow UDP traffic from/to computers behind them, so you'll need to route UDP through a central server, or (much worse) tunnel over TCP. Back on the subject of latency and error correction in WebRTC, here are some fun links: Draft spec for FEC in WebRTC: https://tools.ietf.org/html/draft-ietf-rtcweb-fec-08 https://tools.ietf.org/html/draft-ietf-rtcweb-fec-08 Mozilla article from when they first turned on Opus FEC. Includes sample audio for calls with 19% packet loss. (19% packet loss is very, very bad. My startup makes a browser-to-browser video calling tool, and we try hard to deal well with packet loss that high, but it's a losing battle.) https://blog.mozilla.org/webrtc/audio-fec-experiments/ https://blog.mozilla.org/webrtc/audio-fec-experiments/ Notes from the very knowledgeable folks at Callstats.io about WebRTC FEC. Covers some of the same material as this thread's original post: https://www.callstats.io/2016/11/09/how-to-recover-lost-media-packets-in-webrtc-with-fec/ https://www.callstats.io/2016/11/09/how-to-recover-lost-medi... Tsahi Levent-Levi's benchmarks showing how a few different media servers perform in the context of 10% packet loss: https://testrtc.com/webrtc-media-server-packet-loss/ https://testrtc.com/webrtc-media-server-packet-loss/

urlgrey 9y ago

Periscope developed a Low-Latency HTTP Live Streaming (LHLS) technique that relies on HTTP chunked transfer-encoding to stream video bytes as they are encoded at the origin. This is still subject to TCP packet retransmission overhead, but the time-to-first-byte is reduced significantly and leads to less buffering on the client. Here's a Periscope post about LHLS: https://medium.com/@periscopecode/introducing-lhls-media-streaming-eb6212948bef https://medium.com/@periscopecode/introducing-lhls-media-str... Most systems that serve HLS media use fixed content-length segments, which requires knowledge of the length of a segment before the first byte can be sent over the wire. So, for a 5 second segment you would need to encode the entire 5 seconds before the first byte can be sent; this does not apply when streaming the segments with chunked transfer encoding. Incidentally, at Mux we also use chunked transfer-encoding to stream video that is encoded on-the-fly with great performance.

reggieband 9y ago
I've heard from colleagues that this won't be possible with DASH due to the switch to fMP4 format. One of my co-workers tells me that fMP4 requires the entire segment to be loaded before playback can begin while TS segments don't require this. We've been looking into very small segments (e.g. 1s duration) to reduce latency but I've been interested in the LHLS approach since I first heard of it.
- TD-Linux 9y ago
  fMP4 has an index for each chunk, so you have to buffer the whole thing to create the index on the writing side. However, with DASH you do have the option for WebM as well, which does not need and index and can be streamed. Or really small fMP4 segments work too.
- urlgrey 9y ago
  Very short segment durations are effective only when latency is more important than quality. Each TS segment must start with a key-frame, and the GOP size can't exceed the duration of a segment (e.g. one second). Lowering the segment duration increases the frequency of key-frames, which has the effect of lowering the quality you can achieve at a given bitrate.

rjeli 9y ago

Anyone know what PSNow/ cloud gaming services use? I’m using it to play some ps3 games and the delay is immediately noticeable when you start playing, but your brain adjusts and it’s not noticeable. Has to be around 300-400 ms round trip