4 ms·

Author here, there's lots of interesting innovations happening in this space. If anyone knows of anything interesting I've missed, please let me know!

cagenut 8y ago
Is there anything more you can talk, write, or link to about "cdn" support for sub-200ms streaming. Like is there a "varnish for WebRTC" or something like it (really more of an ircd for video)? Such that someone could build out a live streaming platform that can be conversationally interactive and yet not require a 1:1 back to origin or mesh connection setup.
- GeneticGenesis 8y ago
  As far as I'm aware there aren't any public CDNs that support the sub-200ms approaches which don't involve buying a full solution from that vendor. * Limelight's solution is only really accessible through their low latency streaming products provided by Red5. * Fastly and Akamai are both chasing Ultra low latency through chunked CMAF delivery. On a slightly smaller scale, Cloudflare supports websockets, so you could use a protocol like the one Wowza are using (WOWZ). https://blog.cloudflare.com/cloudflare-now-supports-websockets/ https://blog.cloudflare.com/cloudflare-now-supports-websocke... If you're interested in a toolkit to build out a webRTC style CDN edge, I think the best place to start would probably be with Gstreamer's new WebRTC tooling https://opensource.com/article/19/1/gstreamer https://opensource.com/article/19/1/gstreamer
  - newman314 8y ago
    IIRC BitGravity had a focus on the streaming video CDN space when I looked a few years ago. Not clear if they have any special sauce in this area.
- jkarneges 8y ago
  Pushpin could be considered a “varnish for streaming”, and it doesn’t require 1:1 connections with the origin. This may get you close to low latency multimedia streaming with the right tuning. There’s a live music example here: http://audiostream.fanoutapp.com http://audiostream.fanoutapp.com (code: https://github.com/fanout/audiostream https://github.com/fanout/audiostream). It plays a song in a loop using GStreamer.
kimburgess 8y ago
Nice summary! You touched on it briefly with real(ish) time uses for comms, but there’s another segment below this with even tighter latency constraints - in room signal transport. This is a world that’s migrated significantly towards to streaming based distribution over the past ~5 years. Gear in this space has to hit sub-frame latency (best in market at the moment is around 0.02 milliseconds), maintain extremely high signal quality (4K60 4:4:4 or 4:2:0) and provide perfect sync across all decode endpoints. Due to the these constraints it’s a world that lives exclusively in the dedicated hardware space. Some main players that are worth checking out are SDVoE and their associated implementers, Crestron NVX, SVSI, Atlona Omnistream and Lightware UBEX. Dante have also just released their new board that handles all the sync and transmission but let’s inplementers pick there codec of choice too which should be interesting. Most gear shipping at the moment uses either JPEG2000 or VC-2 based codecs or hand wavey “proprietary” ones that chip vendors refuse to give info on. There some interesting work being done in the form of the JPEG XS codec too. It provides low latency, but also super low quality loss over multiple encode / decode steps too. Given some of the engineering constraints, it’s a super interesting area to watch.
- Heff 8y ago
  Kieran Kunhya gave a good talk on related things at Demuxed https://www.youtube.com/watch?v=z1R3QlaaaUA https://www.youtube.com/watch?v=z1R3QlaaaUA
deleted 8y ago
[deleted]
keithwinstein 8y ago
One (emerging) area is ABR video delivered over a WebSocket. Throughput estimation can be done using server-side TCP statistics that are updated at each ACK segment (the tcp_info structure includes a delivery_rate member that is pretty helpful), which is more reliable than using client-side info that comes from the reconstructed reliable byte stream. The MPEG-DASH part 6 spec starts going in this direction, and we tried to run with these ideas in Puffer (https://puffer.stanford.edu https://puffer.stanford.edu). Empirically you can get quick channel changes and better first-chunk quality with this approach; we haven't tried to minimize end-to-end latency below standard levels. And there's probably nothing fundamental that you can do with a WebSocket that you can't do with a sufficiently smart chunked HTTP response. For really low latency, you want to couple the video codec parameters with the transport's capacity estimate in a way that the WebRTC.org/Chromium codebase is not really capable of doing. See https://snr.stanford.edu/salsify https://snr.stanford.edu/salsify .
- Game_Ender 8y ago
  What kind of performance did you get client side? A problem I see in the web space is that fastest encoder you can get is locked inside the browser so you need something in JS or WebAssembly.
  - keithwinstein 8y ago
    Not sure I quite follow -- the video goes through the same pipeline that it would with conventional DASH (MSE to a video element to whatever decoder the browser provides), and the performance is basically the same. You're welcome to try it! https://puffer.stanford.edu https://puffer.stanford.edu
wardbradt 8y ago
Nice article. High frequency trading is probably one of the most advanced fields in low latency live streaming due to the significant impact latency, which in some cases is measured in nanoseconds, has on profits. One standard in HFT is the FAST Protocol[1], an adaptation of the FIX Protocol. One of FIX's newer developments is Simple Binary Encoding[2], which is designed for minimal latency. There's many differences between FIX and "normal" protocols that are really interesting when you realize how much thought went into shaving off as much latency as possible with FIX. Unfortunately, FIX is one of the few solutions in HFT that is publicly available as many firms make money from being the fastest in a particular area. I doubt many HFT developments could be adapted for use in video streaming, as bandwidth is rarely an issue and hardware can cost much more. However, I would not be surprised if at some point in the future an HFT firm creates a general solution applicable for video. [1] https://www.fixtrading.org/standards/fast/ https://www.fixtrading.org/standards/fast/ [2] https://en.wikipedia.org/wiki/Financial_Information_eXchange#Simple_Binary_Encoding_(SBE) https://en.wikipedia.org/wiki/Financial_Information_eXchange... Edit: Grammar
- GeneticGenesis 8y ago
  I've always found the HFT space interesting, but don't really have any desire to move into the financial sector. How large are the payloads in general for HFT?
  - wardbradt 8y ago
    It depends on the application/ use case, but typically they are small. For a strategy my team is currently developing, each message contains 2-4 dynamic data points. Messages also include a minimal header[1] and "trailer" (checksum). The header includes data about body length, message type, etc. Messages in my use case are about 64 bytes, but my use case is on the lesser end in terms of payload size. I'd say a typical payload is ~64 to 128 bytes. It is probably important to mention that some firms communicate with exchanges/ brokers via a direct connection rather than the internet. [1] https://btobits.com/fixopaedia/fixdic44/block_Standard_Message_Header_.html https://btobits.com/fixopaedia/fixdic44/block_Standard_Messa...
samsonradu 8y ago
Thank you for the article. As I have used Nanocosmos H5Live Server in production for over a year I can provide some insight into their product. It works by repackaging an RTMP stream into GOP chunks and pushes them via Websockets to the browser where they're piped to a video player using MSE (Media Source Extension). On iPhones, where there is (was) no MSE available it used something like very short, chunked HLS segments. Not sure I fully understood that part but somehow they tricked the m3u8 playlist into loading short segments very very often. The beauty of it was that it was a drop-in plugin for your existing RTMP-based infrastructure. You just add an instance of their server software and you would use their JS player to your pull your RTMP stream, repackaged on-the-fly. let player = new NanoPlayer({ server: 'wss://nano_server_url', rtmp: 'rtmp://rtmp_server_url/app/stream.mp4' });
zacwebb 8y ago
Great article - just a small issue: It's standard to refer to esports as "esports" or "Esports" rather than "eSports". (see: https://www.dexerto.com/news/esports-esports-associated-press-puts-end-popular-debate/32233 https://www.dexerto.com/news/esports-esports-associated-pres...)
errantspark 8y ago
I'm really surprised to see no mention of FTL or Mixer!
mncharity 8y ago
Integrating codec and network stack permits higher quality and lower latency. Eg, https://snr.stanford.edu/salsify/ https://snr.stanford.edu/salsify/
mncharity 8y ago
Game streaming (running a game on a remote server) is notable for requiring low latency. And VR will be even worse (<20 ms).

cagenut 8y ago

Is there anything more you can talk, write, or link to about "cdn" support for sub-200ms streaming. Like is there a "varnish for WebRTC" or something like it (really more of an ircd for video)? Such that someone could build out a live streaming platform that can be conversationally interactive and yet not require a 1:1 back to origin or mesh connection setup.

GeneticGenesis 8y ago
As far as I'm aware there aren't any public CDNs that support the sub-200ms approaches which don't involve buying a full solution from that vendor. * Limelight's solution is only really accessible through their low latency streaming products provided by Red5. * Fastly and Akamai are both chasing Ultra low latency through chunked CMAF delivery. On a slightly smaller scale, Cloudflare supports websockets, so you could use a protocol like the one Wowza are using (WOWZ). https://blog.cloudflare.com/cloudflare-now-supports-websockets/ https://blog.cloudflare.com/cloudflare-now-supports-websocke... If you're interested in a toolkit to build out a webRTC style CDN edge, I think the best place to start would probably be with Gstreamer's new WebRTC tooling https://opensource.com/article/19/1/gstreamer https://opensource.com/article/19/1/gstreamer
- newman314 8y ago
  IIRC BitGravity had a focus on the streaming video CDN space when I looked a few years ago. Not clear if they have any special sauce in this area.
jkarneges 8y ago
Pushpin could be considered a “varnish for streaming”, and it doesn’t require 1:1 connections with the origin. This may get you close to low latency multimedia streaming with the right tuning. There’s a live music example here: http://audiostream.fanoutapp.com http://audiostream.fanoutapp.com (code: https://github.com/fanout/audiostream https://github.com/fanout/audiostream). It plays a song in a loop using GStreamer.

kimburgess 8y ago

Nice summary! You touched on it briefly with real(ish) time uses for comms, but there’s another segment below this with even tighter latency constraints - in room signal transport. This is a world that’s migrated significantly towards to streaming based distribution over the past ~5 years. Gear in this space has to hit sub-frame latency (best in market at the moment is around 0.02 milliseconds), maintain extremely high signal quality (4K60 4:4:4 or 4:2:0) and provide perfect sync across all decode endpoints. Due to the these constraints it’s a world that lives exclusively in the dedicated hardware space. Some main players that are worth checking out are SDVoE and their associated implementers, Crestron NVX, SVSI, Atlona Omnistream and Lightware UBEX. Dante have also just released their new board that handles all the sync and transmission but let’s inplementers pick there codec of choice too which should be interesting. Most gear shipping at the moment uses either JPEG2000 or VC-2 based codecs or hand wavey “proprietary” ones that chip vendors refuse to give info on. There some interesting work being done in the form of the JPEG XS codec too. It provides low latency, but also super low quality loss over multiple encode / decode steps too. Given some of the engineering constraints, it’s a super interesting area to watch.

Heff 8y ago
Kieran Kunhya gave a good talk on related things at Demuxed https://www.youtube.com/watch?v=z1R3QlaaaUA https://www.youtube.com/watch?v=z1R3QlaaaUA

deleted 8y ago

[deleted]

keithwinstein 8y ago

One (emerging) area is ABR video delivered over a WebSocket. Throughput estimation can be done using server-side TCP statistics that are updated at each ACK segment (the tcp_info structure includes a delivery_rate member that is pretty helpful), which is more reliable than using client-side info that comes from the reconstructed reliable byte stream. The MPEG-DASH part 6 spec starts going in this direction, and we tried to run with these ideas in Puffer (https://puffer.stanford.edu https://puffer.stanford.edu). Empirically you can get quick channel changes and better first-chunk quality with this approach; we haven't tried to minimize end-to-end latency below standard levels. And there's probably nothing fundamental that you can do with a WebSocket that you can't do with a sufficiently smart chunked HTTP response. For really low latency, you want to couple the video codec parameters with the transport's capacity estimate in a way that the WebRTC.org/Chromium codebase is not really capable of doing. See https://snr.stanford.edu/salsify https://snr.stanford.edu/salsify .

Game_Ender 8y ago
What kind of performance did you get client side? A problem I see in the web space is that fastest encoder you can get is locked inside the browser so you need something in JS or WebAssembly.
- keithwinstein 8y ago
  Not sure I quite follow -- the video goes through the same pipeline that it would with conventional DASH (MSE to a video element to whatever decoder the browser provides), and the performance is basically the same. You're welcome to try it! https://puffer.stanford.edu https://puffer.stanford.edu

wardbradt 8y ago

Nice article. High frequency trading is probably one of the most advanced fields in low latency live streaming due to the significant impact latency, which in some cases is measured in nanoseconds, has on profits. One standard in HFT is the FAST Protocol[1], an adaptation of the FIX Protocol. One of FIX's newer developments is Simple Binary Encoding[2], which is designed for minimal latency. There's many differences between FIX and "normal" protocols that are really interesting when you realize how much thought went into shaving off as much latency as possible with FIX. Unfortunately, FIX is one of the few solutions in HFT that is publicly available as many firms make money from being the fastest in a particular area. I doubt many HFT developments could be adapted for use in video streaming, as bandwidth is rarely an issue and hardware can cost much more. However, I would not be surprised if at some point in the future an HFT firm creates a general solution applicable for video. [1] https://www.fixtrading.org/standards/fast/ https://www.fixtrading.org/standards/fast/ [2] https://en.wikipedia.org/wiki/Financial_Information_eXchange#Simple_Binary_Encoding_(SBE) https://en.wikipedia.org/wiki/Financial_Information_eXchange... Edit: Grammar

GeneticGenesis 8y ago
I've always found the HFT space interesting, but don't really have any desire to move into the financial sector. How large are the payloads in general for HFT?
- wardbradt 8y ago
  It depends on the application/ use case, but typically they are small. For a strategy my team is currently developing, each message contains 2-4 dynamic data points. Messages also include a minimal header[1] and "trailer" (checksum). The header includes data about body length, message type, etc. Messages in my use case are about 64 bytes, but my use case is on the lesser end in terms of payload size. I'd say a typical payload is ~64 to 128 bytes. It is probably important to mention that some firms communicate with exchanges/ brokers via a direct connection rather than the internet. [1] https://btobits.com/fixopaedia/fixdic44/block_Standard_Message_Header_.html https://btobits.com/fixopaedia/fixdic44/block_Standard_Messa...

samsonradu 8y ago

Thank you for the article. As I have used Nanocosmos H5Live Server in production for over a year I can provide some insight into their product. It works by repackaging an RTMP stream into GOP chunks and pushes them via Websockets to the browser where they're piped to a video player using MSE (Media Source Extension). On iPhones, where there is (was) no MSE available it used something like very short, chunked HLS segments. Not sure I fully understood that part but somehow they tricked the m3u8 playlist into loading short segments very very often. The beauty of it was that it was a drop-in plugin for your existing RTMP-based infrastructure. You just add an instance of their server software and you would use their JS player to your pull your RTMP stream, repackaged on-the-fly. let player = new NanoPlayer({ server: 'wss://nano_server_url', rtmp: 'rtmp://rtmp_server_url/app/stream.mp4' });

zacwebb 8y ago

Great article - just a small issue: It's standard to refer to esports as "esports" or "Esports" rather than "eSports". (see: https://www.dexerto.com/news/esports-esports-associated-press-puts-end-popular-debate/32233 https://www.dexerto.com/news/esports-esports-associated-pres...)

errantspark 8y ago

I'm really surprised to see no mention of FTL or Mixer!

mncharity 8y ago

Integrating codec and network stack permits higher quality and lower latency. Eg, https://snr.stanford.edu/salsify/ https://snr.stanford.edu/salsify/

mncharity 8y ago

Game streaming (running a game on a remote server) is notable for requiring low latency. And VR will be even worse (<20 ms).