6 ms·

Salsify – A New Architecture for Real-time Internet Video

250 points by dedalus 7y ago

moltar 7y ago
Middle out
- giancarlostoro 7y ago
  To this day I would love to see some Pied Piper type of breakthrough.
  - Mathnerd314 7y ago
    https://blogs.dropbox.com/tech/2016/07/lepton-image-compression-saving-22-losslessly-from-images-at-15mbs/ https://blogs.dropbox.com/tech/2016/07/lepton-image-compress... Not really a breakthrough though, JPEG is just an old format. But maybe machine learning stuff will get the 4x video compression seen in the show.
xt00 7y ago
If this somehow fixes the situations where I have to click pause and un-pause to get a video to actually play, that’d be a step forward for humanity...
- schlipity 7y ago
  I think this is caused by your browser not allowing auto play video, and the web site not accounting for or checking for this state. So by toggling the pause / play button, you get the video control in a state that finally mirrors the browser's state.
  - xt00 7y ago
    I was thinking more of the case where the video was playing fine then it starts buffering and pauses then will sit there indefinitely after buffering unless I intervene by clicking pause unpause etc. Basically caused by an imperfect connection. I think like myself many people have thought, “oh if I just wait it’ll start playing again”.. but I suspect many of the people who make the video sites don’t want each of the client pages to go into a polling loop that spams their servers so they sort of have a back off scheme where if your internet link is not perfect it progressively gets worse and worse and then just sort of sits in slow retry loop. So the user hitting a button basically interrupts the slow retry loop and effectively commands the player instance to try streaming again.
ralphm 7y ago
Earlier: https://news.ycombinator.com/item?id=16964112 https://news.ycombinator.com/item?id=16964112 https://news.ycombinator.com/item?id=16802079 https://news.ycombinator.com/item?id=16802079
sansnomme 7y ago
Anyone have suggestions what protocol should be used for in-browser streaming cloud gaming?
- dochtman 7y ago
  WebTransport (https://wicg.github.io/web-transport/ https://wicg.github.io/web-transport/) is looking interesting for the mid-term future perhaps. In any case, building on QUIC is probably a good idea.
- Mathnerd314 7y ago
  https://parsecgaming.com/game-streaming-technology https://parsecgaming.com/game-streaming-technology looks interesting for an off-the-shelf type thing, but I guess Salsify would work too. As far as keystrokes I guess you would just use TCP or one of its improved variants.
dillonmckay 7y ago
So, how is this better or worse than RTSP and/or h.265?
- keithwinstein 7y ago
  It does a different thing. RTSP is mostly about control and framing. It doesn't specify any particular algorithm to estimate the network's capacity, how a video encoder should try to match that estimated capacity, or how to recover from lost packets. H.265 is a format for compressed video that defines a bit-exact decoder. It doesn't specify the way that an encoder encodes anything into the compressed format, how the encoder should try to match an externally supplied target frame size / bitrate target (while also meeting a latency target), or what the API should be. The Salsify techniques could work fine with RTSP, and could work fine using H.265 as the coded video format. The special thing about Salsify is really about where the control lies. Traditionally (in Skype, FaceTime, or the WebRTC.org codebase), there is a drop-in codec with its own control loop (making frame-by-frame decisions), and a congestion-control protocol with its own control loop (making packet-by-packet decisions), and these control loops are at close enough timescales that they end up doing poorly together. And the API to the codec is generally too limited (especially when it's a very general API, as in WebRTC.org, that tries to abstract across pre-existing implementations of VP9/H.264/H.265 to give the application agility across different formats) to achieve the kind of rapid adaptation to network flakiness that you need over these cellular or bad Wi-Fi networks. Salsify basically says, "hey, if your codec supports a functional-style API, and your transport protocol too, and you can extract the long-lived control state from each of those individual modules and just have one control loop that jointly controls both the codec module and the transport module, you can do a heck of a lot better."
  - dillonmckay 7y ago
    Thank you for clarifying that for me.
swami26 7y ago
is this worse or better than zoom?
- edoceo 7y ago
  Different. This is a codec and protocol. Zoom is an app.
  - ec109685 7y ago
    I think they mean is this better than what Zoom does.
- keithwinstein 7y ago
  Well, we've never installed a rootkit on anybody's computer, so at least we've got that going for us... More to the point, we don't have an empirical end-to-end measurement of Zoom the way we do for Skype, FaceTime, Google Hangouts, and WebRTC-in-Chrome (with and without VP9-SVC). But my understanding is that Zoom is architected similarly to those systems and can be expected to behave within the same envelope. Would need to measure it to know for sure. On the other hand, Zoom is an actual product with users, and Salsify is a research prototype that doesn't even have audio, much less users. So hard to compare outside the narrow technical questions of video quality and latency over imperfect networks.
  - ianlevesque 7y ago
    These might be the most patient and detailed replies to low effort questions I’ve ever seen.
keithwinstein 7y ago
Hi all -- Salsify co-author here. Surprised to see us here again, but happy to be part of the conversation (here's a previous one: https://news.ycombinator.com/item?id=16964112 https://news.ycombinator.com/item?id=16964112). This work was led by my student Sadjad Fouladi. If you liked Salsify, you might really like Sadjad's talk in June at USENIX ATC about "gg", his system for letting people use AWS Lambda as a rented supercomputer (e.g. he can compile inkscape and Chromium really fast by outsourcing the computation to 8,000 Lambda nodes that all talk to each other directly over UDP): https://www.youtube.com/watch?v=Cc_MVldSijA https://www.youtube.com/watch?v=Cc_MVldSijA (code here: https://github.com/StanfordSNR/gg https://github.com/StanfordSNR/gg) You might also be interested in our current video project, led by my student Francis Yan, on trying to improve live video streaming. If you visit and watch some live TV you can help contribute to our study: https://puffer.stanford.edu https://puffer.stanford.edu
- EGreg 7y ago
  Any chance you can get this added into WebRTC of the next browser iterations? Submit patches? Is it patented?
- microcolonel 7y ago
  Without going in to too much detail, how much would need to change about conventional, deployed VP9/H.264 + WebRTC systems to perform like your system? It seems to me that the vast majority of what these systems do is not in the way of your goals, so I kinda wonder why it's billed as something completely separate rather than something incremental. AFAIK codecs exposing rate control functionality to applications is not new, and most of WebRTC is functionally equivalent to any protocol of its sort; so before I go reading your paper it'd be nice to know the reasoning behind the seeming ground-up approach.
  - keithwinstein 7y ago
    Realistically it would probably be a major refactor to adapt the WebRTC.org codebase to a Salsify-style design. The WebRTC.org codebase is about a million lines of code (about half of which is vendored third-party code, e.g. libvpx) so any serious refactor is probably out of the realistic capabilities of a university-based research group. For comparison -- the entire Salsify codebase is about 16,000 lines of C++ (including the codec), plus about 7,000 lines of assembly we took from libvpx. It does a heck of a lot less than WebRTC.org, but it's a lot easier to prototype new designs that way. Salsify is mainly about the benefits you can get if you (a) extract the control loop out of the video codec, and make it expose a functional-style API, (b) use a Sprout-like congestion control algorithm that tries to follow evolving network capacity quickly and estimate "how many bytes can we send right now while trying to maintain a bound on end-to-end delay", and (c) have a single control loop that works every frame and has the choice of whether to send (1) a frame whose coded length is already known and is about equal to what we think the network can handle [it is very hard to get this from any existing video encoder on a single pass!], or (2) no frame at all. You can certainly do this within the WebRTC protocol, but doing it within the WebRTC.org codebase is going to be a lot of work. :-( I think unfortunately the codebase has some pretty deeply ingrained assumptions about how the control flow is going to go. Inverting that (as we propose), I don't think is an easy incremental change. Now, you might ask whether there are some incremental improvements you could make to WebRTC.org to get 80% of the benefits of Salsify without a major refactor. E.g., maybe you don't need the functional API if you just make some tweaks to the rate-control algorithm and congestion control. I don't think we know for sure though, but I suspect that for somebody already experienced with that codebase, probably yes, there are gains to be had. There's another question about whether the gains of Salsify (which are on a particular type of flaky cellular network) might come at the expense of costs on other types of (dependable?) networks. To be confident on that we'd probably need to try this stuff much more broadly and on real people (this is the motivation for Puffer).
summm 7y ago
TL;DR: they created a VP8 implementation that is purely functional and exploited that property to integrate control loops of codec and transport protocol. Sounds promising.
acd 7y ago
A little bit of latency comes from bufferbloat where network device buffers cause latency to increase. https://www.bufferbloat.net https://www.bufferbloat.net
j1elo 7y ago
REMB (actually the correct name would be GCC) is the de-facto rate control and bandwith congestion estimation algorithm that's used by WebRTC (I have some short notes with links to the algorithms that were proposed, in [0]), and it's a technology that for all practical effects has been kind of abandoned since 2014. So it's only natural that it should be possible to greatly improve upon what we're using today. I'm glad to see advances in this field, because it's sad that video calls are still a hugely worse experience compared with good old phone calls... [0]: https://doc-kurento.readthedocs.io/en/latest/knowledge/congestion_rmcat.html https://doc-kurento.readthedocs.io/en/latest/knowledge/conge...
vdnkh 7y ago
From what I understand, Salsify requires a unique connection for each device being served a stream. Therefore (like WebRTC) you can't cache frames in a CDN; and each viewer requires a connection to the origin. This gets a little expensive (which is one of the reasons why WebRTC isn't very common relative to HTTP streaming). Have you explored any kind of multicast or cache-friendly variation of Salsify?
jasonhansel 7y ago
I'm surprised that sites that serve videos--like YouTube and Netflix--haven't already come up with this idea, of using a codec optimized for transmission over the network. I wonder if we can do this with WebSockets? (The decoder might need to be in WASM for performance to be worthwhile.)
- labawi 7y ago
  I believe the main improvement here is latency, which in a video streaming context is a non-issue.
Expez 7y ago
Is anyone working on putting these ideas into a working product? I'm taking guitar lessons over Skype and for the most part it works just fine. However, every once in a while we have to resort to recording and sending snippets over the wire to get the required fidelity. With digital amp simulators this is thankfully a trivial exercise, but it would be great to do without. I've looked for other alternatives, but couldn't really find any that fit the bill. The last thing I looked at was a few Audio over IP products, but all of them were designed to be run on a LAN.
tr33house 7y ago
Glad to see Sprout-related work going on Keith + Sadjad! I remember being amazed by sprout a few years ago.

250 points by dedalus 7y ago

moltar 7y ago

Middle out

giancarlostoro 7y ago
To this day I would love to see some Pied Piper type of breakthrough.
- Mathnerd314 7y ago
  https://blogs.dropbox.com/tech/2016/07/lepton-image-compression-saving-22-losslessly-from-images-at-15mbs/ https://blogs.dropbox.com/tech/2016/07/lepton-image-compress... Not really a breakthrough though, JPEG is just an old format. But maybe machine learning stuff will get the 4x video compression seen in the show.

xt00 7y ago

If this somehow fixes the situations where I have to click pause and un-pause to get a video to actually play, that’d be a step forward for humanity...

schlipity 7y ago
I think this is caused by your browser not allowing auto play video, and the web site not accounting for or checking for this state. So by toggling the pause / play button, you get the video control in a state that finally mirrors the browser's state.
- xt00 7y ago
  I was thinking more of the case where the video was playing fine then it starts buffering and pauses then will sit there indefinitely after buffering unless I intervene by clicking pause unpause etc. Basically caused by an imperfect connection. I think like myself many people have thought, “oh if I just wait it’ll start playing again”.. but I suspect many of the people who make the video sites don’t want each of the client pages to go into a polling loop that spams their servers so they sort of have a back off scheme where if your internet link is not perfect it progressively gets worse and worse and then just sort of sits in slow retry loop. So the user hitting a button basically interrupts the slow retry loop and effectively commands the player instance to try streaming again.

ralphm 7y ago

Earlier: https://news.ycombinator.com/item?id=16964112 https://news.ycombinator.com/item?id=16964112 https://news.ycombinator.com/item?id=16802079 https://news.ycombinator.com/item?id=16802079

sansnomme 7y ago

Anyone have suggestions what protocol should be used for in-browser streaming cloud gaming?

dochtman 7y ago
WebTransport (https://wicg.github.io/web-transport/ https://wicg.github.io/web-transport/) is looking interesting for the mid-term future perhaps. In any case, building on QUIC is probably a good idea.
Mathnerd314 7y ago
https://parsecgaming.com/game-streaming-technology https://parsecgaming.com/game-streaming-technology looks interesting for an off-the-shelf type thing, but I guess Salsify would work too. As far as keystrokes I guess you would just use TCP or one of its improved variants.

dillonmckay 7y ago

So, how is this better or worse than RTSP and/or h.265?

keithwinstein 7y ago
It does a different thing. RTSP is mostly about control and framing. It doesn't specify any particular algorithm to estimate the network's capacity, how a video encoder should try to match that estimated capacity, or how to recover from lost packets. H.265 is a format for compressed video that defines a bit-exact decoder. It doesn't specify the way that an encoder encodes anything into the compressed format, how the encoder should try to match an externally supplied target frame size / bitrate target (while also meeting a latency target), or what the API should be. The Salsify techniques could work fine with RTSP, and could work fine using H.265 as the coded video format. The special thing about Salsify is really about where the control lies. Traditionally (in Skype, FaceTime, or the WebRTC.org codebase), there is a drop-in codec with its own control loop (making frame-by-frame decisions), and a congestion-control protocol with its own control loop (making packet-by-packet decisions), and these control loops are at close enough timescales that they end up doing poorly together. And the API to the codec is generally too limited (especially when it's a very general API, as in WebRTC.org, that tries to abstract across pre-existing implementations of VP9/H.264/H.265 to give the application agility across different formats) to achieve the kind of rapid adaptation to network flakiness that you need over these cellular or bad Wi-Fi networks. Salsify basically says, "hey, if your codec supports a functional-style API, and your transport protocol too, and you can extract the long-lived control state from each of those individual modules and just have one control loop that jointly controls both the codec module and the transport module, you can do a heck of a lot better."
- dillonmckay 7y ago
  Thank you for clarifying that for me.

swami26 7y ago

is this worse or better than zoom?

edoceo 7y ago
Different. This is a codec and protocol. Zoom is an app.
- ec109685 7y ago
  I think they mean is this better than what Zoom does.
keithwinstein 7y ago
Well, we've never installed a rootkit on anybody's computer, so at least we've got that going for us... More to the point, we don't have an empirical end-to-end measurement of Zoom the way we do for Skype, FaceTime, Google Hangouts, and WebRTC-in-Chrome (with and without VP9-SVC). But my understanding is that Zoom is architected similarly to those systems and can be expected to behave within the same envelope. Would need to measure it to know for sure. On the other hand, Zoom is an actual product with users, and Salsify is a research prototype that doesn't even have audio, much less users. So hard to compare outside the narrow technical questions of video quality and latency over imperfect networks.
- ianlevesque 7y ago
  These might be the most patient and detailed replies to low effort questions I’ve ever seen.

keithwinstein 7y ago

Hi all -- Salsify co-author here. Surprised to see us here again, but happy to be part of the conversation (here's a previous one: https://news.ycombinator.com/item?id=16964112 https://news.ycombinator.com/item?id=16964112). This work was led by my student Sadjad Fouladi. If you liked Salsify, you might really like Sadjad's talk in June at USENIX ATC about "gg", his system for letting people use AWS Lambda as a rented supercomputer (e.g. he can compile inkscape and Chromium really fast by outsourcing the computation to 8,000 Lambda nodes that all talk to each other directly over UDP): https://www.youtube.com/watch?v=Cc_MVldSijA https://www.youtube.com/watch?v=Cc_MVldSijA (code here: https://github.com/StanfordSNR/gg https://github.com/StanfordSNR/gg) You might also be interested in our current video project, led by my student Francis Yan, on trying to improve live video streaming. If you visit and watch some live TV you can help contribute to our study: https://puffer.stanford.edu https://puffer.stanford.edu

EGreg 7y ago
Any chance you can get this added into WebRTC of the next browser iterations? Submit patches? Is it patented?
microcolonel 7y ago
Without going in to too much detail, how much would need to change about conventional, deployed VP9/H.264 + WebRTC systems to perform like your system? It seems to me that the vast majority of what these systems do is not in the way of your goals, so I kinda wonder why it's billed as something completely separate rather than something incremental. AFAIK codecs exposing rate control functionality to applications is not new, and most of WebRTC is functionally equivalent to any protocol of its sort; so before I go reading your paper it'd be nice to know the reasoning behind the seeming ground-up approach.
- keithwinstein 7y ago
  Realistically it would probably be a major refactor to adapt the WebRTC.org codebase to a Salsify-style design. The WebRTC.org codebase is about a million lines of code (about half of which is vendored third-party code, e.g. libvpx) so any serious refactor is probably out of the realistic capabilities of a university-based research group. For comparison -- the entire Salsify codebase is about 16,000 lines of C++ (including the codec), plus about 7,000 lines of assembly we took from libvpx. It does a heck of a lot less than WebRTC.org, but it's a lot easier to prototype new designs that way. Salsify is mainly about the benefits you can get if you (a) extract the control loop out of the video codec, and make it expose a functional-style API, (b) use a Sprout-like congestion control algorithm that tries to follow evolving network capacity quickly and estimate "how many bytes can we send right now while trying to maintain a bound on end-to-end delay", and (c) have a single control loop that works every frame and has the choice of whether to send (1) a frame whose coded length is already known and is about equal to what we think the network can handle [it is very hard to get this from any existing video encoder on a single pass!], or (2) no frame at all. You can certainly do this within the WebRTC protocol, but doing it within the WebRTC.org codebase is going to be a lot of work. :-( I think unfortunately the codebase has some pretty deeply ingrained assumptions about how the control flow is going to go. Inverting that (as we propose), I don't think is an easy incremental change. Now, you might ask whether there are some incremental improvements you could make to WebRTC.org to get 80% of the benefits of Salsify without a major refactor. E.g., maybe you don't need the functional API if you just make some tweaks to the rate-control algorithm and congestion control. I don't think we know for sure though, but I suspect that for somebody already experienced with that codebase, probably yes, there are gains to be had. There's another question about whether the gains of Salsify (which are on a particular type of flaky cellular network) might come at the expense of costs on other types of (dependable?) networks. To be confident on that we'd probably need to try this stuff much more broadly and on real people (this is the motivation for Puffer).

summm 7y ago

TL;DR: they created a VP8 implementation that is purely functional and exploited that property to integrate control loops of codec and transport protocol. Sounds promising.

acd 7y ago

A little bit of latency comes from bufferbloat where network device buffers cause latency to increase. https://www.bufferbloat.net https://www.bufferbloat.net

j1elo 7y ago

REMB (actually the correct name would be GCC) is the de-facto rate control and bandwith congestion estimation algorithm that's used by WebRTC (I have some short notes with links to the algorithms that were proposed, in [0]), and it's a technology that for all practical effects has been kind of abandoned since 2014. So it's only natural that it should be possible to greatly improve upon what we're using today. I'm glad to see advances in this field, because it's sad that video calls are still a hugely worse experience compared with good old phone calls... [0]: https://doc-kurento.readthedocs.io/en/latest/knowledge/congestion_rmcat.html https://doc-kurento.readthedocs.io/en/latest/knowledge/conge...

vdnkh 7y ago

From what I understand, Salsify requires a unique connection for each device being served a stream. Therefore (like WebRTC) you can't cache frames in a CDN; and each viewer requires a connection to the origin. This gets a little expensive (which is one of the reasons why WebRTC isn't very common relative to HTTP streaming). Have you explored any kind of multicast or cache-friendly variation of Salsify?

jasonhansel 7y ago

I'm surprised that sites that serve videos--like YouTube and Netflix--haven't already come up with this idea, of using a codec optimized for transmission over the network. I wonder if we can do this with WebSockets? (The decoder might need to be in WASM for performance to be worthwhile.)

labawi 7y ago
I believe the main improvement here is latency, which in a video streaming context is a non-issue.

Expez 7y ago

Is anyone working on putting these ideas into a working product? I'm taking guitar lessons over Skype and for the most part it works just fine. However, every once in a while we have to resort to recording and sending snippets over the wire to get the required fidelity. With digital amp simulators this is thankfully a trivial exercise, but it would be great to do without. I've looked for other alternatives, but couldn't really find any that fit the bill. The last thing I looked at was a few Audio over IP products, but all of them were designed to be run on a LAN.

tr33house 7y ago

Glad to see Sprout-related work going on Keith + Sadjad! I remember being amazed by sprout a few years ago.