3 ms·

> I think the large streamers are a loss leader for Twitch. Delivering thousands of extremely high resolution, high frame rate, high bit rate streams costs a fo

> I think the large streamers are a loss leader for Twitch. Delivering thousands of extremely high resolution, high frame rate, high bit rate streams costs a fortune It's the best use-case for webRTC-based delivery though. They could be saving ~90-95%[1] of their bandwidth cost on these use-cases, I find it very intriguing that they haven't adopted it after all that time. [1] actual production number when I was working on that topic.

pjc50 3y ago
How does changing the protocol save 95% of the bandwidth? It's RTSP at the moment, isn't it? You can't be suggesting that the source streamer sends out millions of webRTC streams from their house, can you?
- BiteCode_dev 3y ago
  No but bittorrent scales as the number of user increases. P2P may be explored again.
  - fidotron 3y ago
    Spitballing here . . . The unreliability of such a network for live video would make routing quite a challenge. Assuming that could be solved Twitch could offset this by reducing the number of ads shown to people acting as redistribution nodes.
    - BiteCode_dev 3y ago
      There are a things you can try to see if this solves the problem: - don't do full P2P, keep twitch as a central server, and delegate only part of the load to P2P to those with good connection, with progressive roll in as buffering grows - get a 2 minutes padding between the live and actual video streaming, so that you have a decent buffer and margin of maneuver to switch back to twitch server - create an OBS clone that does streaming and video client with less ads as you said, but that act as an optimized node that works even when you don't what a video (the more you accumulate up time, the less you see adds, you get points, etc)
      - fidotron 3y ago
        2 minutes latency is a non starter, especially for broadcasts with some level of viewer interaction.
        littlestymaar 3y ago
        Good news is that you don't need that much, something like 4 video segments is more than enough. (Edit: I love how I get downvoted when saying that although I literally built a p2p streaming software and I'm citing actual configuration requirements we had in production back then)
    - littlestymaar 3y ago
      It's not “assuming that could be solved” it has existed for years. Lumen (formerly CenturyLink) has had a commercial offering[1] of the tech for years (since they acquired my former employer actually). [1] https://www.lumen.com/help/en-us/cdn/mesh-knowledge-base/faq.html https://www.lumen.com/help/en-us/cdn/mesh-knowledge-base/faq...
      - fidotron 3y ago
        What kind of latency did that approach manage to achieve? From your description elsewhere was it using the WebRTC data channels to deliver DASH/HLS style segments or actually using the MediaStream stuff? I think the magic info we're missing here is what the Twitch viewers are watching on, and my hunch is it's a lot of mobile for which the P2P idea will not work (and/or provoke angry messages from mobile operators). In the case of smart TV then I can see the P2P idea saving a lot of money, but I don't believe there is even an official Twitch app for Samsung TVs so they don't seem to think that is where their audience is.
        littlestymaar 3y ago
        > What kind of latency did that approach manage to achieve? Generally 10-20s depending on the size of the video segments. For streams we controlled entirely, we could even go as low as 5s (at the expense of short video segments). > From your description elsewhere was it using the WebRTC data channels to deliver DASH/HLS style segments or actually using the MediaStream stuff? The former, but we'd split the segments in much smaller chunks for delivery purpose. > I think the magic info we're missing here is what the Twitch viewers are watching on, and my hunch is it's a lot of mobile for which the P2P idea will not work (and/or provoke angry messages from mobile operators). Yeah mobile is more complex than desktop because you must not incur spendings on the viewer side and because battery use is paramount. But you can still get good result by setting the mobile users as leaf node (receiver-only). > In the case of smart TV then I can see the P2P idea saving a lot of money Yeah, smart-TV and set top boxes were a very important segment for us. But as you said it's probably not Twitch's market.
        3y ago
- littlestymaar 3y ago
  Why do you assume your interlocutor is stupid? It's not about just switching protocol nor it is about the source steamer (who should never be exposed, for DDoS/doxing reasons). But instead of delivering the video from the CDN to all users directly, you build a p2p network of viewers and have non-leaf viewers rebroadcast the video segments they've received (either directly from the CDN or from another peer). It's obviously not as trivial as the description above implies, but it's not very hard either and for live streaming with lots of users it works very well. (I used to work on such a system and deployed it in production for streams with up to 300k concurrent viewers)
- etc-hosts 3y ago
  You can use one protocol for the broadcaster to stream to the provider (Twitch). You can use the same protocol, or a different one, to deliver video to viewers. If you use HLS to deliver video to viewers, you can cache segments in a CDN. Your CDN. Someone else's CDN. You'll add some latency, but your bill will be lower.
Bayart 3y ago
You'd get into latency and consistency issues. It seems a poor choice if your main design constraint is real-time or near real-time interactive content delivery.
- littlestymaar 3y ago
  Consistency is a non issue (it's simply doesn't exist), latency could be if your constraint is to be very low latency (less than 5 seconds) but Twitch used to have low-latency opt-in until 2019 and had something like 15s latency at that time while already being an enormous success so I'm not certain how important the factor is, especially for the bigger streams where interactivity is lower than in the smaller ones. Maybe Twitch's management effectively prefers 3s of latency at 20x the bandwidth cost of 7 seconds of latency, but I have to admit that it would surprise me.
  - chongli 3y ago
    Maybe Twitch's management effectively prefers 3s of latency at 20x the bandwidth cost of 7 seconds of latency, but I have to admit that it would surprise me. Twitch is an interactive medium, not merely a video stream delivery platform. Streamers chat with their viewers and play games with them. They run merchandise giveaways based on guessing games. Low latency is critical for these applications. If a stream viewer sees a ton of people guessing random words or numbers in chat before they even see the streamer mention what everyone’s guessing at, they’re going to have a bad experience, leading to disengagement. Twitch goes out of their way to lower the latency. They even offer options in the client to switch to a lower latency mode, at the expense of stability.
    - littlestymaar 3y ago
      I was a bit confused when reading your comment at first because I was a big Twitch user at a time when they had around 10-15s latency for everyone without feeling that this latency was a problem. But then I eventually understood that you're talking about latency between viewers and then indeed it would be a very frustrating experience. Fortunately it's not what I'm talking about: I'm talking about having some latency (or rather “buffer size”) for all users, with synchronized playback between viewers. And again we're not talking about a prohibitive buffer size but rather a 5-15s buffer size, which is pretty common for live streaming services.

pjc50 3y ago

How does changing the protocol save 95% of the bandwidth? It's RTSP at the moment, isn't it? You can't be suggesting that the source streamer sends out millions of webRTC streams from their house, can you?

BiteCode_dev 3y ago
No but bittorrent scales as the number of user increases. P2P may be explored again.
- fidotron 3y ago
  Spitballing here . . . The unreliability of such a network for live video would make routing quite a challenge. Assuming that could be solved Twitch could offset this by reducing the number of ads shown to people acting as redistribution nodes.
  - BiteCode_dev 3y ago
    There are a things you can try to see if this solves the problem: - don't do full P2P, keep twitch as a central server, and delegate only part of the load to P2P to those with good connection, with progressive roll in as buffering grows - get a 2 minutes padding between the live and actual video streaming, so that you have a decent buffer and margin of maneuver to switch back to twitch server - create an OBS clone that does streaming and video client with less ads as you said, but that act as an optimized node that works even when you don't what a video (the more you accumulate up time, the less you see adds, you get points, etc)
    - fidotron 3y ago
      2 minutes latency is a non starter, especially for broadcasts with some level of viewer interaction.
      - littlestymaar 3y ago
        Good news is that you don't need that much, something like 4 video segments is more than enough. (Edit: I love how I get downvoted when saying that although I literally built a p2p streaming software and I'm citing actual configuration requirements we had in production back then)
  - littlestymaar 3y ago
    It's not “assuming that could be solved” it has existed for years. Lumen (formerly CenturyLink) has had a commercial offering[1] of the tech for years (since they acquired my former employer actually). [1] https://www.lumen.com/help/en-us/cdn/mesh-knowledge-base/faq.html https://www.lumen.com/help/en-us/cdn/mesh-knowledge-base/faq...
    - fidotron 3y ago
      What kind of latency did that approach manage to achieve? From your description elsewhere was it using the WebRTC data channels to deliver DASH/HLS style segments or actually using the MediaStream stuff? I think the magic info we're missing here is what the Twitch viewers are watching on, and my hunch is it's a lot of mobile for which the P2P idea will not work (and/or provoke angry messages from mobile operators). In the case of smart TV then I can see the P2P idea saving a lot of money, but I don't believe there is even an official Twitch app for Samsung TVs so they don't seem to think that is where their audience is.
      - littlestymaar 3y ago
        > What kind of latency did that approach manage to achieve? Generally 10-20s depending on the size of the video segments. For streams we controlled entirely, we could even go as low as 5s (at the expense of short video segments). > From your description elsewhere was it using the WebRTC data channels to deliver DASH/HLS style segments or actually using the MediaStream stuff? The former, but we'd split the segments in much smaller chunks for delivery purpose. > I think the magic info we're missing here is what the Twitch viewers are watching on, and my hunch is it's a lot of mobile for which the P2P idea will not work (and/or provoke angry messages from mobile operators). Yeah mobile is more complex than desktop because you must not incur spendings on the viewer side and because battery use is paramount. But you can still get good result by setting the mobile users as leaf node (receiver-only). > In the case of smart TV then I can see the P2P idea saving a lot of money Yeah, smart-TV and set top boxes were a very important segment for us. But as you said it's probably not Twitch's market.
        3y ago
littlestymaar 3y ago
Why do you assume your interlocutor is stupid? It's not about just switching protocol nor it is about the source steamer (who should never be exposed, for DDoS/doxing reasons). But instead of delivering the video from the CDN to all users directly, you build a p2p network of viewers and have non-leaf viewers rebroadcast the video segments they've received (either directly from the CDN or from another peer). It's obviously not as trivial as the description above implies, but it's not very hard either and for live streaming with lots of users it works very well. (I used to work on such a system and deployed it in production for streams with up to 300k concurrent viewers)
etc-hosts 3y ago
You can use one protocol for the broadcaster to stream to the provider (Twitch). You can use the same protocol, or a different one, to deliver video to viewers. If you use HLS to deliver video to viewers, you can cache segments in a CDN. Your CDN. Someone else's CDN. You'll add some latency, but your bill will be lower.

Bayart 3y ago

You'd get into latency and consistency issues. It seems a poor choice if your main design constraint is real-time or near real-time interactive content delivery.

littlestymaar 3y ago
Consistency is a non issue (it's simply doesn't exist), latency could be if your constraint is to be very low latency (less than 5 seconds) but Twitch used to have low-latency opt-in until 2019 and had something like 15s latency at that time while already being an enormous success so I'm not certain how important the factor is, especially for the bigger streams where interactivity is lower than in the smaller ones. Maybe Twitch's management effectively prefers 3s of latency at 20x the bandwidth cost of 7 seconds of latency, but I have to admit that it would surprise me.
- chongli 3y ago
  Maybe Twitch's management effectively prefers 3s of latency at 20x the bandwidth cost of 7 seconds of latency, but I have to admit that it would surprise me. Twitch is an interactive medium, not merely a video stream delivery platform. Streamers chat with their viewers and play games with them. They run merchandise giveaways based on guessing games. Low latency is critical for these applications. If a stream viewer sees a ton of people guessing random words or numbers in chat before they even see the streamer mention what everyone’s guessing at, they’re going to have a bad experience, leading to disengagement. Twitch goes out of their way to lower the latency. They even offer options in the client to switch to a lower latency mode, at the expense of stability.
  - littlestymaar 3y ago
    I was a bit confused when reading your comment at first because I was a big Twitch user at a time when they had around 10-15s latency for everyone without feeling that this latency was a problem. But then I eventually understood that you're talking about latency between viewers and then indeed it would be a very frustrating experience. Fortunately it's not what I'm talking about: I'm talking about having some latency (or rather “buffer size”) for all users, with synchronized playback between viewers. And again we're not talking about a prohibitive buffer size but rather a 5-15s buffer size, which is pretty common for live streaming services.