5 ms·

Your work in this area has been phenomenal. Thank you! I use broadcast-box all the time.

echelon 1y ago
What sort of infrastructure do you need for scaling WebRTC multicast? Are we entering an era where you don't need Amazon's budget to host something like Twitch?
- Sean-Der 1y ago
  Yes we are :) When OBS merges the PR [0] things are going to get very interesting. Before you needed to run expensive transcoding jobs to be able to support heterogenous clients. Once we get Simulcast the only cost will be bandwidth. With Hetzner I am paying $1 a TB. With AV1 or H265 + Simulcast I am getting 4K for hundreds of users on just a single server. We will have some growing pains, but I am not giving up until I can make this accessible to everyone. [0] https://github.com/obsproject/obs-studio/pull/10885 https://github.com/obsproject/obs-studio/pull/10885
  - matt-p 1y ago
    I have found it's hard to get past ~18Gbps on commodity servers and ~90Gbps on high spec, carefully specced servers. I presume you find the same?
    - michaelt 1y ago
      A twitch 720p stream is only 4 Mbps. 1080p? 6-8 Mbps So if you've got ~18 Gbps of upload bandwidth you're ready for 10,000-20,000 viewers.
      - miki123211 1y ago
        With how ubiquitous gigabit symmetric is becoming, I wonder if you could even do P2P nowadays. Assuming a single gigabit symmetric connection could dedicate at most 100mb of upload bandwidth, you'd need one such viewer per each 25 viewers with a worse connection. This feels achievable. You'd have 1 server that broadcasts to at most 10K tier-1 viewers. Tier 2 viewers get 3 tier1 ips for failover, and pre-negotiate a connection with them, e.g. through STUn, to get sub-second failovers in case their primary source quits the stream. A central control plane load balances these. With something like 15s of buffer (acceptable for a gaming stream, not so much for sports, where your neighbors might be watching on satellite and cheer), this feels achievable.
        matt-p 1y ago
        In theory I think you're right, but you do need a really smart control plane. Just because a connection is fast doesn't mean it's not metered (let's say fast 5G, starlink, rural fiber) and so on. Are all of the issues P2P brings really worth it? I'd say this definitely opens up streaming from your desktop with 1 CPU core handling a SFU for say 100 viewers = 500Mb or from a $5/month VPS if you've not got the capacity at home. That's pretty awesome, for most people no need to use P2P.
        esseph 1y ago
        It's not nearly as "ubiquitous" as you may think
        messe 1y ago
        > With how ubiquitous gigabit symmetric is becoming, I wonder if you could even do P2P nowadays. CGNAT is going to make that a hassle.
        chaz6 1y ago
        Ideally any ISP resorting to CGN would be providing IPv6 support, but Tailscale shows that nat hole-punching can work well [1]. I'm not sure if that's feasible to implement in a web browser though. [1] https://tailscale.com/blog/how-nat-traversal-works https://tailscale.com/blog/how-nat-traversal-works
        messe 1y ago
        It can work okay, but still not perfectly. Before I asked my ISP for a static address, tailscale connections between my place and my partner's only managed to maintain a direct connection half the time. The other half of the time, they required a relay.
        naikrovek 1y ago
        yep. I'm still kind of shocked at how little ipv6 has been deployed. i'm so effing tired of NAT here and NAT there and NATs in between NATs. NAT complicates things for almost no reason now that ipv6 was released almost 30 years ago. Wait, what? 30 years? Now I'm VERY shocked at how little ipv6 has been deployed.
      - matt-p 1y ago
        Not sure I follow your maths there. If we assumed an average of 6Mb per stream that's 3000 streams, practically speaking a little lower. It's all relative I guess but it's not that high.
        ablob 1y ago
        Multicast enters the room
        esseph 1y ago
        Finally
        Evidlo 1y ago
        Does multicast actually work from a home network to another set of homes over the internet? I thought this traffic would just get dropped by one of the hops along the way if you tried it. https://networkengineering.stackexchange.com/questions/47994/is-multicast-on-the-public-internet-possible-and-if-yes-how https://networkengineering.stackexchange.com/questions/47994...
        matt-p 1y ago
        It won't even make it through your home "gateway", in practice it's usable at layer 2 only.
        ablob 1y ago
        I think it never took off and is not supported by many if not most routers. However, it is an efficient solution for sending live-stream data over the web. I think eventually the pressure to not send packets for each consumer but at most once per port will take over and force providers to implement this or a better solution.
        matt-p 1y ago
        You can't multicast over the internet.
  - bryancoxwell 1y ago
    I just want to say I am loving your enthusiasm
  - NetOpWibby 1y ago
    Thanks to a post on here a few months ago about Cloudflare R2, I finally felt motivated enough to work on a video platform idea I have. I still don’t understand ffmpeg but my transcoder works! Now I see this news?! Perfect timing. (:
  - naikrovek 1y ago
    > I am not giving up until I can make this accessible to everyone. I like this guy.

What sort of infrastructure do you need for scaling WebRTC multicast? Are we entering an era where you don't need Amazon's budget to host something like Twitch?

Sean-Der 1y ago
Yes we are :) When OBS merges the PR [0] things are going to get very interesting. Before you needed to run expensive transcoding jobs to be able to support heterogenous clients. Once we get Simulcast the only cost will be bandwidth. With Hetzner I am paying $1 a TB. With AV1 or H265 + Simulcast I am getting 4K for hundreds of users on just a single server. We will have some growing pains, but I am not giving up until I can make this accessible to everyone. [0] https://github.com/obsproject/obs-studio/pull/10885 https://github.com/obsproject/obs-studio/pull/10885
- matt-p 1y ago
  I have found it's hard to get past ~18Gbps on commodity servers and ~90Gbps on high spec, carefully specced servers. I presume you find the same?
  - michaelt 1y ago
    A twitch 720p stream is only 4 Mbps. 1080p? 6-8 Mbps So if you've got ~18 Gbps of upload bandwidth you're ready for 10,000-20,000 viewers.
    - miki123211 1y ago
      With how ubiquitous gigabit symmetric is becoming, I wonder if you could even do P2P nowadays. Assuming a single gigabit symmetric connection could dedicate at most 100mb of upload bandwidth, you'd need one such viewer per each 25 viewers with a worse connection. This feels achievable. You'd have 1 server that broadcasts to at most 10K tier-1 viewers. Tier 2 viewers get 3 tier1 ips for failover, and pre-negotiate a connection with them, e.g. through STUn, to get sub-second failovers in case their primary source quits the stream. A central control plane load balances these. With something like 15s of buffer (acceptable for a gaming stream, not so much for sports, where your neighbors might be watching on satellite and cheer), this feels achievable.
      - matt-p 1y ago
        In theory I think you're right, but you do need a really smart control plane. Just because a connection is fast doesn't mean it's not metered (let's say fast 5G, starlink, rural fiber) and so on. Are all of the issues P2P brings really worth it? I'd say this definitely opens up streaming from your desktop with 1 CPU core handling a SFU for say 100 viewers = 500Mb or from a $5/month VPS if you've not got the capacity at home. That's pretty awesome, for most people no need to use P2P.
      - esseph 1y ago
        It's not nearly as "ubiquitous" as you may think
      - messe 1y ago
        > With how ubiquitous gigabit symmetric is becoming, I wonder if you could even do P2P nowadays. CGNAT is going to make that a hassle.
        chaz6 1y ago
        Ideally any ISP resorting to CGN would be providing IPv6 support, but Tailscale shows that nat hole-punching can work well [1]. I'm not sure if that's feasible to implement in a web browser though. [1] https://tailscale.com/blog/how-nat-traversal-works https://tailscale.com/blog/how-nat-traversal-works
        messe 1y ago
        It can work okay, but still not perfectly. Before I asked my ISP for a static address, tailscale connections between my place and my partner's only managed to maintain a direct connection half the time. The other half of the time, they required a relay.
        naikrovek 1y ago
        yep. I'm still kind of shocked at how little ipv6 has been deployed. i'm so effing tired of NAT here and NAT there and NATs in between NATs. NAT complicates things for almost no reason now that ipv6 was released almost 30 years ago. Wait, what? 30 years? Now I'm VERY shocked at how little ipv6 has been deployed.
    - matt-p 1y ago
      Not sure I follow your maths there. If we assumed an average of 6Mb per stream that's 3000 streams, practically speaking a little lower. It's all relative I guess but it's not that high.
      - ablob 1y ago
        Multicast enters the room
        esseph 1y ago
        Finally
        Evidlo 1y ago
        Does multicast actually work from a home network to another set of homes over the internet? I thought this traffic would just get dropped by one of the hops along the way if you tried it. https://networkengineering.stackexchange.com/questions/47994/is-multicast-on-the-public-internet-possible-and-if-yes-how https://networkengineering.stackexchange.com/questions/47994...
        matt-p 1y ago
        It won't even make it through your home "gateway", in practice it's usable at layer 2 only.
        ablob 1y ago
        I think it never took off and is not supported by many if not most routers. However, it is an efficient solution for sending live-stream data over the web. I think eventually the pressure to not send packets for each consumer but at most once per port will take over and force providers to implement this or a better solution.
        matt-p 1y ago
        You can't multicast over the internet.
- bryancoxwell 1y ago
  I just want to say I am loving your enthusiasm
- NetOpWibby 1y ago
  Thanks to a post on here a few months ago about Cloudflare R2, I finally felt motivated enough to work on a video platform idea I have. I still don’t understand ffmpeg but my transcoder works! Now I see this news?! Perfect timing. (:
- naikrovek 1y ago
  > I am not giving up until I can make this accessible to everyone. I like this guy.