5 ms·

So the issue is that Netflix gets its performance from colocating caches of movies in ISP datacenters, and a live broadcast doesn't work with that. It's not jus

So the issue is that Netflix gets its performance from colocating caches of movies in ISP datacenters, and a live broadcast doesn't work with that. It's not just about the sheer numbers of viewers, it's that a live model totally undermines their entire infrastructure advantage. See: https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/

tnvmadhav 2y ago
they will learn :)
pinkmuffinere 2y ago
Damn that sucks. I wonder if they could have intentionally streamed it 5 min late? I don’t have all the context around the fight though — maybe a competing service would win if Netflix intentionally induced delay?
- adrr 2y ago
  they were introducing 5 minute delays on some of the clients. I noticed my ipad was always live and the smart tv had a 5 minute delay but you could fast forward to live.
deleted 2y ago
[deleted]
stingraycharles 2y ago
Correct, this is not Netflix’ regular cup of tea, and it’s a very different problem to solve. They can probably use their edge caches, but it’s challenging.
- nicce 2y ago
  How YouTube does this? Netflix is like drop in the ocean when compared to.
  - Bilal_io 2y ago
    Not sure how Netflix does it. But this is not very time sensitive, and I would have delayed the stream by 15 to 30 seconds to cache it and then deliver to everyone.
  - unsnap_biceps 2y ago
    My wild assed guess is the differences in the edge nodes. Netflix's edge nodes are optimized for streaming already encoded videos to end users. They have to transcode some number of formats from the source and send them all to the edge nodes to flow out. It's harder to manage a ton of different streams flowing out to the edge nodes cleanly. I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Only one stream from source to edge to worry about and is much simpler to support and reason about. But that's just my wild assed guess.
    - vitus 2y ago
      > I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Ha, no, our edge nodes don't have anywhere near enough spare CPU to do transcoding on the fly. We have our own issues with livestreaming, but our system's developed differently over the past 15 years compared to Netflix's. While they've historically focused on intelligent pre-placement of data (which of course doesn't work for livestreaming), such an approach was never feasible for YT with the sheer size of our catalog (thanks to user-generated content). Netflix is still new to the space, and there isn't a good substitute for real-world experience for understanding how your systems behave under wildly different traffic patterns. Give them some time.
      - kortilla 2y ago
        It also helps that youtube serves shit tier quality videos more gracefully. Everyone is used to the step down to pixel-world on youtube to the point where they don’t complain much.
sgarland 2y ago
I wonder how effective it would be to cache live events with a delay. Write to the tail, read from the head.
- thefreeman 2y ago
  that’s totally unacceptable for live sports which people are able to bet on
  - dullcrisp 2y ago
    Live sports require microwave relays for high frequency sports bets
  - rk06 2y ago
    Why should they catering to such an audience in first place? I think this could be one of upsells that Netflix could use. Premium: get no delay Normal users: get cache and delay
    - ctvo 2y ago
      Or, hear me out here, it's a wild concept, just work. You know, like every other broadcaster, streaming platform, and company that does live content has been able to do. Acting like this is a novel, hard problem that needs to be solved and we need to "upsell" it in tiers because Netflix is incompetent and live broadcasting hasn't been around for 80+ years is so fucking stupid.
      - kortilla 2y ago
        Every other live platform has a delay of multiple seconds
  - Brybry 2y ago
    This is kind of silly because the delay between actual event happening to showing up on OTA TV or cable TV to showing up on satellite TV can already be tens of seconds.
bushbaba 2y ago
Not sure I fully buy that. The “live” stream is rarely “live”. It’s often a highly cached buffer that’s a few mins from latest. Those in isp caches can still help here.
h4l 2y ago
That model still works for streaming. You have a central source stream only to the distributed edge locations, then have clients only stream from their local edge location. Even if one region is overwhelmed, the rest can still work. Load on the central source is bounded.
_dark_matter_ 2y ago
I don't think that live doesn't work with caches. No one watching live would care about a O(s) delay, which is highly amenable to caching at ISPs and streaming changes from there to downstream clients. Offhand I'd say that would support O(ms) delay but no less.
ethbr1 2y ago
I'm curious if the root cause is more variable than usual latency. Sample size 1, but... I saw a ton of buffering and failure on an embedded Netflix app on a TV, including some infinite freezes. Moved over to laptop, zero buffering. I assume the web app runs with a lot bigger buffer than whatever is squeezed into the underpowered TV.
- YZF 2y ago
  Likely these devices use different media formats and/or quality levels. And yes, it's possible one device buffers more than the other. Infinite freezes sounds like some routing issues or bugs.
  - ethbr1 2y ago
    When I was watching the behavior on the tv, was wondering if buffering sends some separate, non-business-as-usual requests, and that part of Netflix's delivery architecture was being overloaded. E.g. "give me this previous chunk" vs "send me the current stream"
    - YZF 2y ago
      Buffering typically just consumes the same live stream until there's enough in the buffer. No difference other than the request rate being potentially higher. At least I can confidently say that for the standard players/video platforms. NetFlix could be doing something different. I'm not sure if they have their own protocols. But I'd be very surprised if the buffering used a completely different mechanism.

tnvmadhav 2y ago

they will learn :)

pinkmuffinere 2y ago

Damn that sucks. I wonder if they could have intentionally streamed it 5 min late? I don’t have all the context around the fight though — maybe a competing service would win if Netflix intentionally induced delay?

adrr 2y ago
they were introducing 5 minute delays on some of the clients. I noticed my ipad was always live and the smart tv had a 5 minute delay but you could fast forward to live.

deleted 2y ago

[deleted]

stingraycharles 2y ago

Correct, this is not Netflix’ regular cup of tea, and it’s a very different problem to solve. They can probably use their edge caches, but it’s challenging.

nicce 2y ago
How YouTube does this? Netflix is like drop in the ocean when compared to.
- Bilal_io 2y ago
  Not sure how Netflix does it. But this is not very time sensitive, and I would have delayed the stream by 15 to 30 seconds to cache it and then deliver to everyone.
- unsnap_biceps 2y ago
  My wild assed guess is the differences in the edge nodes. Netflix's edge nodes are optimized for streaming already encoded videos to end users. They have to transcode some number of formats from the source and send them all to the edge nodes to flow out. It's harder to manage a ton of different streams flowing out to the edge nodes cleanly. I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Only one stream from source to edge to worry about and is much simpler to support and reason about. But that's just my wild assed guess.
  - vitus 2y ago
    > I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Ha, no, our edge nodes don't have anywhere near enough spare CPU to do transcoding on the fly. We have our own issues with livestreaming, but our system's developed differently over the past 15 years compared to Netflix's. While they've historically focused on intelligent pre-placement of data (which of course doesn't work for livestreaming), such an approach was never feasible for YT with the sheer size of our catalog (thanks to user-generated content). Netflix is still new to the space, and there isn't a good substitute for real-world experience for understanding how your systems behave under wildly different traffic patterns. Give them some time.
    - kortilla 2y ago
      It also helps that youtube serves shit tier quality videos more gracefully. Everyone is used to the step down to pixel-world on youtube to the point where they don’t complain much.

sgarland 2y ago

I wonder how effective it would be to cache live events with a delay. Write to the tail, read from the head.

thefreeman 2y ago
that’s totally unacceptable for live sports which people are able to bet on
- dullcrisp 2y ago
  Live sports require microwave relays for high frequency sports bets
- rk06 2y ago
  Why should they catering to such an audience in first place? I think this could be one of upsells that Netflix could use. Premium: get no delay Normal users: get cache and delay
  - ctvo 2y ago
    Or, hear me out here, it's a wild concept, just work. You know, like every other broadcaster, streaming platform, and company that does live content has been able to do. Acting like this is a novel, hard problem that needs to be solved and we need to "upsell" it in tiers because Netflix is incompetent and live broadcasting hasn't been around for 80+ years is so fucking stupid.
    - kortilla 2y ago
      Every other live platform has a delay of multiple seconds
- Brybry 2y ago
  This is kind of silly because the delay between actual event happening to showing up on OTA TV or cable TV to showing up on satellite TV can already be tens of seconds.

bushbaba 2y ago

Not sure I fully buy that. The “live” stream is rarely “live”. It’s often a highly cached buffer that’s a few mins from latest. Those in isp caches can still help here.

h4l 2y ago

That model still works for streaming. You have a central source stream only to the distributed edge locations, then have clients only stream from their local edge location. Even if one region is overwhelmed, the rest can still work. Load on the central source is bounded.

_dark_matter_ 2y ago

I don't think that live doesn't work with caches. No one watching live would care about a O(s) delay, which is highly amenable to caching at ISPs and streaming changes from there to downstream clients. Offhand I'd say that would support O(ms) delay but no less.

ethbr1 2y ago

I'm curious if the root cause is more variable than usual latency. Sample size 1, but... I saw a ton of buffering and failure on an embedded Netflix app on a TV, including some infinite freezes. Moved over to laptop, zero buffering. I assume the web app runs with a lot bigger buffer than whatever is squeezed into the underpowered TV.

YZF 2y ago
Likely these devices use different media formats and/or quality levels. And yes, it's possible one device buffers more than the other. Infinite freezes sounds like some routing issues or bugs.
- ethbr1 2y ago
  When I was watching the behavior on the tv, was wondering if buffering sends some separate, non-business-as-usual requests, and that part of Netflix's delivery architecture was being overloaded. E.g. "give me this previous chunk" vs "send me the current stream"
  - YZF 2y ago
    Buffering typically just consumes the same live stream until there's enough in the buffer. No difference other than the request rate being potentially higher. At least I can confidently say that for the standard players/video platforms. NetFlix could be doing something different. I'm not sure if they have their own protocols. But I'd be very surprised if the buffering used a completely different mechanism.