8 ms·

It’s insane the excuses being made here for Netflix’s apparently unique circumstances. They failed. Full stop. There is no valid technical reason they couldn’t

by ctvo 2y ago

It’s insane the excuses being made here for Netflix’s apparently unique circumstances. They failed. Full stop. There is no valid technical reason they couldn’t have had a smooth experience. There are numerous people with experience building these systems they could have hired and listened to. It isn’t a novel problem. Here are the other companies that are peers that livestream just fine, ignoring traditional broadcasters: - Google (YouTube live), millions of concurrent viewers - Amazon (Thursday Night Football, Twitch), millions of concurrent viewers - Apple (MLS) NBC live streamed the Olympics in the US for tens of millions.

csallen 2y ago
> They failed. Full stop. It's not full stop. There are reasons why they failed, and for many it's useful and entertaining to dissect them. This is not "making excuses" and does not get in the way of you, apparently, prioritizing making a moral judgment.
y-c-o-m-b 2y ago
Amazon had their fair share of livestream failures and for notably less viewers. I don't think they deserve a spot on that list. I briefly worked in streaming media for sports and while it's not a novel problem, there are so many moving parts and points of failure that it can easily all go badly.
- deanCommie 2y ago
  There is no one "Amazon" here, there are at least 3: * Twitch: Essentially invented live streaming. Fantastic. * Amazon Interactive Video Service [0]: Essentially "Twitch As A Service", built by Twitch engineers. Fantastic. * Prime Video. Same exact situation as Netflix: original expertise is all in static content. Lots of growing pains with live video and poor reports. But they've figured it out: now there are regular live streams (NHL and NFL), and other channel providers do live streaming on Prime Video as a distribution platform. [0] https://aws.amazon.com/ivs/ https://aws.amazon.com/ivs/
  - margaretdouglas 2y ago
    Doesn't twitch almost fall over (other non-massive streams impacted) when anyone gets close to 4-5m concurrent viewers? I remember last time it happened everything started falling over, even for smaller streams. Even if Netflix struggled with the event, streaming other content worked just fine for me.
  - ukuina 2y ago
    IVS does not scale past 1080p: https://ivs.rocks/ https://ivs.rocks/
    - deanCommie 2y ago
      4K is overrated for streaming. Most people's connections can't handle it so it always ends up getting downscaled. pisses me off. physical media for the win, tho.
fredgrott 2y ago
it could be that they made use of the same advice X followed :)
mikeryan 2y ago
The big difference of all the examples you’ve mentioned is dedicated full-time crews on the ground where the events are produced. I’m pretty confident that when the post mortem is done the issues are going to be way closer to the broadcast truck than the user.
mfiguiere 2y ago
The examples given here are not on the same scale. The numbers known so far: - 120m viewers [1] - Entire Netflix CDN Traffic grew 4x when the live stream started [2] [1] https://www.rollingstone.com/culture/culture-news/jake-paul-mike-tyson-boxing-win-1235166434/ https://www.rollingstone.com/culture/culture-news/jake-paul-... [2] https://x.com/DougMadory/status/1857634875257294866 https://x.com/DougMadory/status/1857634875257294866
- prasadjoglekar 2y ago
  Disney HotStar managed to stream ~60M livestreams for the Cricket world cup a year ago. The problem has been solved. Livestreaming sports just have a different QoS expectations than on demand. https://www.geeksforgeeks.org/how-did-hotstar-managed-5-9-cr-viewers-during-world-cup23/ https://www.geeksforgeeks.org/how-did-hotstar-managed-5-9-cr...
  - achow 2y ago
    Size of Hotstar team = ~2000. Enggrs will be less than that number. https://www.linkedin.com/company/disney-hotstar/people/ https://www.linkedin.com/company/disney-hotstar/people/
    - manquer 2y ago
      The size of engineering head count is not informative, it really depends on how much is in-house and how much is external for Hotstar that would be i.e parent Disney or before Fox or staffing from IT consulting organizations who will not be on payroll. For what it is worth, all things being equal there would be lot more non engineering in Hotstar for 2000 employees versus a streaming company of similar size or scale of users. Hotstar operates in challenging and fragmented market, India has 10+ major languages(and corresponding TV, music and movie markets) Technically there is not much difference to what Netflix or Disney has to do for i18n, however operationally each market needs separate sales, distribution and operations. --- P.S. Yes Netflix operates in more markets including India than anybody else, however if you are actually using Netflix for almost any non English content, you will know how weak their library and depth in other markets are, their usual model in most of these markets is to have few big high quality(for that market) content rather than build depth. P.P.S. Also yes, Indian market is seeing consolidation in the sense that many releases on streaming are multiple lingual and use major stars from more than one language to draw talent ( not new, but growing in popularity as distribution becomes cheaper with streaming), however this is only seen in big banner productions as tastes are quite different in each market and can't scale for all run of the mill content.
toast0 2y ago
I don't disagree that Netflix could have / should have done better. But everybody screws these things up. Even broadcast TV screws these things up. Live events are difficult. I'll also add on, that the other things you've listed are generally multiple simultaneous events; when 100M people are watching the same thing at the same time, they all need a lot more bitrate at the same time when there's a smoke effect as Tyson is walking into the ring; so it gets mushy for everyone. IMHO, someone on the event production staff should have an eye for what effects won't compress well and try to steer away from those, but that might not be realistic. I did get an audio dropout at that point that didn't self correct, which is definitely a should have done better. I also had a couple of frames of block color content here and there in the penultimate bout. I've seen this kind of stuff on lots of hockey broadcasts (streams or ota), and I wish it wouldn't happen... I didn't notice anything like that in the main event though. Experience would likely be worse if there were significant bandwidth constraints between Netflix and your player, of course. I'd love to see a report from Netflix about what they noticed / what they did to try to avoid those, but there's a lot outside Netflix's control there.
freefaler 2y ago
As a cofounder of a CDN company that pushed a lot of traffic, the problem with live streaming is that you need to propagate peak viewership trough a loooot of different providers. The peering/connectivity deals are usually not structured for peak capacity that is many times over the normal 95th percentile. You can provision more connectivity, but you don't know how many will want to see the event. Also, live events can be trickier than stored files, because you can't offload to the edges beforehand to warm up the caches. So Netflix had 2 factors outside of their control - unknown viewership - unknown peak capacities outside their own networks Both are solvable, but if you serve "saved" content you optimize for different use case than live streaming.

by ctvo 2y ago

csallen 2y ago

> They failed. Full stop. It's not full stop. There are reasons why they failed, and for many it's useful and entertaining to dissect them. This is not "making excuses" and does not get in the way of you, apparently, prioritizing making a moral judgment.

y-c-o-m-b 2y ago

Amazon had their fair share of livestream failures and for notably less viewers. I don't think they deserve a spot on that list. I briefly worked in streaming media for sports and while it's not a novel problem, there are so many moving parts and points of failure that it can easily all go badly.

deanCommie 2y ago
There is no one "Amazon" here, there are at least 3: * Twitch: Essentially invented live streaming. Fantastic. * Amazon Interactive Video Service [0]: Essentially "Twitch As A Service", built by Twitch engineers. Fantastic. * Prime Video. Same exact situation as Netflix: original expertise is all in static content. Lots of growing pains with live video and poor reports. But they've figured it out: now there are regular live streams (NHL and NFL), and other channel providers do live streaming on Prime Video as a distribution platform. [0] https://aws.amazon.com/ivs/ https://aws.amazon.com/ivs/
- margaretdouglas 2y ago
  Doesn't twitch almost fall over (other non-massive streams impacted) when anyone gets close to 4-5m concurrent viewers? I remember last time it happened everything started falling over, even for smaller streams. Even if Netflix struggled with the event, streaming other content worked just fine for me.
- ukuina 2y ago
  IVS does not scale past 1080p: https://ivs.rocks/ https://ivs.rocks/
  - deanCommie 2y ago
    4K is overrated for streaming. Most people's connections can't handle it so it always ends up getting downscaled. pisses me off. physical media for the win, tho.

fredgrott 2y ago

it could be that they made use of the same advice X followed :)

mikeryan 2y ago

The big difference of all the examples you’ve mentioned is dedicated full-time crews on the ground where the events are produced. I’m pretty confident that when the post mortem is done the issues are going to be way closer to the broadcast truck than the user.

mfiguiere 2y ago

The examples given here are not on the same scale. The numbers known so far: - 120m viewers [1] - Entire Netflix CDN Traffic grew 4x when the live stream started [2] [1] https://www.rollingstone.com/culture/culture-news/jake-paul-mike-tyson-boxing-win-1235166434/ https://www.rollingstone.com/culture/culture-news/jake-paul-... [2] https://x.com/DougMadory/status/1857634875257294866 https://x.com/DougMadory/status/1857634875257294866

prasadjoglekar 2y ago
Disney HotStar managed to stream ~60M livestreams for the Cricket world cup a year ago. The problem has been solved. Livestreaming sports just have a different QoS expectations than on demand. https://www.geeksforgeeks.org/how-did-hotstar-managed-5-9-cr-viewers-during-world-cup23/ https://www.geeksforgeeks.org/how-did-hotstar-managed-5-9-cr...
- achow 2y ago
  Size of Hotstar team = ~2000. Enggrs will be less than that number. https://www.linkedin.com/company/disney-hotstar/people/ https://www.linkedin.com/company/disney-hotstar/people/
  - manquer 2y ago
    The size of engineering head count is not informative, it really depends on how much is in-house and how much is external for Hotstar that would be i.e parent Disney or before Fox or staffing from IT consulting organizations who will not be on payroll. For what it is worth, all things being equal there would be lot more non engineering in Hotstar for 2000 employees versus a streaming company of similar size or scale of users. Hotstar operates in challenging and fragmented market, India has 10+ major languages(and corresponding TV, music and movie markets) Technically there is not much difference to what Netflix or Disney has to do for i18n, however operationally each market needs separate sales, distribution and operations. --- P.S. Yes Netflix operates in more markets including India than anybody else, however if you are actually using Netflix for almost any non English content, you will know how weak their library and depth in other markets are, their usual model in most of these markets is to have few big high quality(for that market) content rather than build depth. P.P.S. Also yes, Indian market is seeing consolidation in the sense that many releases on streaming are multiple lingual and use major stars from more than one language to draw talent ( not new, but growing in popularity as distribution becomes cheaper with streaming), however this is only seen in big banner productions as tastes are quite different in each market and can't scale for all run of the mill content.

toast0 2y ago

I don't disagree that Netflix could have / should have done better. But everybody screws these things up. Even broadcast TV screws these things up. Live events are difficult. I'll also add on, that the other things you've listed are generally multiple simultaneous events; when 100M people are watching the same thing at the same time, they all need a lot more bitrate at the same time when there's a smoke effect as Tyson is walking into the ring; so it gets mushy for everyone. IMHO, someone on the event production staff should have an eye for what effects won't compress well and try to steer away from those, but that might not be realistic. I did get an audio dropout at that point that didn't self correct, which is definitely a should have done better. I also had a couple of frames of block color content here and there in the penultimate bout. I've seen this kind of stuff on lots of hockey broadcasts (streams or ota), and I wish it wouldn't happen... I didn't notice anything like that in the main event though. Experience would likely be worse if there were significant bandwidth constraints between Netflix and your player, of course. I'd love to see a report from Netflix about what they noticed / what they did to try to avoid those, but there's a lot outside Netflix's control there.

freefaler 2y ago

As a cofounder of a CDN company that pushed a lot of traffic, the problem with live streaming is that you need to propagate peak viewership trough a loooot of different providers. The peering/connectivity deals are usually not structured for peak capacity that is many times over the normal 95th percentile. You can provision more connectivity, but you don't know how many will want to see the event. Also, live events can be trickier than stored files, because you can't offload to the edges beforehand to warm up the caches. So Netflix had 2 factors outside of their control - unknown viewership - unknown peak capacities outside their own networks Both are solvable, but if you serve "saved" content you optimize for different use case than live streaming.