9 ms·

I wonder if there will be any long term reputational repercussions for Netflix because of this. Amongst SWEs, Netflix is known for hiring the best people and th

• lawgimenez 2y ago
  There's an upcoming NFL game on Netflix next month. They need to get their shit together.
  • caseyohara 2y ago
    Two games actually, both on Christmas Day. A day when most people are at home or the home of family or friends, and they are both pretty good late-season matchups (Chiefs-Steelers and Ravens-Texans) so I imagine viewership will be high. If they botch the NFL games, it will surely hurt their reputation.
• ocdtrekkie 2y ago
  So the issue is that Netflix gets its performance from colocating caches of movies in ISP datacenters, and a live broadcast doesn't work with that. It's not just about the sheer numbers of viewers, it's that a live model totally undermines their entire infrastructure advantage. See: https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/
  • tnvmadhav 2y ago
    they will learn :)
  • pinkmuffinere 2y ago
    Damn that sucks. I wonder if they could have intentionally streamed it 5 min late? I don’t have all the context around the fight though — maybe a competing service would win if Netflix intentionally induced delay?
    • adrr 2y ago
      they were introducing 5 minute delays on some of the clients. I noticed my ipad was always live and the smart tv had a 5 minute delay but you could fast forward to live.
  • deleted 2y ago
    [deleted]
  • stingraycharles 2y ago
    Correct, this is not Netflix’ regular cup of tea, and it’s a very different problem to solve. They can probably use their edge caches, but it’s challenging.
    • nicce 2y ago
      How YouTube does this? Netflix is like drop in the ocean when compared to.
      • Bilal_io 2y ago
        Not sure how Netflix does it. But this is not very time sensitive, and I would have delayed the stream by 15 to 30 seconds to cache it and then deliver to everyone.
      • unsnap_biceps 2y ago
        My wild assed guess is the differences in the edge nodes. Netflix's edge nodes are optimized for streaming already encoded videos to end users. They have to transcode some number of formats from the source and send them all to the edge nodes to flow out. It's harder to manage a ton of different streams flowing out to the edge nodes cleanly. I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Only one stream from source to edge to worry about and is much simpler to support and reason about. But that's just my wild assed guess.
        • vitus 2y ago
          > I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Ha, no, our edge nodes don't have anywhere near enough spare CPU to do transcoding on the fly. We have our own issues with livestreaming, but our system's developed differently over the past 15 years compared to Netflix's. While they've historically focused on intelligent pre-placement of data (which of course doesn't work for livestreaming), such an approach was never feasible for YT with the sheer size of our catalog (thanks to user-generated content). Netflix is still new to the space, and there isn't a good substitute for real-world experience for understanding how your systems behave under wildly different traffic patterns. Give them some time.
  • sgarland 2y ago
    I wonder how effective it would be to cache live events with a delay. Write to the tail, read from the head.
    • thefreeman 2y ago
      that’s totally unacceptable for live sports which people are able to bet on
      • dullcrisp 2y ago
        Live sports require microwave relays for high frequency sports bets
      • rk06 2y ago
        Why should they catering to such an audience in first place? I think this could be one of upsells that Netflix could use. Premium: get no delay Normal users: get cache and delay
        • ctvo 2y ago
          Or, hear me out here, it's a wild concept, just work. You know, like every other broadcaster, streaming platform, and company that does live content has been able to do. Acting like this is a novel, hard problem that needs to be solved and we need to "upsell" it in tiers because Netflix is incompetent and live broadcasting hasn't been around for 80+ years is so fucking stupid.
          • kortilla 2y ago
            Every other live platform has a delay of multiple seconds
      • Brybry 2y ago
        This is kind of silly because the delay between actual event happening to showing up on OTA TV or cable TV to showing up on satellite TV can already be tens of seconds.
  • bushbaba 2y ago
    Not sure I fully buy that. The “live” stream is rarely “live”. It’s often a highly cached buffer that’s a few mins from latest. Those in isp caches can still help here.
  • h4l 2y ago
    That model still works for streaming. You have a central source stream only to the distributed edge locations, then have clients only stream from their local edge location. Even if one region is overwhelmed, the rest can still work. Load on the central source is bounded.
  • _dark_matter_ 2y ago
    I don't think that live doesn't work with caches. No one watching live would care about a O(s) delay, which is highly amenable to caching at ISPs and streaming changes from there to downstream clients. Offhand I'd say that would support O(ms) delay but no less.
  • ethbr1 2y ago
    I'm curious if the root cause is more variable than usual latency. Sample size 1, but... I saw a ton of buffering and failure on an embedded Netflix app on a TV, including some infinite freezes. Moved over to laptop, zero buffering. I assume the web app runs with a lot bigger buffer than whatever is squeezed into the underpowered TV.
    • YZF 2y ago
      Likely these devices use different media formats and/or quality levels. And yes, it's possible one device buffers more than the other. Infinite freezes sounds like some routing issues or bugs.
      • ethbr1 2y ago
        When I was watching the behavior on the tv, was wondering if buffering sends some separate, non-business-as-usual requests, and that part of Netflix's delivery architecture was being overloaded. E.g. "give me this previous chunk" vs "send me the current stream"
        • YZF 2y ago
          Buffering typically just consumes the same live stream until there's enough in the buffer. No difference other than the request rate being potentially higher. At least I can confidently say that for the standard players/video platforms. NetFlix could be doing something different. I'm not sure if they have their own protocols. But I'd be very surprised if the buffering used a completely different mechanism.
• jimbob45 2y ago
  Yeah, the funny part is that Hulu, Amazon Prime, and Peacock have all demonstrated the ability to handle an event of this caliber with no issue. Netflix now may never get another opportunity like this again.
  • dyauspitr 2y ago
    Sure they will. They’ll just set up the next event and outside of some tech folks no one will remember this.
    • CitrusFruits 2y ago
      I mean, I guarantee you every boxing fan is never going to trust Netflix again for an event like this.
      • dyauspitr 2y ago
        What are they going to do? Just not watch the next fight?
      • kortilla 2y ago
        0 people give a shit about boxing fans. It’s not up to them.
  • silisili 2y ago
    AFAIK those three farm out to CDNs with tons of edges who know what they're doing. I have a feeling Netflix said 'how hard could this be?' and is finding out right now.
    • deleted 2y ago
      [deleted]
  • anshumankmr 2y ago
    Chances are Jake might fight Connor McGregor. Sure, Connor is not as famous as Tyson but that will also invite a large number of people to stream it.
• bsimpson 2y ago
  I don't spend much time streaming, but I got a glimpse of the Amazon Prime catalog yesterday, and was surprised at how many titles on the front page were movies I'd actually watch. Reminded me of Netflix a dozen years ago.
  • nicce 2y ago
    Amazon Prime front page includes currently a lot of ads for movies that you can rent or ”buy”. Are you sure these movies weren’t them?
    • joshuakcockrell 2y ago
      This is my biggest issue with Prime Video. You never know what's included and what costs extra.
      • squeaky-clean 2y ago
        There's also the "FreeVee" items, which have ads regardless of whether you're a prime subscriber or not. And it feels like a lot of their catalog has been transferred over to FreeVee.
      • HDThoreaun 2y ago
        You can toggle to only show free content. Still get ads but theyre obvious.
  • LollipopYakuza 2y ago
    Prime Video has to be the worst of all major streaming services. The video quality is horrible, its crippled with ads (3 not skippable ads for an episode of 45 minutes, lastly), and a lot of interesting titles are behind a "partner paywall".
    • hughesjj 2y ago
      It's also super annoying to try to watch on a computer compared to Netflix or YouTube
• ilaksh 2y ago
  I think they have to refund the fees for a month to anyone who streamed this fight. That's the only thing that seems fair. It has been pretty useless. At the moment seems to be working only when running in non-live mode several minutes behind. So if there are 1 million trying to stream it, that means they would lose $15 million. So.. they might only give a partial refund. But people should push for an automatic refund instead of a class action.
  • SoftTalker 2y ago
    Is it really that big a deal if you are watching a few minutes behind? I've watched ball games on streaming networks where I can also hear a local radio broadcast, and the stream is always delayed compared to the radio, sometimes by quite a lot. But you'd never know it if you were just watching the stream.
    • notatoad 2y ago
      >Is it really that big a deal if you are watching a few minutes behind? i don't bet on sports. but from friends who do: yes, it's a really really big deal.
      • throwaway2037 2y ago
        I have seen this type of comment a few times here. Why does it matter to a bettor if the stream is delayed by 1-5 mins? How would they even know?
        • brokenmachine 2y ago
          I remember a few years ago reading about a scam at the Australian Open Tennis where there were people inside the stadium who were betting on individual points as they happened. I guess they could bet before the betting streams caught up. It seems ridiculous to me that you can bet on individual points, but here we are.
    • defrost 2y ago
• jeromegv 2y ago
  Not the same demographic but their last large attempt at live was through a Love is blind reunion. It was the same thing, millions of people logging in, epic failure, nothing worked. They never tried to do a live reunion again. I suppose they should have to get the experience. Because they are hitting the same problems with a much bigger stake event.
  • barkingcat 2y ago
    yup wanted to say that live stream stuttering has happened before on Netflix - I don't think the reputation is deserved. From a livestreaming standpoint, netflix is 0/x - for many large events such as love is blind, etc. From a livestreaming standpoint, look to broadcast news, sports / Olympics broadcasters, etc and you'll see technology, equipment, bandwidth, planning, and professionalism at 1000x of netflix. Heck, for publicly traded quarterly earnings livestream meetings, they book direct satellite time in addition to fiber to make sure they don't rely only on terrestrial networks which can fail. From a business standpoint, failure during a quarterly meeting stream can mean the destruction of a company (by making shareholders mad that they can't see and vote during the meeting making them push for internal change) - so the stakes are much higher than live entertainment streaming. Netflix is good at many things, livestreaming is not one of those things.
    • glimshe 2y ago
      Even some of the old guard can do this. The Olympics worked pretty well (despite the awkward UI), and that was Peacock/NBC. Perhaps Netflix still needs a dozen more microservices to get this right...
    • emeril 2y ago
      All valid points though each of those examples seemingly only has a fraction of the viewers of the netflix events, right?
      • barkingcat 2y ago
        for livestreams, individual events like the Olympics probably has a surge audience of 10x of netflix events. Netflix events is small potatoes compared to other livestream stalwarts. Imagine having to stream a cricket match internationally to UK / India / Australia with combined audience that crushes the Superbowl or a football match to all of Europe, or even something like livestreaming F1 racing that has multiple magnitudes of audience than a boxing match and also has 10x the number of cameras (at 8K+ resolution) across a large physical staging arena (the size of the track/course) in realtime, in addition to streaming directly from the cockpit of cars that are racing 200mph++. Livestream focused outfits do this all day, everyday. Netflix doesn't even come close to scratching the "beginner" level of these kinds of live events. It's a matter of competencies. We wouldn't expect Netflix to be able to serve burgers like McDonald's does - Livestreaming is a completely different discipline and it's hubris on Netflix's part to assume just because they're good at sending video across the internet they can competently do livestreaming.
• Spooky23 2y ago
  It may vary by ISP. It’s been fine for me.
  • graton 2y ago
    I think you are correct. Ziply Fiber said they were seeing 2.1 times their normal peak [1]. But also people were saying they weren't having any issues streaming on Ziply. [1] https://www.reddit.com/r/ZiplyFiber/comments/1gsenik/netflix_fight_night_peering_stats/ https://www.reddit.com/r/ZiplyFiber/comments/1gsenik/netflix...
• rvz 2y ago
  I don't think Netflix is even designed to handle very extreme multi-region live-streaming at scale as evidenced in this event with hundreds of millions simultaneously watching. YouTube, Twitch, Amazon Prime, Hulu, etc have all demonstrated to stream simultaneously to hundreds of millions live without any issues. This was Netflix's chance to do this and they have largely failed at this. There are no excuses or juniors to blame this time. Quite the inexperience from the 'senior' engineers at Netflix not being able to handle the scale of live-streaming which they may lose contracts for this given the downtime across the world over this high impact event. Very embarrassing for a multi-billion dollar publicly traded company.
  • gonzo41 2y ago
    They must have try to do this on the cheap, thinking they could dynamically scale on the fly for this. Big mistake.
    • fragmede 2y ago
      Their CDN is colo and doesn't run on AWS.
    • anakaine 2y ago
      This is a total supposition without any proof.
      • colesantiago 2y ago
        What more proof do you need other than the fact that streams went down worldwide on a highly anticipated event from a public company? I wouldn't be surprised if lots of engineers at Netflix are currently now writing up a length post mortem of this. And this is from the company that created the discipline of chaos engineering for resilience. It is clear they under invested and took the eye of the ball with this. This is bad, like very very bad.
        • jhugo 2y ago
          The assumption that it was related to insufficient investment isn’t supported by any evidence. Flawed technical decisions can be made by the most expensive engineers too.
• 2OEH8eoCRo0 2y ago
  Is cable and broadcast better for live TV? No scaling issues. Doesn't matter how many people tune in.
  • Valord 2y ago
    Seems like it
  • betaby 2y ago
    Interestingly that TV nowadays is delivered through IP.
  • vitus 2y ago
    It's fundamentally different, for sure. Most third-party internet-based streaming solutions are overlaid on top of a point-to-point network, while broadcast is one-to-many, and even cable tends to use multicast within the cable provider's network. You have potentially different problems, e.g. limited bandwidth / spectrum. If, say, there are multiple games going on at the same time, you can only watch whichever feed the broadcaster decides to air. And, of course, regardless of the technology in use, there are matters of acquiring rights for various events. One benefit of internet-based streaming is that one service can acquire the rights and be able to reach everyone, whereas an individual cable provider might only reach its direct subscribers.
    • Ekaros 2y ago
      On cable(terrestrial is entirely different) even the bandwidth or spectrum is less limiting for broadcasting multiple games. Hard thing is the other parts of production, like cameras, live-directing and live commentary. Adding new channels is less challenging than actual producing content at expected level there.
• oaththrowaway 2y ago
  I used to work for a live streaming platform once. We always joked that VOD (Netflix) was "easy" compared to live.
  • taeric 2y ago
    Not really a joke, though? VOD has obvious methods to cheat a bit. Redundancy abounds and you can even network shape for costs. Could probably get even better compression for clear reasons. Live, not so much. One source that you have to fanout from and absolutely no way to get cheap redundancy. Right?
    • oaththrowaway 2y ago
      Yes, of course but it was still a cope because we never saw more than maybe 3M concurrent viewers at a time
• noncoml 2y ago
  > their streaming service normally seems very solid Not trying to downplay their complexity, but last I heard Netflix is splitting the shows in small data chunks and just serves them as static files. Live streaming is a different beast
  • treflop 2y ago
    Static files have been pretty much the standard streaming protocols for both VOD and live for the last 15 years ago. Before, it was Adobe Flash (RTMP). With the way that they are designed, you can even use a regular CDN.
    • adrr 2y ago
      You can push these files to all the edges before you release the content which will protect your origin. Livestream all your edge servers are grabbing content from the origin unless you have another tier of regional servers to alleviate load.
      • treflop 2y ago
        Sure but that’s why your edge servers do request collapsing. And there are full blown CDN companies that will write an enterprise contract with you that can do this stuff with ease. Akamai is like 25 years old now. Scale has increased but the techniques were figured out 20 years ago. There is not much left to invent in this space at the current moment so screwing up more than once is a bit unacceptable.
  • astrange 2y ago
    That's how streaming (usually) works. The main URL is a playlist of transport stream files and it just downloads them in the background as you go.
• a_random_name 2y ago
  If Netflix still interviews on hacker rank puzzles I think this should be a wake up call. Interviewing on irrelevant logic puzzles is no match for systems engineering.
  • rvz 2y ago
    For systems design and engineering, absolutely this. I expected the very highest standards and upmost uptime from Netflix, similar to Google and Amazon. Tells you the uselessness of their engineering blogs.
  • VirusNewbie 2y ago
    I did a round of netflix interviews, didn't get an offer (but passed the technical coding rounds) they absolutely had the best interview process of any company I've interviewed at my entire career. They do make you code but the questions were 1. Not on hacker rank or leetcode 2. Pratical coding questions that didn't require anything more than basic hashmaps/lists/loops/recursion if you want. Some string parsing, etc. They were still hard, you had to code a fast, but no tricky algorithms required. It also felt very collaborative, it felt like you were driving pair programming. Highly recommended even though didn't get an offer!
• that_guy_iain 2y ago
  I don't think it'll be long-term. Most people will forget about this really quickly. It's not like there will be many people saying "Oh, you don't want to sign up for Netflix, the Tyson fight wasn't well streamed" in even 6 months nevermind 10 years.
• jpalawaga 2y ago
  From what I've heard, Netflix has really diluted the culture that people know of from the Patty McCord days. In particular, they have been revising their compensation structure to issue RSUs, add in a bunch of annoying review process, add in a bunch of leveling and titles, begin hiring down market (e.g. non-sr employees), etc. In addition to doing this, shuffling headcount, budgets, and title quotas around has in general made the company a lot more bureaucratic. I think, as streaming matured as a solution space, this (what is equivalent to cost-cutting) was inevitable. If Netflix was running the same team/culture as it was 10 years ago, I'd like to say that they would have been able to pull of streaming.
  • neilv 2y ago
    Were they not able to hire enough top-skilled people? If not, why not? Or did they have a lot of needs that they decided didn't require top-skilled people? Or was this a beancounter thing, of someone deciding that the company was paying more money on staffing than they needed to, without understanding it?
    • jpalawaga 2y ago
      Combination of 2 and 3. The business changed. Streaming was more or less a solved problem for Netflix. They needed money for content, not expensive engineers. Ted is co-ceo… you can see where the priority is.
    • AtlasBarfed 2y ago
      My observation is that Netflix is one of those places that brags about how they do so much with so little employees.
      • dboreham 2y ago
        Few. Little employees would be...small.
• squeaky-clean 2y ago
  Has Netflix ever live streamed something before? People on reddit are reporting that if you back up the play marker by about 3 minutes the lag goes away. They've got a handle on streaming things when they have a day in advance to encode it into different formats and push it to regional CDNs. But I can't recall them ever live streaming something. Definitely nothing this hyped.
  • ipaddr 2y ago
    Love is blind reunion which had major problems. Chris Rock comedy which only had a few issues. The Netflix Cup which had issues.
  • StevePerkins 2y ago
    Chris Rock comedy special, and the Tom Brady roast. Nothing on this scale, though.
• Fire-Dragon-DoL 2y ago
  For me, netflix constantly forget the last episode/spot I was in a TV show. Beyond frustrating
• geodel 2y ago
  I think Netflix will have even more sw engineers looking to work there once they notice even for average quality of work they can get paid 3 times more than their current pay.
• test6554 2y ago
  Yea, it’s a bad look. But I switched to watching some other Netflix video and it seemed fine. Just this event had some early issues. Looks fine now though.
• rapind 2y ago
  > ut my impression still has always been that Netflix is a step above the rest technically. I always assumed youtube was top dog for performance and stability. I can’t remember the last time I had issues with them and don’t they handle basically more traffic than any other video service?
  • normie3000 2y ago
    Maybe a client issue, but i've got a low-end smart tv which handles netflix fine, but youtube is unwatchable due to buffering and failed cuts to adverts
    • rapind 2y ago
      Maybe that’s it. I pay from premium though so don’t have the advert issue (apples to apples).
• canadianfella 2y ago
  [dead]
• kenjackson 2y ago
  Streamed glitch free for me both on my phone and Xbox. The fight wasn’t so great though, but still a fun event. Jake Paul is a money machine right now.
  • tnvmadhav 2y ago
    same. could be a bandwidth issue at CDN/ISP level at a certain region(s) ?
• ikt 2y ago
  Based on this I'm wondering whether it was straight up they did not expect it to be this popular? > Some Cricket graphs of our #Netflix cache for the #PaulVsTyson fight. It has a 40 Gbps connection and it held steady almost 100% saturated the entire time. https://fosstodon.org/@atoponce/113491103342509883 https://fosstodon.org/@atoponce/113491103342509883
• dh2022 2y ago
  I think why I will remember about this fight is not the (small) streaming issue I encountered as much as the poor quality of the fight itself. For me that was the reputational loss. Netflix was touting “NFL is coming to Netflix”. This fight did not really make me want to watch that.
  • BLKNSLVR 2y ago
    I don't care about boxing or UFC or the grade-A douchebags that are the Paul brothers, but I tuned in just because I had the time and a Netflix subscription. It was actually great that the fight itself was so boring because it justifies never having to spend time / money on that kind of bullshit. It was a farce. A very bright, loud, sparkly, and expensive (for some people) farce. The value I got from it was the knowledge that missing out on that kind of thing isn't really missing out on anything at all.
• tass 2y ago
  Was live streaming much of a use case for them before this? They stream plenty of pre recorded video, often collocated. Live streaming seems like something they aren’t yet good at.
  • someothherguyy 2y ago
    https://www.cnn.com/2023/03/04/entertainment/chris-rock-netflix-live-special/index.html https://www.cnn.com/2023/03/04/entertainment/chris-rock-netf... (2023-03-06)
  • fred_is_fred 2y ago
    If places like Paramount+ can figure it out, Netflix, given their 10+ year head start on streaming and strong engineering culture, should also have been able to. And if you don't like my example, literally every other streaming service has streamed live sports without issue. YT TV, Hulu, Paramount+, Amazon Prime, Peacock, even Apple TV streams live sports. It may be "new" to them, but they should have been ready.
    • tass 2y ago
      I won’t argue that they shouldn’t have done better, I’m only pointing out that this is fairly different from their usual product. Amazon, YouTube, and Hulu all have a ton of experience with live streaming by now. Apple has live streamed wwdc for several years. I did expect that Netflix would have appropriately accounted for demand and scale, though, especially given the hype for this particular event.
      • bobdvb 2y ago
        Even Amazon doesn't have it perfected for Live. Live sports and VOD movies/TV are very different beasts.
• apexalpha 2y ago
  >but my impression still has always been that Netflix is a step above the rest technically. Maybe if we're not counting Youtube as 'streaming', but in my mind no one holds a candle to YT quality in (live)streaming.
• Yeul 2y ago
  There's a difference between live broadcasts and serving up content that's sitting on a server I guess? In my country every time there's a big football match the people who try to watch it on the internet face issues.
• MyFedora 2y ago
  Netflix won't take a hit here. Most people pay Netflix to watch movies and tv shows, not sports. If I hadn't checked Hacker News today, I wouldn't even know they streamed sports, let alone that they had issues with it. Even now that I do, it doesn't affect how I see their core offering, which is their library of on-demand content. Netflix's infrastructure is clearly built for static content, not live events, so it's no shock they aren't as polished in this area. Streaming anything live over the internet is a tough technical challenge compared to traditional cable.
  • oplav 2y ago
    Netflix is trying to expand into live sports. This event wasn’t a one off thing. There is an NFL game they are streaming at the end of the year.