23 ms·
Netflix buffering issues: Boxing fans complain about Jake Paul vs. Mike Tyson
- yapyap 2y agoHell, I’d complaing about Jake Paul vs Mike Tyson as well if I was a boxing fan. Even without buffering issues
- koolba 2y agoThey better get have some better judges and refs too. The co-headline title fight was a joke.
- smolder 2y agoYes, it was utterly boring, but they made their money. I don't like either Paul brother, so I only watched in hopes shorter, much-older Tyson would make Jake look as foolish as he is.
- Trasmatta 2y agoSilicon Valley predicted this: https://youtu.be/ddTbNKWw7Zs https://youtu.be/ddTbNKWw7Zs
- johnny_canuck 2y agoIt's far from perfect here in Canada, I keep having to pause it or go back and then load it again. Oddly having watched PPV events via the high seas for years, it feels normal...
- ad9204197 2y ago[dead]
- suzzer99 2y agoOn a few forum sites I'm on, people are just giving up. Looking forward to the post-mortem on how they weren't ready for this (with just a tiny bit of schadenfreude because they've interviewed and rejected me twice).
- itsthecourier 2y agoThey sabotaging OP just for a reverse schadenfreude play
- Amfy 2y agoCan you share which forums
- dyauspitr 2y agoIt’s not everyone. Works fine for me though I did have to reload the page when I skipped past the woman match to the Barrios Ramos fight and it was stuck buffering at 99%.
- icameron 2y agoYou skipped the best part.
- moralestapia 2y agoThe post-mortem will be interesting indeed.
- scruple 2y agoAB84 streamed it live from a box at the arena to ~5M viewers on Twitter. I was watching it on Netflix, I didn't have any problems, but I also put his live stream up for the hell of it. He didn't have any issues that I saw.
- deleted 2y ago[deleted]
- tiffanyh 2y agoDupe: https://news.ycombinator.com/item?id=42153906 https://news.ycombinator.com/item?id=42153906
- josh2600 2y agoFWIW, works fine for me.
- iancmceachern 2y agoUs too
- roamerz 2y agoBeen working great for me as well. Starlink in Oregon.
- whoknowsidont 2y agoPlease don't make these types of comments, they mean nothing and they serve no purpose.
- pentagrama 2y agoIt means, it is different if the service goes down to 100, 50, 10 percent of users. I watched the show with no issues.
- whoknowsidont 2y agoComments on forums do not provide that data. And if you want to extrapolate self-reports, it's obviously fine (to varying degrees) for the vast majority of people, but that's not the "issue." These kind of reports are the equivalent of saying "I have power" when you're hundreds of miles away from where a hurricane landed. It's uninteresting, it's likely you have power, and it does literally nothing for people that do not have power. It doesn't advance the topic anywhere. There are other places to report these issues directly and in aggregate with other people -- HN (and many other forums) are not that place.
- kortilla 2y agoYou’re in a fucking thread about people commenting in a forum about an outage that may or may not have been caused by Netflix
- Justin_K 2y agoI love how I can come to HN to instantly find out if it’s Netflix or my WiFi.
- jfdi 2y agoRight?!
- slicktux 2y agoThis! I was checking my WiFi and then I instinctively checked HN and what do you know!
- Jiahang 2y agometoo！
- dghughes 2y agoWifi wifi or wifi as in your ISP Internet connection? Sp many people now call an Internet connection "wifi". Anyway, network cable is the only way to go!
- anakaine 2y ago"Many people", those who call their ISP connection WiFi, are technology potato's.
- fred_is_fred 2y agoTo be fair, a lot of people pay their ISP for a modem/router combo and connect to something like "Xfinity" at their house. So to them, there is no difference.
- tatersolid 2y agoThey may not know technology, but most of those potatoes at least know the difference between a plural and a possessive noun.
- rascul 2y agoI have the wifis with the geebees.
- bluSCALE4 2y agoLooks like shit for me. Buffered a bit as well.
- freditup 2y agoI wonder if there will be any long term reputational repercussions for Netflix because of this. Amongst SWEs, Netflix is known for hiring the best people and their streaming service normally seems very solid. Other streaming services have definitely caught up a bit and are much more reliable then in the early days, but my impression still has always been that Netflix is a step above the rest technically. This sure doesn't help with that impression, and it hasn't just been a momentary glitch but hours of instability. And the Netflix status page saying "Netflix is up! We are not currently experiencing an interruption to our streaming service." doesn't help either...
- lawgimenez 2y agoThere's an upcoming NFL game on Netflix next month. They need to get their shit together.
- caseyohara 2y agoTwo games actually, both on Christmas Day. A day when most people are at home or the home of family or friends, and they are both pretty good late-season matchups (Chiefs-Steelers and Ravens-Texans) so I imagine viewership will be high. If they botch the NFL games, it will surely hurt their reputation.
- ocdtrekkie 2y agoSo the issue is that Netflix gets its performance from colocating caches of movies in ISP datacenters, and a live broadcast doesn't work with that. It's not just about the sheer numbers of viewers, it's that a live model totally undermines their entire infrastructure advantage. See: https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/
- tnvmadhav 2y agothey will learn :)
- pinkmuffinere 2y agoDamn that sucks. I wonder if they could have intentionally streamed it 5 min late? I don’t have all the context around the fight though — maybe a competing service would win if Netflix intentionally induced delay?
- Willingham 2y agoI can feel the pressure on the network engineers from here XD
- nightowl_games 2y agoFrom my experience, it works if your not watching it 'live'. But the moment I put my devices to 'live' it perma-breaks. 504 gateway timed out in web developer tools hitting my local CDN. probably works on some CDNs, doesnt on others. Probably works if your not 'live' edit: literally a nginx gateway timed out screen if you view the response from the cdn... wow
- jeffbee 2y agoWhy do they want to get into the live business? It doesn't seem to synergize with their infrastructure. Sending the same stream in real time to numerous people just isn't the same task as letting people stream optimized artifacts that are prepositioned at the edge of the network.
- dyauspitr 2y agoThey want to break into sports because it’s such a big business and if you do sports you need to be able to stream live.
- smegger001 2y agoBecause its the one area traditional networks have the advantage in
- jeffbee 2y agoThe marginal cost to add a viewer to broadcast sports is zero! That's what I am getting at. I know why someone would want this business, I just don't see what aspect of Netflix's existing business made them think they'd be good at it.
- VirusNewbie 2y agoMost PPV is what, $50-$70? So subscribing to Netflix for $20 or whatever per month sounds like a bargain for anyone who is interested and not already a customer. Then assume some large percentage doesn’t cancel either because they forgot, or because they started watching a show and then decided to keep paying.
- dilyevsky 2y agoNot sure why this is being downvoted. I can see your point - it’s much harder to this live but a lot of their cdn infra can be reused.
- SkyPuncher 2y agoLive is the only thing that won’t be commodified entirely. “Anyone” can pump out stream-when-you-want TV shows. Live events are generally exclusive, unpredictable, and cultural moments .
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- jairuhme 2y agoEvery time it buffers for me, Netflix does an internet test only for it to come back and say its fast...
- cranberryturkey 2y agoyeah i'm using iptv which is just a rip of NF and its stuck buffering.
- rvz 2y agoAll these engineering blog posts, distributed systems and these complex micro-services clearly didn't help with this issue. Netflix is clearly not designed nor prepared for scalable multi-region live-streaming, no matter the amount of 'senior' engineers they throw at the problem.
- ilrwbwrkhv 2y agoThis. Overly complex nonsense that they have built up is crazy and the fact that we on a tech forum agree to all of this is crazier. It's almost like this platform has been taken over by JavaScripters completely.
- sgarland 2y ago> taken over by JavaScripters That is an incredible way to phrase the sentiment, thank you.
- robertlagrant 2y ago> Netflix is clearly not designed nor prepared for scalable multi-region live-streaming Well, yes. Who would think Netflix was designed for that? They do VOD. They're only trying to move into this now.
- yoshamano 2y agoI did some VPN hopping and connecting to an endpoint in Dallas has allowed me to start watching again. Not live though, that throws me back into buffering hell.
- runjake 2y agoIt’s like watching a Minecraft cosplay of the event.
- ackbar03 2y agoThis reminds me of that scene in Silicon Valley
- neilv 2y agohttps://www.youtube.com/watch?v=9IGvzb-KCpY&t=50s https://www.youtube.com/watch?v=9IGvzb-KCpY&t=50s
- abixb 2y agoI clicked on this thread to type that exact thing, holy smokes. You're referring to Hooli's streaming of UFC fight that goes awry and Gavin Belson totally loses it, lol. Great scene and totally relevant to what's happening with Netflix rn.
- a_random_name 2y agoOta broadcasts are clearer
- causality0 2y agoWeird that an organization like Netflix is having problems with this considering their depth of both experience and pockets. I wonder if they didn't expect the number of people who were interested in finding out what the pay-per-view experience is like without spending any extra money. Still, I suppose we can all be thankful Netflix is getting to cut their live event teeth on "alleged rapist vs convicted rapist" instead of something more important.
- marxisttemp 2y ago> alleged rapist vs convicted rapist And you’ll never guess which Presidential candidate they both support!
- benreesman 2y agoThis Serrano fight is just an insane display of excellence. If anyone was waiting for the main card to tune in, I recommend tuning in now.
- Waterluvian 2y agoI know nothing about boxing and this fight was just ridiculously impressive. I kept tuning out of the earlier fights. They felt like some sort of filler. I didn’t get the allure. But Taylor v Serrano was just obvious talent that even I could appreciate it.
- vFunct 2y agoSerrano was robbed!
- benreesman 2y ago[flagged]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- Waterluvian 2y agoWell said. People tuning in late missed out on the real main event. I have a feeling this Tyson fight will be a waste of staying up late and battling Netflix over.
- dools 2y ago[flagged]
- benreesman 2y ago[flagged]
- aaron695 2y ago[dead]
- sgarland 2y agoA friend and I, in separate states, found that it wouldn’t stream from TVs, Roku, etc. but would stream from mobile. And for me, using a mobile hotspot to a laptop; though that implies checking IP address range instead of just user-agent, so that seems unlikely. Anyway, I wouldn’t be surprised if they were prioritizing mobile traffic because it’s more forgiving of shitty bitrate.
- arcbyte 2y agoI just left a bar streaming it on a smart TV and back in my home it's streaming on the Roku just fine.
- sgarland 2y agoGuess I was looking for explanations too hard.
- anakaine 2y agoI wonder if this points to network peering and edge nodes. Mobile network vs cabled network likely being routed to different places.
- wonderwonder 2y agoIs this potentially an aws issue?
- spencerchubb 2y agoI would assume not because twitch runs on aws. I think netflix engineers haven't optimized as much for livestreaming like twitch has
- anshumankmr 2y agoI do not think twitch has the amount of concurrent users netflix might have had today morning for the fight.
- chevman 2y agoGuess they should have livestreamed it on X to be safe!
- kevinventullo 2y agoOr Facebook. Or YouTube. Or Vimeo. Or LiveLeak.
- spyda56 2y agoWorking okay for me
- voyagerfoil 2y agoOver promised and under delivered. That’s a bad look
- boppo1 2y agoI'm watching on a 'pirate' stream because my netflix stream is absolutely frozen.
- walrushunter 2y agoI'm an engineering manager at a Fortune 500 company. The dumbest engineer on our team left for Netflix. He got a pay raise too. Our engineers are fucking morons. And this guy was the dumbest of the bunch. If you think Netflix hires top tier talent, you don't know Netflix.
- Lammy 2y agoI hope to never have a manager who is mentally stack ranking me and my coworkers in terms of perceived dumbness instead of in terms of any positive trait.
- geodel 2y agoBtw how do you know your current manager is not doing that.
- bryanrasmussen 2y agodumbness is ranking intelligence, which is a positive trait, dumbness is just a metric for how often intelligence fails. Example - the manager who started this sub-thread may be a pretty smart guy and able to accurately rate the intelligence of the engineers at his organization - but he had a minor momentary failing of intelligence to post on HN calling those engineers fucking morons. You've got to rank how often the intelligence fails in someone to be able to figure out how reliable their intelligence is.
- JamesBarney 2y agoAlmost everyone I know manager or not is usually ranking everyone they work with on various attributes. In fact it would be incredibly weird to ask a close friend who at their work kicks ass and who sucks and have them respond back, "I've never really thought about how good any of my coworkers were at their jobs"
- harimau777 2y agoCurrently trying to watch it and it's not loading at all for me. Re-subscribed specifically for the fight.
- Jabbs 2y agoLooks like I’m playing Tysons Punchout right now
- selimthegrim 2y agoGlass Jake?
- ralph84 2y agoMain event hasn’t even started yet. Traffic will probably 10x for that. They’re screwed. Should have picked something lower profile to get started with live streaming.
- pythonguython 2y agoI don’t work in tech. Is this something that engineers could respond to and reallocate resources to fix mid stream?
- anothermoron 2y ago[dead]
- GauntletWizard 2y agoNot a chance. This level of infrastructure was set up days in advance - I would be unsurprised if they'd had a code freeze all week for this fight.
- tonygiorgio 2y agoHad issues all stream but was perfect during the final fight.
- deleted 2y ago[deleted]
- jdub 2y agoThey've done quite a bit of lower profile live streams... various events, and the Everybody's in LA chat show series.
- dools 2y agoCan Mike Judge please stop predicting everything?
- RaftPeople 2y agoI've been re-watching Silicon Valley the last few weeks and just watched the Nucleus live stream episode 2 days ago, pretty funny seeing it in real life.
- junglistguy 2y ago[dead]
- Jamie9912 2y agoWorks in Australia. Maybe their CDN is under a lot of stress?
- kalesh 2y agoIt's down permanently for me in India. We have Hotstar, which has a record of 58 million viewers during the cricket World Cup final. Way ahead.
- anakaine 2y agoProbably less about the level of advancement and more about their ability to stream vs play VOD. Two different kinds of infrastructure optimisation.
- ta1243 2y ago59 million at the world cup was concurrent live streams > https://www.icc-cricket.com/news/biggest-cricket-world-cup-ever-smashes-broadcast-and-digital-records https://www.icc-cricket.com/news/biggest-cricket-world-cup-e... An impressive achievement and the scale netflix failed to do a year later.
- margaretdouglas 2y agoWasn't that the biggest concurrent stream ever?
- jameson 2y agoDumb question Isn't live streaming at scale already solved problem by cable companies? I never seen ESPN going down during a critical event
- ilrwbwrkhv 2y agoYes, as I have said again and again on hacker news in different comments Netflix went overboard with their microservices and tried to position itself as a technological company when it's not. It has made everything more complex and that's why any Netflix tech blog is useless because it is not the way to build things correctly. To understand how to do things correctly look at something like pornhub who handle more scale than Netflix without crying about it. The other day I was having this discussion with somebody who was saying distributed counter logic is hard and I was telling them that you don't even need it if Netflix didn't go completely mental on the microservices and complexity.
- dilyevsky 2y agoThis is not the same streaming - netflix is doing that over HTTP. Totally different tech and scaling issues
- hmcq6 2y agoYes and no. It's not the "same" but this is a solved problem. Fastly regularly delivers the Super Bowl to 10x as many viewers. Netflix dropped the ball hard
- dilyevsky 2y agoFastly says they do 6M ccv for superbowl (i'm actually surprised they let them do the entire thing and don't mix different CDNs) and I'm not sure they do encoding and manifest serving - they might just cache/deliver chunks. Do you really think tyson vs other guy was only 600k ccv? I'd be shocked if Netflix can't handle this.
- bastard_op 2y agoYou would think, but technology always finds a way to screw things up. Cox Communications has had ongoing issues with their video for weeks because of Juniper router upgrades and even the vendor can't fix it. They found this out AFTER they put it in production. Shit happens.
- scrapcode 2y agoThey have absolutely shit the bed here, and of course their socials are completely ignoring it.
- quickslowdown 2y agoBet they wish they'd gone with middle out compression
- robertlagrant 2y agoWhen they come up with that idea it's the most 18-rated and accurate way an engineer would think about it.
- ksec 2y agoNetflix is good only on streaming ready made content, not live streaming, but; 1. Netflix is a 300B company, this isn't a resources issue. 2. This isn't the first time they have done live streaming at this scale either. They already have prior failure experience, you expect the 2nd time to be better, if not perfect. 3. There were plenty of time between first massive live streaming to second. Meaning plenty of time to learn and iterate.
- crowcroft 2y agoYea, the issue here isn't just that they're having issues, it's that they're having the same issues they've had before.
- ilrwbwrkhv 2y agoYou can't solve your way out of a complex problem that you have created and which wasn't needed in the first place. The entire microservices thing was overly complex with zero benefits I spoke to multiple Netflix senior technicians about this. They said that's the whole shtick.
- iLoveOncall 2y agoThat's a ridiculous statement. PrimeVideo is the leader in terms of sports events streaming over internet and it is composed of hundreds of microservices. Live streaming is just much harder than streaming, and it takes a years of work and a huge headcount to get something good.
- sgarland 2y agoPrime famously undid some amount of their microservices recently because it couldn’t keep up, and was hideously expensive.
- iLoveOncall 2y agoIt was a single team for a very specific use case. To be clear when I said that PrimeVideo is composed of hundreds of microservices, I actually meant that it's composed of hundreds of services, themselves composed, more often than not, of multiple microservices. Depending on your definition of a microservice, my team alone owns dozens.
- curiousDog 2y agoUtter incompetence from senior leadership at Netflix. They had so much time to prepare for this.
- PittleyDunkin 2y agoI want to index everyone sneering at this situation and never work with any of them.
- seizethecheese 2y agoEh, punching up, while still punching, doesn’t seem that distasteful to me.
- deleted 2y ago[deleted]
- robertlagrant 2y agoThere's no up. There's just punching, and making excuses for punching.
- kpw94 2y agoyep, especially knowing this isn't their first rodeo... 18 months since https://time.com/6272470/love-is-blind-live-reunion-netflix/ https://time.com/6272470/love-is-blind-live-reunion-netflix/ > But the real indicator of how much Sunday’s screw-up ends up hurting Netflix will be the success or failure of its next live program—and the next one, and the one after that, and so on. There’s no longer any room for error. Because, like the newly minted spouses of Love Is Blind, a streaming service can never stop working to justify its subscribers’ love. Now, Netflix has a lot of broken trust to rebuild.
- deleted 2y ago[deleted]
- purpleidea 2y agoI think they must be noticing the issues, because I've noticed they've been dropping the stream quality quite substantially... It's a clever trick, but kind of cheap to do so, because who wants to watch pixelated things?
- nomilk 2y agoTo be brutally honest if it’s a choice between pixelated and constantly buffering, pixelated is way less bad. Constantly buffering is incredibly annoying during live sports. (but this doesn’t negate your main point which is that if people paid to watch they expect decent resolution)
- nomilk 2y agoIt will never not annoy and amuse me that illegal options (presumably run by randoms in their spare time) are so much better than the offerings of big companies and their tech ‘talent’.
- Bilal_io 2y agoI have Netflix purchased legally with hard earned money. But because I had issues I looked for illegal streams, and they were bad, crashes, buffering.. you name it. So I went back to Netflix and watched it at 140p quality.
- samschooler 2y agoThis twitter stream was the most reliable for me. Completely took Netflix out of the equation; just some dude at the event with his phone: https://x.com/i/broadcasts/1mrxmMRmXpQxy https://x.com/i/broadcasts/1mrxmMRmXpQxy
- koolba 2y ago> just some dude at the event with his phone Antonio Brown is not “just some dude”. He’s a national treasure.
- bayarearefugee 2y agoIts a good thing he's rich and famous otherwise there might actually be consequences for him illegally broadcasting this.
- pbhjpbhj 2y agoIf he's rich then he's financially worth employing an IP lawyer to pursue for copyright infringement, if it was infringement. Maybe he was licensed?
- rk06 2y agoIllegal options would have lot less active users. So it is not a fair comparison
- owenpalmer 2y agoI'm watching the event as I'm writing this. I've been needing to exit the player and resume it constantly. Pretty surprising that Netflix hasn't weeded out these bugs.
- bluedino 2y agoIt's been fine since 11:00 EST, I wonder if they started using the CDN more effectively and pushed everyone back a few minutes?
- unsnap_biceps 2y agoMine just crashed and reloaded to "Envoy Overloaded"
- JoyfulTurkey 2y agoNetflix has some NFL games on Christmas Day. Wonder how those will go for them. I remember when ESPN started streaming years back, it was awful. Now I almost never have problems with their live events, primarily their NHL streams.
- zoklet-enjoyer 2y agoWow I feel scammed. I paid for a Netflix subscription specifically for this but it's not loading so I'm watching on an illegal streaming website
- djbusby 2y agoMine is glitchy, but if I refresh i get a good steam for a bit, then it gets low res, then freeze. If I wait for auto-reconnect it takes forever. Hard refresh and I'm good. Like, new streams to new server, then overloaded, then does as if their cluster is crashing and healing is rapid cycles. Sawtooth patterns on their charts. And then all these sessions lag, or orphan taking up space, so many reconnections at various points in the stream. System getting hammered. Can't wait for this writeup.
- magic_man 2y agoIllegal streams are working but netflix is not. That is crazy.
- gurjeet 2y ago> envoy overloaded That's the plain-text message I see when I tried to refresh the stream. Follow-up: My location: East SF Bay. Now even the Netflix frontpage (post login, https://www.netflix.com/browse https://www.netflix.com/browse ) shows the same message. The same message even in a private window when trying to visit https://www.netflix.com/browse https://www.netflix.com/browse The first round of the fight just finished, and the issues seem to be resolved, hopefully for good. All this to say what others have noted already, this experience does not evoke a lot of confidence in Netflix's live-streaming infrastructure.
- smitelli 2y agoAh, envoy. Now that is a name I have not missed.
- tinyhouse 2y agoIt's not lagging for me. It crashed and not coming back. Update: Switched to the app on my phone and so far so good.
- KingOfCoders 2y agoNot enough chaos monkey engineering.
- margaretdouglas 2y agoThere was some blog post on HN the other day where someone said they don't do chaos monkey anymore... Even then, how do you chaos test a novel event ahead of time?
- betaby 2y agoSounds like a scene from: Silicon Valley - Nucleus fails https://www.youtube.com/watch?v=9IGvzb-KCpY https://www.youtube.com/watch?v=9IGvzb-KCpY
- Sateeshm 2y agoMy kid woke me up complaining internet is not working. Turns out he is trying to watch the fight and it's not working at all here in India.
- blinded 2y agoIt’s a learning experience! I remember Conor and Floyd broke hbo and the ufc. It’s a hard problem for sure! Some buffering issues for us, but I bet views are off the charts. Huge for Netflix, bad for espn, paramount, etc etc
- everly 2y agoCable TV (or even OTA antenna in the right service area) is simply a superior live product compared to anything streaming. The Masters app is the only thing that comes close imo. Cable TV + DVR + high speed internet for torrenting is still an unmatched entertainment setup. Streaming landscape is a mess. It's too bad the cable companies abused their position and lost any market goodwill. Copper connection direct to every home in America is a huge advantage to have fumbled.
- damontal 2y agoApple TV MLB games look incredible compared to live cable tv.
- ericcholis 2y agoThe Masters app is truly incredible, I don't know if it gets enough praise.
- mmooss 2y agoWhat's so great about it?
- crote 2y agoThe interesting thing is that a lot of TV infrastructure is now running over IP networks. If I were to order a TV connection for my home I'd get an IPTV box to connect to my broadband router via Ethernet, and it'd simply tell the upstream router to send a copy of a multicast stream my way. Reliable and redundant multicast streaming is pretty much a solved problem, but it does require everyone along the way to participate. Not a problem if you're an ISP offering TV, definitely a problem if you're Netflix trying to convince every single provider to set it up for some one-off boxing match.
- nitwit005 2y agoI have wondered if better multicast support will happen just for cost savings, as the amount of live content increases. So far, no one seems particularly motivated.
- muddi900 2y agoI can't see the fight right now.
- gamblor956 2y agoAfter a few buffering timeouts during the first match, the rest of the event had no technical difficulties (in SoCal, so close to one of Netflix's HQs). Unfortunately, except for the women's match, the fights were pretty lame...4 of the 6 male boxers were out of shape. Paul and Tyson were struggling to stay awake and if you were to tell me that Paul was just as old as Tyson I would have believed it.
- shrubble 2y agoOn X.com someone had a stream that was stable to at least 5 million simultaneous viewers, but then (as I expected) someone at Netflix got them to pull the plug on it. So I would expect this fight to have say, 50 million + watching? Maybe as many as 150-250 million worldwide, given this is Tyson's last fight.
- wayoverthecloud 2y agoI watched the whole fight with a 2 minute delay. That was frustrating and it didn't help that Tyson lost.
- smegger001 2y agoWell he is almost 60 and the average retirement age for pro boxers is in the mid 30s. He is well past his prime and in physically demanding sport that is very hard on participants.
- Dem_Boys 2y agoWhat do you think were the dynamics of the engineering team working on this? I'd think this isn't too crazy to stress test. If you have 300 million users signed up then you're stress test should be 300 million simultaneous streams in HD for 4 hours. I just don't see how Netflix screws this up. Maybe it was a management incompetence thing? Manager says something like "We only need to support 20 million simultaneous streams" and engineers implement to that spec even if the 20 million number is wildly incorrect.
- margaretdouglas 2y agoHas there ever been a 300m concurrent live stream? I thought Disney+ had the record at something like 60m.
- JCharante 2y agoprobably an esports match hosted on bilibili
- markus92 2y agoWorld Cup final, if you add up all streams worldwide?
- ta1243 2y agoNot through a single system, the advantage of diversity rather than winner-takes-all. The world cup final itself (and other major events) is distributed from the host broadcaster to either on site at the IBC or at major exchange points. When I've done major events of that magnitude there's usually a backup scanner and even a tertiary backup. Obviously feeds get sent via all manner - the international feed for example may be handed off at an exchange point, but the reserve is likely available on satelite for people to downlink on. If the scanner goes (fire etc), then at least some camera/sound feeds can be switched direct to these points, on some occasions there's a full backup scanner too. Short of events that take out the venue itself, I can't think of a plausible scenario which would cause the generation or distribution of the broadcast to break on a global basis. I don't work for OBS/HBS/etc but I can't imagine they are any worse than other broadcast professionals. The IT part of this stuff is pretty trivial nowadays, even the complex parts like the 2110 networks in the scanner tend to be commoditised and treated as you'd treat any other single system. The most technically challenging part is unicast streaming to millions of people at low latency (DASH etc). I wouldn't expect an enormous architectural difference between a system that can broadcast to 10 million or 100 million though.
- iamjackg 2y agoJust adding a data point, here in Canada on my nVidia Shield it went down to 360p a dozen times or so, but never paused at all. I guess I got lucky.
- greenranger 2y ago[dead]
- jaarse 2y agoEveryone pointing out that their illegal streams, X streams, etc. work fine are kind of missing the point. These secondary streams might be serving a couple thousand users at best. Initial estimates are in the hundreds of millions for Netflix. Kind of a couple of orders of magnitude difference there.
- samatman 2y agoPiracy is distributed, yes. I think that is the point, in fact.
- robertlagrant 2y agoEverything's easy when it's someone else's content.
- samatman 2y agoI have to assume this is some snarky way of saying "violating copyright is Bad, m'kay". Because taken at face value it's false. Any technical challenges involved in distributing a stream cannot possibly be affected by the legal status of the bits being pushed across the network.
- robertlagrant 2y agoIt means that someone who spends 100% of their money on distribution is going to have an easier time than someone who pays for content and distribution.
- robertlagrant 2y agoThey're not used to live. I imagine that's it. All their caching infrastructure is there assuming the content isn't currently being generated.
- impulser_ 2y agoHopefully they fix it because they are hosting two Christmas NFL games this year and if you want to really piss people off you have buffering issues during NFL games lol.
- 0xpgm 2y agoMaybe this was a stress test for the NFL games? I'd expect the NFL games to have a largely American audience, but today's boxing event attracted a global audience.
- MuffinFlavored 2y agoI wonder what % of the "60-120m" viewers were from Europe vs America vs Asia
- uptownfunk 2y agoIt was so bad. So so bad. Like don’t use your customers as guinea pigs for live streaming. So lame. They need a new head of content delivery. You can’t charge customers like that and market a massive event and your tech is worse than what we had from live broadcast tv.
- subless 2y agoI ended up turning my TV off and watching from my phone because of the buffering/freezing. The audio would continue to play and the screen would be frozen with a loading percentage that never changed. I have Spectrum (600 Mbps) for ISP and Verizon for mobile.
- iamzycon 2y agoFrom my limited understanding, the NFL heavily depends on the Netflix Open Connect platform to stream media to edge locations, which is different from live streaming. Probably, they over-pushed the HD contents.
- odinthedog 2y agoDoes anyone have any thoughts besides "bad engineering" on what could've gone wrong? It seems like taking on a new endeavor like streaming an event that would possibly draw many hundreds of millions of viewers doesn't make sense. Is there any obvious way that this would just work, or is there obviously a huge mistake deeply rooted in the whole thing. Also, are there any educated guesses on some fine details in the codebase and patterns that could result in this?
- normie3000 2y agoDid anyone else see different behaviour with different clients? My TV failed on 25% loaded, my laptop loaded but played for a minute or two before getting stuck buffering, and my iphone played the whole fight fine. All on the same wifi network.
- rdtsc 2y agoLive streaming and streaming prerecorded movies is a whole different ballgame. In fact, optimizing for later can hurt the former. Would be interesting to read any postmortems on this failure. Maybe someone will be kind enough to share the technical details for the curious crowd.
- pests 2y agoAmazon had issues last year too when they started broadcasting TNF but its fine these days. I'm sure they will get it figured out.
- thr0waway001 2y agoShoulda used middle out compression.
- new_user_final 2y agoI am curious about their live streaming infrastructure. I have done live streaming for around 100k concurrent users. I didn't setup infrastructure because it was CloudFront CDN. Why it is hard for Netflix. They have already figured out CDN part. So it should not be a problem even if it is 1M or 100M. because their CDN infrastructure is already handling the load. I have only work with HLS live streaming where playlist is constantly changing compared to VOD. Live video chunks work same as VOD. CloudFront also has a feature request collapsing that greatly help live streaming. So, my question is if Netflix has already figured out CDN, why their live infrastructure failing? Note: I am not saying my 100k is same scaling as their 100M. I am curious about which part is the bottleneck.
- vitus 2y ago> Why it is hard for Netflix. They have already figured out CDN part. So it should not be a problem even if it is 1M or 100M. because their CDN infrastructure is already handling the load ... Note: I am not saying my 100k is same scaling as their 100M. I am curious about which part is the bottleneck. 100k concurrents is a completely different game compared to 10 million or 100 million. 100k concurrents might translate to 200Gbps globally for 1080p, whereas for that same quality, you might be talking 20T for 10 million streams. 100k concurrents is also a size such that you could theoretically handle it on a small single-digit number of servers, if not for latency. > CloudFront also has a feature request collapsing that greatly help live streaming. I don't know how much request coalescing Netflix does in practice (or how good their implementation is). They haven't needed it historically, since for SVOD, they could rely on cache preplacement off-peak. But for live, you essentially need a pull-through cache for the sake of origin offload. If you're not careful, your origin can be quickly overwhelmed. Or your backbone if you've historically relied too heavily on your caches' effectiveness, or likewise your peering for that same reason. 200Gbps is a small enough volume that you don't really need to provision for that explicitly; 20Tbps or 200Tbps may need months if not years of lead time to land the physical hardware augments, sign additional contracts for space and power, work with partners, etc.
- junior44660 2y agoThis is probably a naive question but very relevant to what we have here. In a protocol where a oft-repeated request goes through multiple intermediaries, usually every intermediate will be able to cache the response for common queries (Eg: DNS). In theory, ISPs would be able to do the same with the HTTP. Although I am not aware of anyone doing such (since it will rightfully raise concerns of privacy and tampering). Now TLS (or other encryption) will break this abstraction. Every user, even if they request a live stream, receives a differently encrypted response. But live stream of a popular boxing match has nothing to do with the "confidentiality" of encryption protocol, only integrity. Do we have a protocol which allows downstream intermediates eg ISPs to cache content of the stream based on demand, while a digital signature / other attestation being still cryptographically verified by the client?
- dboreham 2y agoWhat you describe is called a CDN and has been widely used for 20 years.
- flas9sd 2y agothere's Named Data Networking (went by Content-Centric Networking earlier). You request data, not a url, the pipe/path becomes the CDN. If any of your nearest routers have the bytes, your request will go no further. I don't see it much mentioned the last few years, but the research groups have ongoing publications. There's an old 2006 Van Jacobson video that is a nice intro.
- ilaksh 2y agoI guarantee this is a management issue. Somebody needed to bear down at some point and put the resources into load testing. The engineers told them it probably won't be sufficient. I assume this came down to some technical manager saying they didn't have the human and server resources for the project to work smoothly and a VP or something saying "well, just do the best you can.. surely it will be at least a little better than last time we tried something live, right?" I think there should be a $20 million class action lawsuit, which should be settled as automatic refunds for everyone who streamed the fight. And two executives should get fired. At least.. that's how it would be if there was any justice in the world. But we now know there isn't -- as evidenced by the fact that Jake Paul's head is still firmly attached to his body.
- 2-3-7-43-1807 2y agoI thought it's only the best of the best of the best working at Netflix ... or maybe we can just put this myth to sleep that Netflix even knows what it's doing. The suggestions are shit, the UX is shit, apparently even the back end sucks.
- hawk_ 2y agoI thought Hooli was Google, but may be it was Netflix after all.
- skwee357 2y agoI’m very disappointed. Woke up at 4am (EU here), to tune for the main event. Bought Netflix just for this. The women fight went good, no buffering, 4K. As it approached the time for Paul vs Tyson, it started to first drop to 140p, and then constantly buffer. Restarted my chromecast a few times, tried from laptop, and finally caught a stream on my mobile phone via mobile network rather than my wifi. The TV Netflix kept blaming my internet which kept coming back as “fast”. Ended up watching the utterly disappointing, senior abuse, live stream on my mobile phone with 360p quality. Gonna cancel Netflix and never pay for it it again, nor watch hyped up boxing matches.
- brokenmachine 2y agoYou should ask for a refund if that's the only reason you paid for it.
- hiyer 2y agoOn a tangential note, the match totally looked fixed to me - Tyson was barely throwing any punches. I understand age is not on his side, but he looked plenty spry when he was ducking, weaving and dodging. It seemed to me he could have done better in terms of attacking as well.
- adamtaylor_13 2y agoYeah the biggest thing to me, and the commentators mentioned this as well, his legs looked REALLY wobbly. All your attacking power comes from your legs and hips, so if his legs weren’t stable he didn’t have much attacking power. I think he gave it everything he had in rounds 1 and 2. Unfortunately, I just don’t think it was ever going to be enough against a moderately trained 27 year old.
- nubinetwork 2y agoI would argue Tyson has a shorter reach, Jake was whiffing a lot of superman punches, and all that does is waste energy. Jake might be able to throw punches, but he clearly wasn't interested in taking them. If they stood closer and slugged it out, the fight could have gone either way.
- nightowl_games 2y agoHow is this story not on the front page anymore? 375 comments. Seems like a big story to me.
- npilk 2y agoI believe HN's algorithm tends to relatively downrank stories with a high comment-to-upvote ratio, because they are more often flamewars on divisive topics.
- yodon 2y agoAnother major algorithmic down-ranker is vote wars on comments. If lots of people are upvoting and downvoting the same comments, that's treated as a signal the topic is contentious and people are likely to start behaving badly. HN is very clear they prioritize good behavior as the long term goal, and they are as a result comfortable having contentious topics fall in the ranking even if everyone involved in the discussion feels the topic is important.
- ajdude 2y agoIt wasn't even just buffering issues, the feed would just stop and never start again until I paused it and then clicked "watch live" with the remote. It was really bad. My Dad has always been a fan of boxing so I came over to watch the whole thing with him. He has his giant inflatable screen and a projector that we hooked up in the front lawn to watch it, But everything kept buffering. We figured it was the Wi-Fi so he packed everything up and went inside only to find the same thing happening on ethernet. He was really looking forward to watching it on the projector and Netflix disappointed him.
- mmooss 2y ago> My Dad has always been a fan of boxing What did your Dad think about the 'boxing'?
- manquer 2y agoCommercial boxing has always been like WWE or MMA with a thin veneer of actual sport to it, i.e. it is just entertainment[1]. To rephrase your question then what does someone think of the entertainment on display? I don't think it was good entertainment. None of the hallmarks of a good show was present. i.e. It wasn't close, nor was it bloody or anything unexpected like say a KO everything went pretty much as expected. It wasn't nice watch as all,no skill or talent was on disply, all Paul had to do was use his speed to backpedal from the slow weak punches of a visibly older tyson with a bum knee and land some points occasionally to win. -- [1] There is a deeper argument here is any spectator sports just entertainment or is truly about skill talent and competition. Boxing however including the ones promoted by traditional four major associations falls clearly on the entertainment side than say another sport like NFL to me.
- mvandermeulen 2y agoWas this necessary? The comment was on a tech forum about the tech issues, do we really need to reprosecute the argument that it wasn’t real boxing here too? There are plenty of other places for those so painfully inclined to do so
- 2y ago
- beanjuiceII 2y agoi thought they did DSA interviews at netflix what happened? I had to watch the fight on someone streaming to X from their phone at the event and it was better than watching on netflix..if you could watch at all. extremely embarrassing!
- bobobob420 2y agoIts because they use medium-hard leetcode instead of hard. I suggest 8 rounds instead of 4
- taneq 2y agoSurely more whiteboard questions are the key? Reversing a binary tree is so last year, they should make candidates reverse an octree.
- BowBun 2y agoMy theory is they've so heavily optimized for static content and distributing content on edge nodes that they were probably poorly setup for live-streaming.
- moritonal 2y agoThis, I feel bad for their engineers who were told every bonus would be a matter of how low can they get the cost-per-GB of transferred data, leading to the glorious Netflix-in-a-box (https://openconnect.netflix.com/deploymentguide.pdf https://openconnect.netflix.com/deploymentguide.pdf) and then management casually asks for a live show with a wildly different set of guarantee requirements.
- gosub100 2y agosomeone's going to have to reverse a linked list of subscriptions.
- sylens 2y agoReminds me of Nucleus stuttering during UFC
- xerox13ster 2y agoI could hear Gavin Belson screaming during the broadcast when my stream was freezing as they were each making their entrance. Mike Judge is a prophet.
- r_hanz 2y agoThat show ages better every single day.
- criddell 2y agoI watched on an AppleTV and the stream was rock solid. I don’t know if it’s still the case, but in the past some devices worked better than others during peak times because they used different bandwidth providers. This was the battle between Comcast and Cogent and Netflix.
- eqvinox 2y agoYour device type has no influence on your provider and its bandwidth characteristics. If you're on Comcast, Apple can't magically make it not suck.
- fidotron 2y agoNative apps have a lot more scope for client side load balancing due to having a different security model than browsers.
- immibis 2y agoIf you're on Apple, Comcast can make it magically not suck. Not sure if it's relevant though. Cogent just seems to love picking fights with everyone (see Hurricane Electric). Why are they still in business?
- criddell 2y agoIt isn't Apple, it's Netflix. Remember back in 2014 or so when Netflix users on Comcast were getting slow connections and buffering pauses? It didn't affect people who watched Netflix via Apple TV because Netflix served Apple TV users with a different network. > In a little known, but public fact, anyone who is on Comcast and using Apple TV to stream Netflix wasn’t having quality problems. The reason for this is that Netflix is using Level 3 and Limelight to stream their content specifically to the Apple TV device. What this shows is that Netflix is the one that decides and controls how they get their content to each device and whether they do it via their own servers or a third party. Netflix decides which third party CDNs to use and when Netflix uses their own CDN, they decide whom to buy transit from, with what capacity, in what locations and how many connections they buy, from the transit provider. Netflix is the one in control of this, not Comcast or any ISP. https://www.streamingmediablog.com/2014/02/heres-comcast-netflix-deal-structured-numbers.html https://www.streamingmediablog.com/2014/02/heres-comcast-net...
- booleandilemma 2y agoWas this their first time doing live content? I figured something would go wrong. I'm sure lots of people were watching.
- l33t7332273 2y agoI thought Netflix’s biggest advantage was the quality/salary of its engineers. I think that every time I wait for Paramount+ to restart after its gone black in picture on picture, and yet, I’n still on Paramount+ and not Netflix, so maybe that advantage isn’t real.
- A4ET8a8uTh0 2y agoSigh, none of the competitors are much better. Disney, who has more than enough cash to throw at streaming, is a near constant hassle for us ( after 3 or more episodes it throws an inscrutable error on Playstation ). I would drop it, but this is the only remaining streaming service and wife is not willing to drop it ( I guess until 1 it is one error per one episode ).
- immibis 2y agoI keep hearing that a lot of people are switching to the illegal competitor for better service.
- deleted 2y ago[deleted]
- A4ET8a8uTh0 2y agoEh, the beef that I have is that I am already a paying customer. Why does it seem like I am getting subpar service in terms of delivery? I know it is a tired conversation on this forum, but corporations big and small do what they can do mess with experience to eke out few more cents from customers. It almost does not matter which industry one looks at, it is the same story; the only difference is typically how egregious things get[1]. [1]https://www.marketwatch.com/story/i-opt-to-fly-private-no-matter-what-the-cost-the-caviar-in-first-class-tastes-like-burnt-plastic-whatever-happened-to-airline-standards-6f5bcd47 https://www.marketwatch.com/story/i-opt-to-fly-private-no-ma...
- IncreasePosts 2y agoDo you think YouTube couldn't handle it?
- J05ephu5M13r 2y ago[dead]
- dylan604 2y agoPeople just do not appreciate how many gotchas can pop up doing anything live. Sure, Netflix might have a great CDN that works great for their canned content and I could see how they might have assumed that's the hardest part. Live has changed over the years from large satellite dishes beaming to a geosat and back down to the broadcast center($$$$$), to microwave to a more local broadcast center($$$$), to running dedicated fiber long haul back to a broadcast center($$$), to having a kit with multiple cell providers pushing a signal back to a broadcast center($$), to having a direct internet connection to a server accepting a live http stream($). I'd be curious to know what their live plan was and what their redundant plan was.
- diggan 2y ago> People just do not appreciate how many gotchas can pop up doing anything live. Sure thing, but also, how much resources do you think Netflix threw on this event? If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) across the globe without major hiccups on (relatively) tiny budgets and smaller infrastructure overall, how could Netflix not?
- phyrex 2y agoScale changes everything, I don't think it's fair to shrug this off
- tiluha 2y agoThis is true, but scale comes after production. Once you have the video encoded on a server with a stable connection the hard part is over. What netflix failed to do is spread the files to enough servers around the globe to handle the load. I'm surprised they were unable(?) to use their network of edge servers to handle the live stream. Just run the stream with a 10 second delay and in that time push the stream segments to the edge server
- dylan604 2y agoThis right here is where I'd expect the failure to occur. This isn't Joey Beercan running OBS using their home internet connectivity. This is a major broadcast. I'd expect a full on broadcast truck/trailer. If they were attempting to broadcast this with the ($) option directly to a server from onsite, then I would demand my money back. Broadcasting a live IP signal just falls on its face so many times it's only the cheap bastard option. Get the video signal as a video signal away from the live location to a facility with stable redundant networking. This is the kind of thinking someone only familiar with computers/software/networking would think of rather than someone in broadcasting. It's nice to think about disrupting, but this is the kind of failure that disruptors never think about. Broadcasters have been there done that with ensuring live broadcasts don't go down because an internet connection wasn't able to keep up.
- _fat_santa 2y agoWhen you step back and look at the situation, it's not hard to see why Netflix dropped the ball here. Here's now I see it (not affiliated with Netflix, pure speculation): - Months ago, the "higher ups" at Netflix struck a deal to stream the fight on Netflix. The exec that signed the deal was probably over the moon because it would get Netflix into a brand new space and bring in large audience numbers. Along the way the individuals were probably told that Netflix doesn't do livestreaming but they ignored it and assumed their talented Engineers could pull it off. - Once the deal was signed then it became the Engineer's problem. They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events that you don't really have to think about when streaming static content. - Engineering probably did their absolute best to pull this off but they had two main disadvantages, first off they don't have any of the institutional knowledge about live streaming and they don't really know how to predict demand for something like this. In the end they probably beefed up livestreaming as much as they could but still didn't go far enough because again, no one there really knows how something like this will pan out. - Evening started off fine but crap hit the fan later in the show as more people tuned in for the main card. Engineering probably did their best to mitigate this but again, since they don't have the institutional knowledge of live events, they were shooting in the dark hoping their fixes would stick. Yes Netflix as a whole screwed this one up but I'm tempted to give them more grace than usual here. First off the deal that they struck was probably one they couldn't ignore and as for Engineering, I think those guys did the freaking best they could given their situation and lack of institutional knowledge. This is just a classic case of biting off more than one can chew, even if you're an SV heavyweight.
- deleted 2y ago[deleted]
- grapesodaaaaa 2y agoI hope they do a postmortem
- yarrowy 2y agothey should also do a business postmortem, how did the exec greenlight this without livestreaming infrastructure in place?
- ldjkfkdsjnv 2y agoWe all know netflix was built for static content, but its still hilarious that they have thousands of engineers making 500-1M in total comp and they couldnt live stream a basic broadcast. You probably could have just run this on AWS with a CDK configuration and quota increase from amazon
- cryptozeus 2y agoEveryone here talking like this something unique netflix had to deal with. Hotstar live streamed india va Pakistan cricket match with zero issues with all time high live viewership ever in the history of live telecast. Why would viewers paying $20 month want to think about their technical issues, they dropped the ball pure and simple. Tech already exists for this, it’s been done before even by espn, nothing new here.
- al_borland 2y agoThe Independent reports 35m viewers of that cricket match [0]. Rolling Stone reported 120m for Tyson and Paul on Netflix [1]. These are very different numbers. 120m is Super Bowl territory. Could Hotstar handle 3-4 of those cricket matches at the same time without issue? [0] https://www.the-independent.com/sport/cricket/india-pakistan-world-cup-hotstar-record-fans-b2430423.html https://www.the-independent.com/sport/cricket/india-pakistan... [1] https://www.rollingstone.com/culture/culture-news/jake-paul-mike-tyson-boxing-win-1235166434/ https://www.rollingstone.com/culture/culture-news/jake-paul-...
- dilyevsky 2y agoMajority of superbowl viewers watch it on cable. Streaming gets fewer than 10M concurrents
- firesteelrain 2y agoUntil it switches entirely to Netflix one day
- DaSHacka 2y agoGoing by their failure this time around, that's not happening anytime soon.
- SoftTalker 2y agoDo people even have cable TV anymore? I have internet from my "cable" company but I don't have the "cable" connected to anything but the modem. Everything I watch is streamed. The only thing connected to my TV is a Roku.
- colesantiago 2y agoIf you're going to be having intense algorithm interviews, paying top dollar for only hiring senior engineers, building high intensity and extreme distributed systems and having SRE engineers, we best see insanely good results and a high ROI out of it. All of the conditions was perfect for Netflix, and it seems that the platform entirely flopped. Is this what chaos engineering is all about that Netflix was marketing heavily to engineers? Was the livestream supposed to go down as Netflix removed servers randomly?
- nova22033 2y agoAmazon prime streams the Thursday night NFL game and they seem to have no problem.
- margaretdouglas 2y agoIsn't the scale a bit different though? Surely this event was an order of magnitude more concurrent viewers than some NFL game.
- jarsin 2y agoI blame RTO and AI
- ironhaven 2y agoThis is why we need ipv6. If ipv6 was fully rolled out this livestream could have been an efficient multicast stream like what happens with ipTV.
- BrandoElFollito 2y agoOne similar crash I remember very well was CNN on 9/11 - I tired to connect from France but is down the whole day. Since then I am very used to it because our institutional web sites traditionally crash when there is a deadline (typically the taxes or school inscriptions). As for that one, my son is studying in Europe (I am also in Europe), he called me desperate at 5 am or so to check if he is the only one with the problem (I am the 24/7 family support for anything plugged in). After having liberally insulted Netflix he realized he confirmed with his grandparents that he will be helping them at 10 :)
- softwaredoug 2y agoThe way to deal with this is to constantly do live events, and actually build organizational muscle. Not these massive one off events in an area the tech team has no experience in.
- _dark_matter_ 2y agoAgreed. This is a management failure, full stop. Unbelievable that they'd expect engineering to handle a single Livestream event of this magnitude.
- mbrumlow 2y agoI have this argument a lot in tech. We should always be doing (the thing we want to do) Somme examples that always get me in trouble (or at least big heated conversations) 1. Always be building: It does not matter if code was not changed, or there has been no PRs or whatever, build it. Something in your org or infra has likely changed. My argument is "I would rather have a build failure on software that is already released, than software I need to release". 2. Always be releasing: As before it does not matter if nothing changed, push out a release. Stress the system and make it go through the motions. I can't tell you how many times I have seen things fail to deploy simply because they have not attempted to do so in some long period of time. There are more just don't have time to go into them. The point is if "you did it, and need to do it again ever in the future, then you need to continuously do it"
- simple10 2y agoI watched the event last night and didn't get any buffering issues, but I did notice frequent drop in video quality when watching the live feed. If I backed the video up a bit, the video quality suddenly went back up to 4k. I had some technical experience with live video streaming over 15 years ago. It was a nightmare back then. I guess live video is still difficult in 2024. But congrats to Jake Paul and boxing fans. It was a great event. And breaking the internet just adds more hype for the next one.
- lunatuna 2y agoI wonder how localized the issues were. I watched the Taylor/Seranno fight and the Paul/Tyson without issue and the picture quality was the best in every seen for live sports. Was blown away by how good it was. No where near what I’m getting with steaming NFL. This is what I want the future of live sports to look like. Though the commentary was so so. I’m in the Pacific Northwest. I wonder if we got lucky on this or just some areas got unlucky.
- ctvo 2y agoIt’s insane the excuses being made here for Netflix’s apparently unique circumstances. They failed. Full stop. There is no valid technical reason they couldn’t have had a smooth experience. There are numerous people with experience building these systems they could have hired and listened to. It isn’t a novel problem. Here are the other companies that are peers that livestream just fine, ignoring traditional broadcasters: - Google (YouTube live), millions of concurrent viewers - Amazon (Thursday Night Football, Twitch), millions of concurrent viewers - Apple (MLS) NBC live streamed the Olympics in the US for tens of millions.
- csallen 2y ago> They failed. Full stop. It's not full stop. There are reasons why they failed, and for many it's useful and entertaining to dissect them. This is not "making excuses" and does not get in the way of you, apparently, prioritizing making a moral judgment.
- y-c-o-m-b 2y agoAmazon had their fair share of livestream failures and for notably less viewers. I don't think they deserve a spot on that list. I briefly worked in streaming media for sports and while it's not a novel problem, there are so many moving parts and points of failure that it can easily all go badly.
- deanCommie 2y agoThere is no one "Amazon" here, there are at least 3: * Twitch: Essentially invented live streaming. Fantastic. * Amazon Interactive Video Service [0]: Essentially "Twitch As A Service", built by Twitch engineers. Fantastic. * Prime Video. Same exact situation as Netflix: original expertise is all in static content. Lots of growing pains with live video and poor reports. But they've figured it out: now there are regular live streams (NHL and NFL), and other channel providers do live streaming on Prime Video as a distribution platform. [0] https://aws.amazon.com/ivs/ https://aws.amazon.com/ivs/
- margaretdouglas 2y ago
- JSDevOps 2y agoWhy didn’t they use Netflix AI to solve the problems?
- frankharv 2y agoHow dare you insult the AI Gods
- aucisson_masque 2y agoHonestly you didn't miss much, every (real) boxing fan thought of this as a disgrace and a shame when announced. putting a 58 year old Tyson against a crackhead filled with steroids (Jake Paul) ? Either case it would have been a shame on Jake Paul for even getting in the ring with such an old boxer. In boxing you are old by 32 or maybe 35 year old for heavy weight, and everything goes down very very fast. End of rant.
- chx 2y agoPeople still pay real world money to Netflix after they cancelled and how and why Warrior Nun just to see grandpa being beaten up. I guess in the year when Trump is being reelected this is hardly a surprise.
- bloomingeek 2y agoServes Netflix right for killing my beloved DVD rentals.
- duxup 2y agoI’m a little amused at folks tuning in for meme / low quality personalities doing things … and getting the equivalent production values.
- eigenvalue 2y agoThey should have partnered with every major CDN and load balanced across all of them. It’s ironic how we used to be better at broadcasting live events way back in the day versus today.
- badgersnake 2y agoI’m not sure buffering was the biggest issue with this event. How was as 58 year old Tyson fighting a man in his 20s?
- xyst 2y agoThis wasn’t a “real” fight in the ring. It was clearly a hype/money fight only. The late 20 year old boxer has a massive following (or hate following) with younger age groups; and Mike Tyson brings the older age groups out. Mike has earned somewhat of a legendary status. Leading up to the fight, there were many staged interactions meant to rile up the audience and generate hype and drive subscription revenue, and subsequently make ad spots a premium ($$$). Unfortunately, American television/entertainment is so fake. I can’t get even be bothered to engage or invest time into it anymore.
- phatfish 2y agoIt may as well have been a WWE special. As good as scripted. It was a business venture not a competitive fight.
- xyst 2y agoWhat a massive blow to NFLX. They have been in the streaming game for years (survived COVID-19) and this silly exhibition match is what does them in? I didn’t watch it live (boxing has lot its allure for me) but vicariously lived through it via social feed on Bluesky/Mastadon. Billions of dollars at their disposal and they just can’t get it right. Probably laid off the highly paid engineers and teams that made their shit work.
- wannacboatmovie 2y ago> Probably laid off the highly paid engineers and teams that made their shit work. More likely overpaying a bunch of posers without the chops, a victim of their own arrogance.
- thesausageking 2y agoIt's not a "massive blow" at all. Consumers will only vaguely remember this in a month. Netflix got a lot of new signups and got to test out their streaming infrastructure to figure out what needs work. The fight itself was lame which worked in their favor. No one really cared about not being able to see every second of the "action". It's not like it was an NBA game that came down to the last second.
- Thaxll 2y agoIn 2012 Youtube did the Red Bull stratos live stream with 8m concurrent users. We're 12 years later, Netflix fucked up.
- MichaelZuo 2y agoThe average quality of talent has gone way down compared to 2012 though. E.g. the median engineer, excluding entry level/interns, at YouTube in 2012 was a literal genius at their niche or quite close to it. Netflix simply can’t hire literal geniuses with mid six figure compensation packages in 2024 dollars anymore… though that may change with a more severe contraction.
- spaceywilly 2y agoTo me the difference is that in 2012, you had companies focusing on delivering a quality product, whether it made money or not. Today, the economic environment has shifted a lot and companies are trying to increase profits while cutting costs. The result is inevitably a decline in quality. I'm sure that Netflix could deliver a flawless live stream to millions of viewers, but the question is can they do it while making a profit that Wall Street is happy with. Apparently not.
- Mistletoe 2y agoThe funny thing is I was just reading something on HN like three days ago about how light years ahead Netflix tech was compared to other streaming providers. This is the first thing I thought of when I saw the reports that the fight was messing up.
- lxgr 2y agoBut is there a way that Netflix might have learned from all of Youtube's past mistakes? The only reasonable way to scale something like this up is probably to... scale it up. Sure, there are probably some generic lessons, but I bet that the pain points in Netflix's architecture (historically grown over more than a decade and optimized towards highly cacheable content) are very different from Youtube, which has ramped up live content gradually over as many years.
- SoftTalker 2y ago
- throwawayUS9 2y agoWhy no one mentioned the term “vaporware”? Isn’t this a classic example of one?
- sourcecodeplz 2y agoThis livestream broke the internet, no joke. youtube was barely loading and a bunch of other sites too. 130M is a conservative number given all the pirate streams.
- SG- 2y agodon't confuse your ISP breaking with every other provider or the rest of the Internet. It was more than fine here.
- brokenmachine 2y agoMy internet was fine.
- m3kw9 2y agoI would have just made it simple, delay the live stream a few seconds and encode it into the same bucket where users already is playing static movies. Just have the player only allow start at the time everyone is at.
- vouwfietsman 2y agoReading the comments here, I think one thing that's overlooked is that Netflix, which has been on the vanguard of web-tech and has solved many complicated problems in-house, may not have had the culture to internally admit that they needed outside help to tackle this problem. A combination of hubris and groupthink.
- kaptainscarlet 2y agoNot invented here syndrome works at first but as time progresses the internally built tools become a liability
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- nikolay 2y agoThe arrogant Netflix! They always brag about how technologically superior they are, and they can't handle a simple technological challenge! I didn't have a buffering issue, I had an error page - for hours! Yet, they kept advertising the boxing match to me! What a joke! If you can't stream it, don't advertise it to save face with people like me who don't care about boxing!
- notimetorelax 2y agoI think you’re oversimplifying it. Live event streaming is very different from movie streaming. All those edge cache servers become kinda useless and you start hitting peering bottlenecks.
- YZF 2y agoEdge caches are not useless for live streaming. They're critical. The upstream from those caches has no way of handling each individual users. The stream needs to hit the edge cache and end users should be served from there. A typical streaming architecture is multi-tiered caches, source->midtier->edge. We don't know what happened but it's possible they ran out of capacity on their edge (or anywhere else).
- ta1243 2y agoBBC had a similar issue in a live stream 5 years ago where events conspired and a CDN "failed open", which effectively DOSsed the entire output via all CDNs > Even though widely used, this pattern has some significant drawbacks, the best illustration being the major incident that hit the BBC during the 2018 World Cup quarter-final. Our routing component experienced a temporary wobble which had a knock-on effect and caused the CDN to fail to pull one piece of media content from our packager on time. The CDN increased its request load as part of its retry strategy, making the problem worse, and eventually disabled its internal caches, meaning that instead of collapsing player requests, it started forwarding millions of them directly to our packager. It wasn’t designed to serve several terabits of video data every second and was completely overwhelmed. Although we used more than one CDN, they all connected to the same packager servers, which led to us also being unable to serve the other CDNs. A couple of minutes into extra time, all our streams went down, and angry football fans were cursing the BBC across the country. https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Finternet%2Fentries%2Ffe629b38-57af-49c8-8922-352174101511 https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk...
- gunapologist99 2y agoNucleix needs to focus on fixing middle-out compression instead of kicking cameras.
- RyeCombinator 2y agoChaos testing, nothing to see here.
- pchwalek 2y agoSounds like a job for Pied Piper
- _HMCB_ 2y agoThey didn’t miss anything.
- manav 2y agoIt seemed to be some capacity issue with the CDNs. When I stopped and restarted the stream it worked again. Perhaps they do not use real time multi-cdn switching.
- gardenhedge 2y agoSo much for Netflix engineering talent aura
- sporkland 2y agoWas this the plot of a silicon valley episode?
- abc-1 2y agoI thought Netflix engineers were the best and could even do mythical leetcode hards. What happened? Why are they paid half a million dollars a year?
- thrw42A8N 2y agoIsn't this more of a management problem, trying to turn a not-livestream system into a livestreaming one?
- throwaway106382 2y agoIt's incomprehensible to me that Netflix, one of the most highly skilled engineering teams in the world - completely sh*t the bed last night and provided a nearly unwatchable experience that was not even in the same league as pre-internet live broadcast from 30 years ago.
- ilaksh 2y agoMy bet is that a technical manager told his executive (multiple times) that he needed more resources and engineering time to make live work properly, and they just told him to make do because they didn't want to spend the money. It could come down to something as stupid as: Executive: "we handled [on demand show ABCD] on day one, that was XX million" Engineering: "live is really different" Executive: (arguing about why it shouldn't be that different and should not need a lot of new infrastructure) Engineering: (can't really argue with his boss about this anymore after having repeated the same conversation 3 or 4 times) -- tells the team: we are not getting new servers or time for a new project. We have to just make do with what we have. You guys are brilliant, I know you can do it!"
- deleted 2y ago[deleted]
- throwaway106382 2y agohopefully their stock takes a big hit monday - these types only understand one thing
- lizknope 2y agoDoes anyone remember IP multicast? I remember a lot of trade magazines in the late 1990's during the dot com boom talked about how important it would be. https://en.wikipedia.org/wiki/IP_multicast https://en.wikipedia.org/wiki/IP_multicast I never hear about it anymore. Is that because everyone wants to watch something different at their own time? Or is it actually working just fine now in the background? I see under the "Deployment" section it mentions IPTV in hotel rooms.
- grogenaut 2y agodumbing it down a bit: Imagine if anyone in your neighborhood could broadcast video and take up %N of the bandwidth to all of the routers in the neighborhood. Imagine this on your campus or at your office. This works for cable tv, as there's only 200 channels. You're just going to slurp up all of the bandwidth instead and maybe no one is watching the tv. Sure you get these black swan events that everyone wants to watch, but they're just that, really infrequent. So instead you have to provision capacity if on the interent to do big events like this. The upside is that you have a billion people able to point to point to another billion people instead of 30 companies who can send to everyone.
- DanAtC 2y agoMulticast isn't broadcast; bandwidth is only used on links with clients who have joined the group.
- shermantanktop 2y agoEvery time a big company screws up, there are two highly informed sets of people who are guaranteed to be lurking, but rarely post, in a thread like this: 1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine. 2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is and aren’t likely to publicly flog someone doing their same job based on uninformed speculation. They know their own systems are Byzantine and don’t look like what random onlookers think it would look like. So that leaves the rest, who offer insights based on how stuff works at a small scale, or better yet, pronouncements rooted in “first principles.”
- grogenaut 2y agoyou are so right about that. tho I'm sure that many of the netflix folks are still doing their after action analysis in prep for Dec 25 NFL. now take this realization and apply it to any news article or forum post you read and think about how uninformed they actually are.
- fragmede 2y agoIf NFL decides to keep Netflix for that, that is. The bandwidth for that fight was rookie numbers, and after that fiasco, why would the NFL not break their contract and choose someone with a proven track record doing bigger live events, like the World Cup?
- phil21 2y agoBecause Netflix pays them either way, I would imagine. Breaking a contract on a sure thing to the tune of tens (hundreds?) of millions of dollars for a maybe is a large business risk. Reputational damage is going to be far more Netflix than the NFL if they totally club it. That and this fight is going to likely be an order of magnitude more viewers than the Christmas NFL games if the media estimates on viewership were remotely accurate. You’re talking Super Bowl type numbers vs a regular season NFL game. The problems start happening at the margin of capacity most of the time.
- grogenaut 2y agoThis topic is really just fun for me to read based on where I work and my role. Live is a lot harder than on demand especially when you can't estimate demand (which I'm sure this was hard to do). People are definitely not understanding that. Then there is that Netflix is well regarded for their engineering not quite to the point of snobbery. What is actually interesting to me is that they went for an event like this which is very hard to predict as one of their first major forays into live, instead of something that's a lot easier to predict like a baseball game / NFL game. I have to wonder if part of the NFL allowing Netflix to do the Christmas games was them proving out they could handle live streams at least a month before. The NFL seems to be quite particular (in a good way) about the quality of the delivery of their content so I wouldn't put it past them.
- marchie_uk 2y ago> The NFL seems to be quite particular (in a good way) about the quality of the delivery of their content Alas, my experience with the NFL in the UK does not reflect that. DAZN have the rights to stream NFL games here, and there are aspects of their service that are very poor. My major, long-standing issue has been the editing of their full game “ad-free” replays - it is common for chunks of play to be cut out, including touchdowns and field goals. Repeated complaints to DAZN haven’t resulted in any improvements. I can’t help but think that if the NFL was serious about the quality of their offering, they’d be knocking heads together at DAZN to fix this.
- grogenaut 2y agoI don't think they think this is a problem actually. Content edited replays are actually very popular with sports fans who are time shifting. Time shifting is also an afterthought for the NFL / MLB / NHL from what I can tell. I live in Seattle but grew up in the midwest so time shift a ton of sports and it's always been horrific. I'm more comparing Thursday Night Football and the quality of the encoding than anything. Delivery glitches are a seperate issue that I think they care about less. NFL: 90+ minutes after the match on NFL Gameday, and it auto plays the most recent video for that team, which is always the post game interview. So you load it up, go to your team and it auto plays the "we won" or "it was a tough loss" like why the f*ck am I paying for a dvr solution when you do that. NFL Sunday Ticket: you can watch the games sometime monday after the fact but not the sunday night games. Good thing I paid well below half price for it with a disciount. NHL: constantly shifting networks each year with worse solutions and not letting you get to half the previous games after a week. Totally useless for deferred unless you only want to watch the game a day or more after. Fubo, you have to 'record' the game and sometimes it's on a slightly different network and doesn't record. And their blackout system is the worst of all, who cares about your mediocre local team sorry you can't watch Chefs/Bills because they overlapped by some amount. MLB: always broken at the top of the year, constantly changing the interface. You often get stuck watching the commercial break which is not actually commercials and is just the same "ohtani / judge highlight video from 2 years ago" and a "stat" about the sluggers that is almost an entire season out of date. The app resets when switching from the live CDN to the on demand one once the game ends which often resets the game and jumps you 6 innings forward, or makes the game unavialable for 30 minutes. And EFF you if you want to watch playoffs on any of them.
- db48x 2y agoIt probably depends more on the ISP than on Netflix. Engineers over in my ISP’s subreddit are talking about how flows from Netflix jumped by over 450Gb/s and it was no big deal because it wasn’t enough to cause any congestion on any of their backbone links.
- heraldgeezer 2y agoEven people on hacker news do not understand. Internet live streaming is harder than cable tv sattelite live streaming over "dumb" TV boxes cable. They should not have used internet for this honestly. A TV signal can go to millions live.
- tiffanyh 2y agoIPTV A see in the comments multiple people talking about how "cable" companies who have migrated to IPTV has solved this problem. I'd disagree. I'm on IPTV and any major sporting event (World Series, Super Bowl, etc) is horrible buffering when I try to watch on my 4K IPTV (streaming) channel. I always have to downgrade to the HD channel and I still occasionally experience buffering. So Netflix isn't alone in this matter.
- more_corn 2y agoStreaming is hard.
- j45 2y agoStreaming live can be a very different thing than on-demand. Hopefully Netflix can share more about what they learned, I love learning about this stuff. I had joked I would probably cancel Netflix after the fight.. since I realized other platforms seemed to have more content both old and new. Then the video started stuttering.
- ravenstine 2y agoHow is this not a solved problem by now? I think this is a result of most software "engineering" having become a self-licking ice cream cone. Besides mere scaling, the techniques and infrastructure should be mostly squared away. Yes, it's all complicated, but I don't think we should excuse ourselves when we objectively fail at what we do. I'm not saying that Netflix developers are bad people, but that it doesn't matter how hard of a job it is; it was their job and what they did was inadequate to say the least. Jonathan Blow is right.
- dilap 2y agoTechnical issues happen, but I wish they would've put up a YouTube stream or something (or at least asked YouTube to stop taking down the indie streams that were popping up). It seems like basically their duty to the boxers and the fans to do everything in their power to let the match be seen live, even if it means eating crow and using another platform.
- chupchap 2y agoLive streaming is hard. Most companies that do live streaming at 2024 scale did it by learning from their mistakes. This is true for Hotstar, Amazon and even Youtube. Netflix stack is made to stream optimised, compressed , cached videos with a manageable concurrent viewers for the same video. Here we had ~65m concurrent viewers in their first live event. The compression they use, distribution etc have not scaled up well. I'll judge them based on how they handle their next live event
- latortuga 2y agoI don't think this is their first live event. They have hosted a pro golf promotional match and they had a live pro tennis match between Nadal and Alcaraz off the top of my head.
- jaarse 2y agoSeems like the magic number was 60 million concurrent streams https://www.theverge.com/2024/11/16/24298338/netflix-mike-ttyson-vs-jake-paul-fight-netflix-60-million-households-streaming-quality-buffering https://www.theverge.com/2024/11/16/24298338/netflix-mike-tt...
- ta1243 2y agoThat's a suspicious number, given the previous world record is 59 million.
- yusyusyus 2y agoI wrote an analysis on doing this kind of unicast streaming in cable networks a decade ago. For edge networks with reasonable 100gig distribution as their standard, these would see some of the minor buffering issues. There is a reason that cable doesn’t stream unicast and uses multicast and QAM on a wire. We’ve just about hit the point where this kind of scale unicast streaming is feasible for a live event without introducing a lot of latency. Some edge networks (especially without local cache nodes) just simply would not have enough capacity, whether in the core or peering edge, to do the trick.
- ta1243 2y agoSaw an Arista presentation about the increase in SFP capacity, it's Moore law style stuff. Arm based kit has a shockingly efficient amount of streams-per-watt too. I can't see traditional DVB/ATSC surviging much beyond 2040 even accounting for the long tail. You're right that large scale parallel live streams has only become feasible in the last few years. The BBC has some insights in how the BBC had to change their approach to scale to getting 10 million in 2021, having had technical issues in the 3 million range in 2018 https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk%2Fblogs%2Finternet%2Fentries%2Ffe629b38-57af-49c8-8922-352174101511 https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk... Personally I don't think the latency is solved yet -- TV is slow enough (about 10 seconds from camera to TV), but IP streaming tends to add another 20-40 seconds on top of that. That's no good when you're watching the penalties. Not only will your neighbours be cheering before you as they watch on normal TV, but even if you're both on the same IPTV you may well 5 seconds of difference. The total end-to-end time is important too, with 30 seconds the news push notifications, tweets, etc on your phone will come in before you see the result.
- yusyusyus 2y ago> Saw an Arista presentation about the increase in SFP capacity, it's Moore law style stuff. SFP itself isnt much the issue. Serdes is, and then secondarily the operating power envelope for the things (especially for the kinds of optics that run hot). Many tradeoffs available. >I can't see traditional DVB/ATSC surviging much beyond 2040 even accounting for the long tail. Tend to agree in well-developed infra, but rural and poorly-developed are well served with more traditional broadcast. Just saying “starlink!” 5 times in a dark bathroom won’t fix that part. > Personally I don't think the latency is solved yet -- TV is slow enough (about 10 seconds from camera to TV), but IP streaming tends to add another 20-40 seconds on top of that. I dont think it will get better. Probably worse, but with net better service quality. HLS/DASH are designed for doing the bursty networking thing. Among good reasons for this, mobile works much better in bursts than strict linear streams, segment caching is highly effective, etc. But I think this makes sense: its a server-side buffering thing that has to happen. So assuming transmuxing (no transcoding lol) and wire latency are 0, we’re hitting the 1-5 seconds for the segment, probably waiting for a fill of 10 seconds to produce the manifest, then buffering client-side another 10 or so. Throw in more cache boxes and it’ll tick up more. It is quite high, but aside from bookies, i dont know how much people will actually care vs complain.
- onetokeoverthe 2y agooff topic but. i thought tyson was in eldercare.
- cryptica 2y agoI don't understand why the media is pushing this a Jake Paul vs Mike Tyson stuff so hard and why people care about it. Boxing is crude entertainment for low intelligence people. I'm tired of all this junk entertainment which only serves to give people second-hand emotions that they can't feel for themselves in real life. It's like, some people can't get sex so they watch porn. People can't fight so they watch boxing. People can't win in real life so they play video games or watch superhero movies. Many people these days have to live vicariously through random people/entities; watch others live the life they wished they had and then they idolize these people who get to have everything... As if these people were an intimate projection of themselves... When, in fact, they couldn't be more different. It's like rooting for your opponent and thinking you're on the same team; when, in fact, they don't even know that you exist and they couldn't be more different from you. You're no Marvel superhero no matter how many comic books you own. The heroes you follow have nothing to do with you. Choose different heroes who are more like you. Or better; do something about your life and give yourself a reason to idolize yourself.
- achow 2y agoThis is Netflix's statement on the fiasco https://x.com/netflix/status/1857906492235723244 https://x.com/netflix/status/1857906492235723244
- rajnathani 2y agoMaybe if jedberg and Brendan Gregg were still a part of Netflix, that this wouldn’t have happened.
- Publius_Enigma 2y agoI'm sure the architecture and scale of NetFlix's operations is truly impressive, but stories like this make me further appreciate the elegant simplicity of scalability of analogue terrestrial TV, and to a similar extent, digital terrestrial TV and satellite.
- PeterStuer 2y agoPure speculation as I have 0 knowledge. Assuming Netflix used its extensive edge cache network to distribute the streams to the ISPs. The software on the caching servers would have been updated to be capable of dealing with receiving and distributing live streamed content, even if maybe the hardware was not optimal for that (throughput vs latency is a classic networking tradeoff). Now inside the ISPs network again everyting would probably be optimized for the 99.99% usecase of the Netflix infra: delivering large bulk data that is not time sensitive. This means very large buffers to shift big gobs of packets in bulk. As everything along the path is trying to fill up those buffers before shipping to the next router on the path, some endpoints aware this is a live stream start cancelling and asking for more recent frames ... Hilarity ensues
- perseysinowatz 2y ago[dead]
- vdm 2y agoTreeDN: Tree-Based CDNs for Mass Audience Live Streaming https://www.youtube.com/watch?v=wRUwsvept-8 https://www.youtube.com/watch?v=wRUwsvept-8
- uptownfunk 2y agoAll your corporate culture, comp Structure, Interview process etc etc is all so much meta if you can’t deliver. They showed they can’t deliver. Huge let down.
- neilstalmans 2y ago[dead]