6 ms·

When you step back and look at the situation, it's not hard to see why Netflix dropped the ball here. Here's now I see it (not affiliated with Netflix, pure spe

When you step back and look at the situation, it's not hard to see why Netflix dropped the ball here. Here's now I see it (not affiliated with Netflix, pure speculation): - Months ago, the "higher ups" at Netflix struck a deal to stream the fight on Netflix. The exec that signed the deal was probably over the moon because it would get Netflix into a brand new space and bring in large audience numbers. Along the way the individuals were probably told that Netflix doesn't do livestreaming but they ignored it and assumed their talented Engineers could pull it off. - Once the deal was signed then it became the Engineer's problem. They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events that you don't really have to think about when streaming static content. - Engineering probably did their absolute best to pull this off but they had two main disadvantages, first off they don't have any of the institutional knowledge about live streaming and they don't really know how to predict demand for something like this. In the end they probably beefed up livestreaming as much as they could but still didn't go far enough because again, no one there really knows how something like this will pan out. - Evening started off fine but crap hit the fan later in the show as more people tuned in for the main card. Engineering probably did their best to mitigate this but again, since they don't have the institutional knowledge of live events, they were shooting in the dark hoping their fixes would stick. Yes Netflix as a whole screwed this one up but I'm tempted to give them more grace than usual here. First off the deal that they struck was probably one they couldn't ignore and as for Engineering, I think those guys did the freaking best they could given their situation and lack of institutional knowledge. This is just a classic case of biting off more than one can chew, even if you're an SV heavyweight.

Loughla 2y ago
>First off the deal that they struck was probably one they couldn't ignore If you can't provide the service you shouldn't sell it?
- immibis 2y ago
  You've never worked in a startup have you? Or any business for that matter. You have to promise something first, then build it.
  - colesantiago 2y ago
    No joke, is this actually true? Do startups really do this? I thought the capability is built or nearly built or at least in testing already with reasonable or amazing results, THEN they go to market? Do startups go to other startups, fortune 500 companies and public companies to make false promises with or without due diligence and sign deals with the knowledge that the team and engineers know the product doesn't have the feature in place at all? In other words: Company A: "We can provide web scale live streaming service around the world to 10 billion humans across the planet, even the bots will be watching." Company B: "OK, sounds good, Yes, here is a $2B contract." Company A: "Now team I know we don't have the capability, but how do we build, test and ship this in under 6 months???"
    - ImPostingOnHN 2y ago
      Many do, as far as initial investment goes. It makes sense when you think about the capital intensive nature of most startups (including more than web startups here, e.g. lab tech commercialization). It also accurately describes a research grant. That's for startups that can't bootstrap (most of them). For ones which can, they may still choose to do this with customers, as you describe, because it means letting their work follow the money.
    - tky 2y ago
      “Aspirational sugar” is as common in startup culture as in Fortune 500 sales contracts, they’re just messaged and “de-risked” differently.
    - 999900000999 2y ago
      Startups absolutely sell things they haven't made yet and might not even be capable of doing. Next thing you know it's 9pm on a Sunday night and your desperately trying to ship a build for a client. Netflix isn't some scrappy company though. If I had to guess they threw money at the problem. A much better approach would of been to slowly scale over the course of a year. Maybe stream some college basketball games first, slowly picking more popular events to get some real prod experience. Instead this is like their 3rd or 4th live stream ever. Even a pre show a week before would of allowed for greater testing. I'm not a CTO of a billion dollar company though. I'm just an IC who's seen a few sites go down underload. To be fair no one knows how it's going to go before it happens. It would of been more surprising for them to pull this off without issues... It's a matter of managing those issues. I know if I had paid 30$ for a Netflix subscription to watch this specific event I'd assume I got ripped off.
- _fat_santa 2y ago
  My speculation here is this was just classic SV cockiness. The team that closed this deal probably knew that they didn't have the capability but I'm sure the arguments for doing it anyways was something along the lines of: "we have the best engineers in the bay area, we can probably figure this out"
- mbesto 2y ago
  There are endless amounts of stories and situations in which selling something before it really exists has helped businesses. It's totally plausible that a team working on video streaming at the scale of Netflix could figure out live streaming. Pre-optimization is definitely a thing and it can massively hurt (i.e. startups go under) businesses. Let's stop pretending any businesses would say 'no' to extra revenue even before the engineering team had full assurance there was no latency drop.
  - sfn42 2y ago
    In my language we have a saying that roughly translates to "Don't sell the hide until you've shot the bear". And sure, there have probably been lots of examples where a business made promises they weren't confident about and succeeded. But there are surely also lots of examples where they didn't succeed. So what's the moral of the story? I don't know, maybe if you take a gamble you should be prepared to lose that gamble. Sounds to me like Netflix fucked up. They sold something they couldn't provide. What are the consequences of that? I don't know, not do I particularly care. Not my problem.
- RicoElectrico 2y ago
  Execs never listen or even ask engineers about feasibility of projects they sign up to. Hope the exec in question will be let go.
  - pilotneko 2y ago
    I mean, the ones that do ask don’t proceed to signing up. I think we are seeing a form of survival bias.
- throw0101b 2y ago
  >> First off the deal that they struck was probably one they couldn't ignore > If you can't provide the service you shouldn't sell it? Then how will the folks in Sales get their commission? Besides, not providing the service hasn't stopped Tesla from selling FSD, and their stock has been going gangbusters. /s
js2 2y ago
> They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events It wasn't their first live event. A previous live event had similar issues.
TheAceOfHearts 2y ago
This isn't Netflix's first foray into livestreaming. They tried a livestream last year for a reunion episode of one of their reality TV shows which encountered similar issues [0]. Netflix already has a contract to livestream a football event on Christmas, so it'll be interesting to see if their engineers are able to get anything done in a little over a month. These failures reflect very poorly on Netflix leadership. But we all know that leadership is never held accountable for their failures. Whoever is responsible for this should at least come forward and put out an apology while owning up to their mistakes. [0] https://time.com/6272470/love-is-blind-live-reunion-netflix/ https://time.com/6272470/love-is-blind-live-reunion-netflix/
- crazygringo 2y ago
  > But we all know that leadership is never held accountable for their failures. You've never heard of a CEO or other C-suite or VP getting fired? It most definitely happens. On the other hand, people at every level make mistakes, and it's preferable that they learn from them rather than be fired, if at all possible.
  - TheAceOfHearts 2y ago
    Accountability can take many forms. I don't think they should be fired for making a mistake, I think they should release a statement recognizing their failure along with a post-mortem. Not a particularly high bar, but most leadership failures are often swept under the rug without any public accountability or evidence that they've learned anything. We have evidence of prior failures with livestreaming from Netflix. Were the same people responsible for that failure or do we have evidence of them having learned anything between events? If anything, I'd expect the best leaders would have a track record that includes failures while showcasing their ability to overcome and learn from those mistakes. But based on what information is publicly available, this doesn't seem to be the case in this situation.
JumpCrisscross 2y ago
Livestreaming is a solved problem. This sounds like NIH [1]. (At the very least, hire them as a back-up.) [1] https://en.wikipedia.org/wiki/Not_invented_here https://en.wikipedia.org/wiki/Not_invented_here
- gregorygoc 2y ago
  Saying live-streaming is a solved problem is like saying search is a solved problem.
  - JumpCrisscross 2y ago
    > Saying live-streaming is a solved problem is like saying search is a solved problem It is. You can hire the people who have solved it to do it for you.
    - talldayo 2y ago
      > It is. You can hire the people who have solved it to do it for you. "GPGPU compute is a solved problem if you buy Nvidia hardware" type comment
      - JumpCrisscross 2y ago
        > "GPGPU compute is a solved problem if you buy Nvidia hardware" type comment You're replacing the word hire with buy. That misconstrues the comment. If you need to do GPGPU compute and have never done it, you work with a team that has. (And if you want to build it in house, you scale to it.)
      - moralestapia 2y ago
        >"GPGPU compute is a solved problem if you buy Nvidia hardware" Which is valid? If your problem can be solved by writing a check, then it's the easiest problem to have on the planet. Netflix didn't have to put out 3 PhD dissertations on how to improve the SOTA of live streaming, they only needed to reliably broadcast a fight for a couple hours. That is a solved problem. Amazon and Cloudflare do that for you as a service(!). Twitch and YouTube do it literally every day. Even X started doing it recently so. No excuses for Netflix, tbh.
jaimsam 2y ago
[flagged]
intelVISA 2y ago
Not sure why Netflix is held in high regard - this proves they're just as much clowns as the other 'big players' in the circus.
- thinkingkong 2y ago
  They arent clowns at all. Ita a totally different engineering problem and you cant just spin up live streaming capacity on demand. The entire system end to end isnt optimized for live streams yet.
ctvo 2y ago
[flagged]
- mensetmanusman 2y ago
  That’s uncharitable. Proposing reasons for institutional failure and discussing those can be ways for humans to improve communication and said challenges.
  - mmooss 2y ago
    It's a way to mislead people into misunderstanding reality and therefore solving the wrong problems, often causing harm, now and in the future. That's why serious analysis requires a factual basis, such as science, law, and good engineering and management. You need analytics data to figure out where the performance and organizational bottlenecks are. Before people tried to understand illness with a factual basis, they wrote speculative essays on leeching and finding 'better' ways to do it.
draw_down 2y ago
I mean, maybe? You just made all this up.

Loughla 2y ago

>First off the deal that they struck was probably one they couldn't ignore If you can't provide the service you shouldn't sell it?

immibis 2y ago
You've never worked in a startup have you? Or any business for that matter. You have to promise something first, then build it.
- colesantiago 2y ago
  No joke, is this actually true? Do startups really do this? I thought the capability is built or nearly built or at least in testing already with reasonable or amazing results, THEN they go to market? Do startups go to other startups, fortune 500 companies and public companies to make false promises with or without due diligence and sign deals with the knowledge that the team and engineers know the product doesn't have the feature in place at all? In other words: Company A: "We can provide web scale live streaming service around the world to 10 billion humans across the planet, even the bots will be watching." Company B: "OK, sounds good, Yes, here is a $2B contract." Company A: "Now team I know we don't have the capability, but how do we build, test and ship this in under 6 months???"
  - ImPostingOnHN 2y ago
    Many do, as far as initial investment goes. It makes sense when you think about the capital intensive nature of most startups (including more than web startups here, e.g. lab tech commercialization). It also accurately describes a research grant. That's for startups that can't bootstrap (most of them). For ones which can, they may still choose to do this with customers, as you describe, because it means letting their work follow the money.
  - tky 2y ago
    “Aspirational sugar” is as common in startup culture as in Fortune 500 sales contracts, they’re just messaged and “de-risked” differently.
  - 999900000999 2y ago
    Startups absolutely sell things they haven't made yet and might not even be capable of doing. Next thing you know it's 9pm on a Sunday night and your desperately trying to ship a build for a client. Netflix isn't some scrappy company though. If I had to guess they threw money at the problem. A much better approach would of been to slowly scale over the course of a year. Maybe stream some college basketball games first, slowly picking more popular events to get some real prod experience. Instead this is like their 3rd or 4th live stream ever. Even a pre show a week before would of allowed for greater testing. I'm not a CTO of a billion dollar company though. I'm just an IC who's seen a few sites go down underload. To be fair no one knows how it's going to go before it happens. It would of been more surprising for them to pull this off without issues... It's a matter of managing those issues. I know if I had paid 30$ for a Netflix subscription to watch this specific event I'd assume I got ripped off.
_fat_santa 2y ago
My speculation here is this was just classic SV cockiness. The team that closed this deal probably knew that they didn't have the capability but I'm sure the arguments for doing it anyways was something along the lines of: "we have the best engineers in the bay area, we can probably figure this out"
mbesto 2y ago
There are endless amounts of stories and situations in which selling something before it really exists has helped businesses. It's totally plausible that a team working on video streaming at the scale of Netflix could figure out live streaming. Pre-optimization is definitely a thing and it can massively hurt (i.e. startups go under) businesses. Let's stop pretending any businesses would say 'no' to extra revenue even before the engineering team had full assurance there was no latency drop.
- sfn42 2y ago
  In my language we have a saying that roughly translates to "Don't sell the hide until you've shot the bear". And sure, there have probably been lots of examples where a business made promises they weren't confident about and succeeded. But there are surely also lots of examples where they didn't succeed. So what's the moral of the story? I don't know, maybe if you take a gamble you should be prepared to lose that gamble. Sounds to me like Netflix fucked up. They sold something they couldn't provide. What are the consequences of that? I don't know, not do I particularly care. Not my problem.
RicoElectrico 2y ago
Execs never listen or even ask engineers about feasibility of projects they sign up to. Hope the exec in question will be let go.
- pilotneko 2y ago
  I mean, the ones that do ask don’t proceed to signing up. I think we are seeing a form of survival bias.
throw0101b 2y ago
>> First off the deal that they struck was probably one they couldn't ignore > If you can't provide the service you shouldn't sell it? Then how will the folks in Sales get their commission? Besides, not providing the service hasn't stopped Tesla from selling FSD, and their stock has been going gangbusters. /s

js2 2y ago

> They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events It wasn't their first live event. A previous live event had similar issues.

TheAceOfHearts 2y ago

This isn't Netflix's first foray into livestreaming. They tried a livestream last year for a reunion episode of one of their reality TV shows which encountered similar issues [0]. Netflix already has a contract to livestream a football event on Christmas, so it'll be interesting to see if their engineers are able to get anything done in a little over a month. These failures reflect very poorly on Netflix leadership. But we all know that leadership is never held accountable for their failures. Whoever is responsible for this should at least come forward and put out an apology while owning up to their mistakes. [0] https://time.com/6272470/love-is-blind-live-reunion-netflix/ https://time.com/6272470/love-is-blind-live-reunion-netflix/

crazygringo 2y ago
> But we all know that leadership is never held accountable for their failures. You've never heard of a CEO or other C-suite or VP getting fired? It most definitely happens. On the other hand, people at every level make mistakes, and it's preferable that they learn from them rather than be fired, if at all possible.
- TheAceOfHearts 2y ago
  Accountability can take many forms. I don't think they should be fired for making a mistake, I think they should release a statement recognizing their failure along with a post-mortem. Not a particularly high bar, but most leadership failures are often swept under the rug without any public accountability or evidence that they've learned anything. We have evidence of prior failures with livestreaming from Netflix. Were the same people responsible for that failure or do we have evidence of them having learned anything between events? If anything, I'd expect the best leaders would have a track record that includes failures while showcasing their ability to overcome and learn from those mistakes. But based on what information is publicly available, this doesn't seem to be the case in this situation.

JumpCrisscross 2y ago

Livestreaming is a solved problem. This sounds like NIH [1]. (At the very least, hire them as a back-up.) [1] https://en.wikipedia.org/wiki/Not_invented_here https://en.wikipedia.org/wiki/Not_invented_here

gregorygoc 2y ago
Saying live-streaming is a solved problem is like saying search is a solved problem.
- JumpCrisscross 2y ago
  > Saying live-streaming is a solved problem is like saying search is a solved problem It is. You can hire the people who have solved it to do it for you.
  - talldayo 2y ago
    > It is. You can hire the people who have solved it to do it for you. "GPGPU compute is a solved problem if you buy Nvidia hardware" type comment
    - JumpCrisscross 2y ago
      > "GPGPU compute is a solved problem if you buy Nvidia hardware" type comment You're replacing the word hire with buy. That misconstrues the comment. If you need to do GPGPU compute and have never done it, you work with a team that has. (And if you want to build it in house, you scale to it.)
    - moralestapia 2y ago
      >"GPGPU compute is a solved problem if you buy Nvidia hardware" Which is valid? If your problem can be solved by writing a check, then it's the easiest problem to have on the planet. Netflix didn't have to put out 3 PhD dissertations on how to improve the SOTA of live streaming, they only needed to reliably broadcast a fight for a couple hours. That is a solved problem. Amazon and Cloudflare do that for you as a service(!). Twitch and YouTube do it literally every day. Even X started doing it recently so. No excuses for Netflix, tbh.

jaimsam 2y ago

[flagged]

intelVISA 2y ago

Not sure why Netflix is held in high regard - this proves they're just as much clowns as the other 'big players' in the circus.

thinkingkong 2y ago
They arent clowns at all. Ita a totally different engineering problem and you cant just spin up live streaming capacity on demand. The entire system end to end isnt optimized for live streams yet.

ctvo 2y ago

[flagged]

mensetmanusman 2y ago
That’s uncharitable. Proposing reasons for institutional failure and discussing those can be ways for humans to improve communication and said challenges.
- mmooss 2y ago
  It's a way to mislead people into misunderstanding reality and therefore solving the wrong problems, often causing harm, now and in the future. That's why serious analysis requires a factual basis, such as science, law, and good engineering and management. You need analytics data to figure out where the performance and organizational bottlenecks are. Before people tried to understand illness with a factual basis, they wrote speculative essays on leeching and finding 'better' ways to do it.

draw_down 2y ago

I mean, maybe? You just made all this up.