6 ms·

The way to deal with this is to constantly do live events, and actually build organizational muscle. Not these massive one off events in an area the tech team h

The way to deal with this is to constantly do live events, and actually build organizational muscle. Not these massive one off events in an area the tech team has no experience in.

_dark_matter_ 2y ago
Agreed. This is a management failure, full stop. Unbelievable that they'd expect engineering to handle a single Livestream event of this magnitude.
mbrumlow 2y ago
I have this argument a lot in tech. We should always be doing (the thing we want to do) Somme examples that always get me in trouble (or at least big heated conversations) 1. Always be building: It does not matter if code was not changed, or there has been no PRs or whatever, build it. Something in your org or infra has likely changed. My argument is "I would rather have a build failure on software that is already released, than software I need to release". 2. Always be releasing: As before it does not matter if nothing changed, push out a release. Stress the system and make it go through the motions. I can't tell you how many times I have seen things fail to deploy simply because they have not attempted to do so in some long period of time. There are more just don't have time to go into them. The point is if "you did it, and need to do it again ever in the future, then you need to continuously do it"
- parasti 2y ago
  This is golden advice, honestly. "If you don't use it, you lose it" applied to software development.
- andai 2y ago
  This is great, but what possible counterargument is there? We should prolong indefinitely a spooky ambiguity about whether the system works or not?
  - mplewis 2y ago
    The common and flawed counterargument is “when we deploy, outages happen.” You’ll hear this constantly at companies with bad habits.
  - ukuina 2y ago
    Finite compute, people, and opportunity cost. It is just a reframing of build vs maintain.
  - macintux 2y ago
    In some environments, deploying to production has a massive bureaucracy tax. Paperwork, approvals, limited windows in time, can’t do them during normal business hours, etc.
    - deleted 2y ago
      [deleted]
    - josho 2y ago
      Those taxes were often imposed because of past engineering errors. For example, Don't deploy during business hours because a past deployment took down production for a day. A great engineering team will identify a tax they dislike and work to remove it. Using the same example, that means improving the success rate of deployments so you have the data (the success record) to take to leadership to change the policy and remove the tax.
  - kortilla 2y ago
    Deploying is expensive for some models. That could involve customer facing written release notes, etc. Sometimes the software has to be certified by a govt authority. Additionally, refactor circle jerks are terrible for back-porting subsequent bug fixes that need to be cherry picked to stable branches. A lot of of the world isn’t CD and constant releases are super expensive.
  - jerf 2y ago
    Easy: Short term risk versus long term risk. If I deploy with minimal changes today, I'm taking a non-zero short-term risk for zero short-term gain. While I too am generally a long-term sort of engineer, it's important to understand that this is a valid argument on its own terms, so you don't try to counter it with just "piffle, that's stupid". It's not stupid. It can be shortsighted, it leads to a slippery slope where every day you make that decision it is harder to release next time, and there's a lot of corpses at the bottom of that slope, but it isn't stupid. Sometimes it is even correct, for instance, if the system's getting deprecated away anyhow why take any risk? And there is some opportunity cost, too. No matter how slick the release, it isn't ever free. Even if it's all 100% automated it's still going to barf sometimes and require attention that not making a new release would not have. You could be doing something else with that time.
- skybrian 2y ago
  Doing dry runs regularly makes sense, but whether actually shipping it makes sense seems context-dependent. It depends on how much you can minimize the side effects of shipping a release. Consider publishing a new version of a library: you'd be bumping the version number all the time and invalidating caches, causing downstream rebuilds, for little reason. Or if clients are lazy about updating, any two clients would be unlikely to have the same version. Or consider the case when shipping results in a software update: millions of customer client boxes wasting bandwidth downloading new releases and restarting for no reason. Even for a web app, you are probably invalidating caches, resulting in slow page loads. With enough work, you could probably minimize these side effects, so that releasing a new version that doesn't actually change anything is a non-event. But if you don't invalidate the caches, you're not really doing a full rebuild. So it seems like there's a tension between doing more end-to-end testing and performance? Implementing a bunch of cache levels and then not using it seems counterproductive.
  - lxgr 2y ago
    It's very hard to do a representative dry run when the most likely potential points of failure are highly load-dependent. You can try and predict everything that'll happen in production, but if you have nothing to extrapolate from, e.g. because this is your very first large live event, the chances of getting that right are almost zero. And you can't easily import that knowledge either, because your system might have very different points of failure than the ones external experts might be used to.
    - deleted 2y ago
      [deleted]
    - leptons 2y ago
      They could have done a dry run. They could have spun up a million virtual machines somewhere, and tested their video delivery for 30 minutes. Even my small team spins up 10,000 EC2 instances on the regular. Netflix has the money to do much more. I'm sure there are a dozen ways they could have stress-tested this beforehand. It's not like someone sprang this on them last week and they had to scramble to put together a system to do it.
- unoti 2y ago
  > 1. Always be building: It does not matter if code was not changed... > 2. Always be releasing... A good argument for this is security. Whatever libraries/dependencies you have, unpin the versions, and have good unit tests. Security vulnerabilities that are getting fixed upstream must be released. You cannot fix and remove those vulnerabilities unless you are doing regular releases. This in turn also implies having good unit tests, so you can do these builds and releases with a lower probability of releasing something broken. It also implies strong monitoring and metrics, so you can be the first to know when something breaks.
  - caseyohara 2y ago
    > Whatever libraries/dependencies you have, unpin the versions, and have good unit tests. Nitpick: unit tests by definition should not be exercising dependencies outside the unit boundary. What you want are solid integration and system tests for that.
  - kortilla 2y ago
    Unless the upstream dependency happens to maintain stable branches, constantly pulling in the latest branches increases your risk of vulnerabilities more than getting the discovered bug patches
- 01HNNWZ0MV43FF 2y ago
  There's two other ways I've seen it phrased: "Test what you fly, and fly what you test" (Supposedly from aviation) "There should be one joint, and it should be greased regularly" (Referring to cryptosystems I think, but it's the same principle. Things like TLS will ossify if they aren't exercised. QUIC has provisions to prevent this.)
- ravenstine 2y ago
  There should be a caveat that such this kind of decision should be based on experience and not treated as a rule that juniors might blindly follow. We all know how "fail fast and early" turned out (or whatever the exact phrase was).
MisterBastahrd 2y ago
The WWE is moving their programming to Netflix next year. If I were them, I'd be horrified at what I saw.
geor9e 2y ago
They've been doing live events since 2023. But it's hard to be prepared for something that's never been done by anyone before — a superbowl scale event, entirely viewed over the internet. The superbowl gets to offload to cable and over the air. Interestingly, I didn't have any problems with my stream. So it sounds like the bandwidth problems might be localized, perhaps by data center or ISP.
- burntalmonds 2y ago
  Yeah, I think people are incorrectly assuming that everyone had the same experience with the stream. I watched the whole thing and only had a few instances of buffering and quality degradation. Not more than 30 seconds total during the stream.
  - DharmaPolice 2y ago
    Even if it was only 30% of people had a problem that's still millions of unhappy users. Not great for a time sensitive event. Also, from lurking in various threads on the topic Netflix's in app messages added to people's irritation by suggesting that they check their WiFi/internet was working. Presumably that's the default error message but perhaps that could have been adjusted in advance somehow.
    - positr0n 2y ago
      One of the times I reloaded the page I got a raw envoy error message!
    - ta1243 2y ago
      The point is that if the problem was different depending on the user, it will be in a distribution layer, not in the encoding or production layer That eliminates a whole raft of problems.
- mastazi 2y ago
  Maybe they considered this event as a rehearsal for the upcoming NFL streams which I am guessing might have a wider audience
ignoramous 2y ago
> ...the tech team has no experience in Unless Netflix eng decides to release a public postmorterm, we can only speculate. In my time organizing small-time live streams, we always had up to 3 parallel "backup" streams (Vimeo, Cloudflare, Livestream). At Netflix's scale, I doubt they could simply summon any of these providers in, but I guess Akamai / Cloudflare would have been up for it.
giantg2 2y ago
Wow, building talent from within? I thought that went out of fashion. I think companies are too impatient to develop their employees.
eh9 2y ago
that’s difficult to reproduce at scale; there are only so many “super bowl” events in a calendar year
don-code 2y ago
Sometimes this just isn't feasible for cost reasons. A company I used to work for ran a few Super Bowl ads. The level of traffic you get during a Super Bowl ad is immense, and it all comes at you in 30 seconds, before going back to a steady-state value just as quickly. The scale pattern is like nothing else I've ever seen. Super Bowl ads famously seven million dollars. These are things we simply can't repeat year over year, even if we believed it'd generate the same bump in recognition each time.
NoPicklez 2y ago
I think Netflix have a fair bit of organisational muscle, perhaps the fight was considered not as large of an event as the NFL streams would be in the future. Also, "No experience in" really? You have no idea if that's really the case

The way to deal with this is to constantly do live events, and actually build organizational muscle. Not these massive one off events in an area the tech team has no experience in.

_dark_matter_ 2y ago

Agreed. This is a management failure, full stop. Unbelievable that they'd expect engineering to handle a single Livestream event of this magnitude.

mbrumlow 2y ago

I have this argument a lot in tech. We should always be doing (the thing we want to do) Somme examples that always get me in trouble (or at least big heated conversations) 1. Always be building: It does not matter if code was not changed, or there has been no PRs or whatever, build it. Something in your org or infra has likely changed. My argument is "I would rather have a build failure on software that is already released, than software I need to release". 2. Always be releasing: As before it does not matter if nothing changed, push out a release. Stress the system and make it go through the motions. I can't tell you how many times I have seen things fail to deploy simply because they have not attempted to do so in some long period of time. There are more just don't have time to go into them. The point is if "you did it, and need to do it again ever in the future, then you need to continuously do it"

parasti 2y ago
This is golden advice, honestly. "If you don't use it, you lose it" applied to software development.
andai 2y ago
This is great, but what possible counterargument is there? We should prolong indefinitely a spooky ambiguity about whether the system works or not?
- mplewis 2y ago
  The common and flawed counterargument is “when we deploy, outages happen.” You’ll hear this constantly at companies with bad habits.
- ukuina 2y ago
  Finite compute, people, and opportunity cost. It is just a reframing of build vs maintain.
- macintux 2y ago
  In some environments, deploying to production has a massive bureaucracy tax. Paperwork, approvals, limited windows in time, can’t do them during normal business hours, etc.
  - deleted 2y ago
    [deleted]
  - josho 2y ago
    Those taxes were often imposed because of past engineering errors. For example, Don't deploy during business hours because a past deployment took down production for a day. A great engineering team will identify a tax they dislike and work to remove it. Using the same example, that means improving the success rate of deployments so you have the data (the success record) to take to leadership to change the policy and remove the tax.
- kortilla 2y ago
  Deploying is expensive for some models. That could involve customer facing written release notes, etc. Sometimes the software has to be certified by a govt authority. Additionally, refactor circle jerks are terrible for back-porting subsequent bug fixes that need to be cherry picked to stable branches. A lot of of the world isn’t CD and constant releases are super expensive.
- jerf 2y ago
  Easy: Short term risk versus long term risk. If I deploy with minimal changes today, I'm taking a non-zero short-term risk for zero short-term gain. While I too am generally a long-term sort of engineer, it's important to understand that this is a valid argument on its own terms, so you don't try to counter it with just "piffle, that's stupid". It's not stupid. It can be shortsighted, it leads to a slippery slope where every day you make that decision it is harder to release next time, and there's a lot of corpses at the bottom of that slope, but it isn't stupid. Sometimes it is even correct, for instance, if the system's getting deprecated away anyhow why take any risk? And there is some opportunity cost, too. No matter how slick the release, it isn't ever free. Even if it's all 100% automated it's still going to barf sometimes and require attention that not making a new release would not have. You could be doing something else with that time.
skybrian 2y ago
Doing dry runs regularly makes sense, but whether actually shipping it makes sense seems context-dependent. It depends on how much you can minimize the side effects of shipping a release. Consider publishing a new version of a library: you'd be bumping the version number all the time and invalidating caches, causing downstream rebuilds, for little reason. Or if clients are lazy about updating, any two clients would be unlikely to have the same version. Or consider the case when shipping results in a software update: millions of customer client boxes wasting bandwidth downloading new releases and restarting for no reason. Even for a web app, you are probably invalidating caches, resulting in slow page loads. With enough work, you could probably minimize these side effects, so that releasing a new version that doesn't actually change anything is a non-event. But if you don't invalidate the caches, you're not really doing a full rebuild. So it seems like there's a tension between doing more end-to-end testing and performance? Implementing a bunch of cache levels and then not using it seems counterproductive.
- lxgr 2y ago
  It's very hard to do a representative dry run when the most likely potential points of failure are highly load-dependent. You can try and predict everything that'll happen in production, but if you have nothing to extrapolate from, e.g. because this is your very first large live event, the chances of getting that right are almost zero. And you can't easily import that knowledge either, because your system might have very different points of failure than the ones external experts might be used to.
  - deleted 2y ago
    [deleted]
  - leptons 2y ago
    They could have done a dry run. They could have spun up a million virtual machines somewhere, and tested their video delivery for 30 minutes. Even my small team spins up 10,000 EC2 instances on the regular. Netflix has the money to do much more. I'm sure there are a dozen ways they could have stress-tested this beforehand. It's not like someone sprang this on them last week and they had to scramble to put together a system to do it.
unoti 2y ago
> 1. Always be building: It does not matter if code was not changed... > 2. Always be releasing... A good argument for this is security. Whatever libraries/dependencies you have, unpin the versions, and have good unit tests. Security vulnerabilities that are getting fixed upstream must be released. You cannot fix and remove those vulnerabilities unless you are doing regular releases. This in turn also implies having good unit tests, so you can do these builds and releases with a lower probability of releasing something broken. It also implies strong monitoring and metrics, so you can be the first to know when something breaks.
- caseyohara 2y ago
  > Whatever libraries/dependencies you have, unpin the versions, and have good unit tests. Nitpick: unit tests by definition should not be exercising dependencies outside the unit boundary. What you want are solid integration and system tests for that.
- kortilla 2y ago
  Unless the upstream dependency happens to maintain stable branches, constantly pulling in the latest branches increases your risk of vulnerabilities more than getting the discovered bug patches
01HNNWZ0MV43FF 2y ago
There's two other ways I've seen it phrased: "Test what you fly, and fly what you test" (Supposedly from aviation) "There should be one joint, and it should be greased regularly" (Referring to cryptosystems I think, but it's the same principle. Things like TLS will ossify if they aren't exercised. QUIC has provisions to prevent this.)
ravenstine 2y ago
There should be a caveat that such this kind of decision should be based on experience and not treated as a rule that juniors might blindly follow. We all know how "fail fast and early" turned out (or whatever the exact phrase was).

MisterBastahrd 2y ago

The WWE is moving their programming to Netflix next year. If I were them, I'd be horrified at what I saw.

geor9e 2y ago

They've been doing live events since 2023. But it's hard to be prepared for something that's never been done by anyone before — a superbowl scale event, entirely viewed over the internet. The superbowl gets to offload to cable and over the air. Interestingly, I didn't have any problems with my stream. So it sounds like the bandwidth problems might be localized, perhaps by data center or ISP.

burntalmonds 2y ago
Yeah, I think people are incorrectly assuming that everyone had the same experience with the stream. I watched the whole thing and only had a few instances of buffering and quality degradation. Not more than 30 seconds total during the stream.
- DharmaPolice 2y ago
  Even if it was only 30% of people had a problem that's still millions of unhappy users. Not great for a time sensitive event. Also, from lurking in various threads on the topic Netflix's in app messages added to people's irritation by suggesting that they check their WiFi/internet was working. Presumably that's the default error message but perhaps that could have been adjusted in advance somehow.
  - positr0n 2y ago
    One of the times I reloaded the page I got a raw envoy error message!
  - ta1243 2y ago
    The point is that if the problem was different depending on the user, it will be in a distribution layer, not in the encoding or production layer That eliminates a whole raft of problems.
mastazi 2y ago
Maybe they considered this event as a rehearsal for the upcoming NFL streams which I am guessing might have a wider audience

ignoramous 2y ago

> ...the tech team has no experience in Unless Netflix eng decides to release a public postmorterm, we can only speculate. In my time organizing small-time live streams, we always had up to 3 parallel "backup" streams (Vimeo, Cloudflare, Livestream). At Netflix's scale, I doubt they could simply summon any of these providers in, but I guess Akamai / Cloudflare would have been up for it.

giantg2 2y ago

Wow, building talent from within? I thought that went out of fashion. I think companies are too impatient to develop their employees.

eh9 2y ago

that’s difficult to reproduce at scale; there are only so many “super bowl” events in a calendar year

don-code 2y ago

Sometimes this just isn't feasible for cost reasons. A company I used to work for ran a few Super Bowl ads. The level of traffic you get during a Super Bowl ad is immense, and it all comes at you in 30 seconds, before going back to a steady-state value just as quickly. The scale pattern is like nothing else I've ever seen. Super Bowl ads famously seven million dollars. These are things we simply can't repeat year over year, even if we believed it'd generate the same bump in recognition each time.

NoPicklez 2y ago

I think Netflix have a fair bit of organisational muscle, perhaps the fight was considered not as large of an event as the NFL streams would be in the future. Also, "No experience in" really? You have no idea if that's really the case