18 ms·

Releasing software to the fleet far too quickly broke stuff

172 points by guiambros 7y ago

yjftsjthsd-h 7y ago
On the one hand, there are decent technical tools to deal with this kind of thing; I'm directly familiar with Ansible's support for controlled rollouts, most container management layers happily support blue/green deploys, etc. On the other hand - > Some people obviously couldn't be bothered to step their release along over the course of a couple of days, and we had to find some way to protect the production environment from them. That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod.
- jameskraus 7y ago
  > That's easy: HR Unfortunately, that approach doesn't scale well. In a larger company, there are always new users discovering the systems and how they work (to varying degrees). If doing the right thing isn't by-and-far the easiest path, then it's highly likely someone will misuse the tool – no matter how smart they are. It's far easier to make the interface really easy to use than it is to educate users on proper usage.
  - thanksforfish 7y ago
    > there are always new users discovering the systems Software engineering turnover rates can be high, like 13%/year [1]. Add to that engineers who internally switch teams. New users is a real concern. In this case, it sounds like a custom deployment tool, so even engineers with good experience at other companies may not find the feature. Good team on-boarding and documentation is important. I think code review should catch something like this. Even with the high rate of new users, there should be enough experienced code reviewers to catch the issue. [1] https://www.techrepublic.com/article/software-had-the-highest-job-turnover-rate-of-any-industry-in-2017/ https://www.techrepublic.com/article/software-had-the-highes...
    - asveikau 7y ago
      The problem is when the distance between the team writing the package manager and random team trying to work on some feature is high. That entire team might not be aware of the team that wants them to deploy gradually and in a specific way. To them, maybe even it's a lot of noise and useless complaining and they don't see it as at all connected to getting their feature done. A common large company problem. People who should be talking are not. New hires are also hired into the "local" subculture of their team and something very, very important elsewhere in the company is not on their radar.
- paxys 7y ago
  HR is exactly the worst approach to deal with organizational and process problems like this. What do you do? Fire the entire dev team involved with the decision? Fire their managers who failed to set the right expectations? Fire the product team for setting deadlines that were too aggressive? Fire the CEO because hey, ultimately they are the one responsible for everything? Even if you do all that, you're not going to go back in time and fix things for your customers, and you're definitely not going to prevent the replacement developers from making the exact same mistakes. It takes really good leadership to build an engineering culture where good practices are valued and people learn from mistakes, which is why filling those positions is so hard and expensive. A company whose solution for every outage is to fire people isn't going to last very long.
  - deleted 7y ago
    [deleted]
  - AnimalMuppet 7y ago
    GP said: >> That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod. If people refuse to be careful. If they make a mistake once, teach them. If they make a mistake again, that happens. But if they make a mistake regularly, warn them, then warn them sternly, then fire them.
    - compiler-guy 7y ago
      Maybe so, but then you still have the problem, because in a big organization, some newbie is always present and going to make that first mistake. So you still need to make it hard for people to make mistakes.
      - csours 7y ago
        To amplify: think about road signs when you are trying to get somewhere new, especially in places with weird parking or entrances. After you've figured it out, it's easy, but everyone is a first timer at some point.
      - 7y ago
- greenyoda 7y ago
  HR is not an organization that should be arbitrating technical issues that they don't know anything about. The solution is for technical managers - at all levels, from team leads up to the CEO - to actually provide leadership to their teams. This can consist of training people about why it's important to do things right and setting up incentives to do things right (e.g., don't promote people who get things done quickly by cutting corners). To me, this story sounds like a classical management failure (been there, done that myself).
- geofft 7y ago
  But I don't want them to burn down someone else's prod, I want them to keep building features for my company. (Worse, if the other company has a misuse-resistant deployment system like the one the author alludes to, they might not burn down that company's prod and then beat us in the market while we're busy interviewing replacements!)
- triceratops 7y ago
  That's dumb. If your system makes it possible for someone to screw up, someone will eventually screw up. And if screwing up (rather than being malicious or stupid) results in people getting fired, then no work will ever get done for fear of getting fired. You have to fix the system so it's really hard to screw up.
- csours 7y ago
  "A person is smart. People are dumb, panicky dangerous animals and you know it." When I was first hired into a manufacturing plant I wondered why it was so hard to get people to make a quality product. It turns out that people are people, and they will try to do the job you tell them to do, with the tools and instruction you give them. The better the tools are, the less instruction is needed. Output rate and quality is also improved. The poorer the tools are, more instruction is needed; output rate and quality suffer, and at some point safety as well. "A poor workman blames their tools"; A poor manager blames their workman after providing them with poor tools. The very first thing you need to do is assess if the worker is following process and if not why not. If they are not following process, is it because of a mistake or a slip-up? Did they intentionally decide not to follow a process because it was difficult? Disaster documentaries are quick to point out where workers failed to follow written procedures, but they VERY RARELY point out how common those contraventions are with no negative impact. If "it's stupid but it works" it may not be stupid, or you may have gotten lucky. https://www.youtube.com/watch?v=WPMMNvYTEyI https://www.youtube.com/watch?v=WPMMNvYTEyI
hpoe 7y ago
I work at a very large IT org (1000+) and the only way I have ever seen anything change substantially and persistently was when the change was introduced and made it far easier to do the right thing. Ultimately if you want people to do the right thing you can't force them into it, you have to make them want it. It seems to me there are lots of opinions on how to change an organization or introduce improvements but ultimately if you aren't making people's lives easier, making them want the change you are trying to bring about, you'll find it almost impossible to succeed, human inertia is just too great.
- pjc50 7y ago
  This is the "poka-yoke" philosophy.
  - deleted 7y ago
    [deleted]
- Cthulhu_ 7y ago
  The only major changes I've seen in IT organizations of companies was where the existing IT department had declined or was let go over time and any development was in slumber mode. Then an excited manager comes around and erects a new IT organization from nothing (usually leaning heavily on consultants and freelancers), having sold a Digital Transformation to the company. The new IT / developer department spends millions a year in conjuring up a new server stack (cloud native) and a new software architecture (microservices woo), often building on top of existing ancient software (probably running on a mainframe three further abstraction levels down), with the intent of eventually replacing them. Eventually. Maybe. Usually those things got bogged down in the end because after 1-2 years, the first wave of developers get bored and move on to other opportunities, and the architecture they made is poorly abstracted microservices communicating via undocumented and unstructured REST APIs, with not nearly enough end-to-end tests or traceability to reliably make changes or add features. [/rant]
  - DerArzt 7y ago
    That got a little to real for me, I started thinking about work.
erulabs 7y ago
Really good post - "making it hard to do the wrong thing" is more or less my entire design ethos when it comes to software, particularly developer tools. Somewhat on-topic, instead of re-inventing the wheel on this one, I've recently been using Flagger (https://flagger.app/ https://flagger.app/) and have been exceptionally happy with it. Automated rollback on error rates, metrics or integration tests. It's so nice to finally be able to stop writing this code at every company I work for!
- sebcat 7y ago
  > making it hard to do the wrong thing" is more or less my entire design ethos when it comes to software, particularly developer tools. See also poka-yoke: https://en.m.wikipedia.org/wiki/Poka-yoke https://en.m.wikipedia.org/wiki/Poka-yoke
  - xyzzy_plugh 7y ago
    That's lovely, thank you!
  - Someone 7y ago
    I found it strange that that page doesn’t mention the example I immediately thought of after reading the first sentence: large dangerous machinery such as presses or lasers that require two hands to operate to prevent the case where the operator, in a hurry, still has one hand in the danger area when he presses the “do it” button with the other.
  - dctoedt 7y ago
    I'm going to add "poka-yoke" to the list of things that I make students commit to memory in my law school contract-drafting course; it'll be a synonym for "R.O.O.F" — Root Out Opportunities for [Foul]-ups. (This is another example of how legal-contract drafters can learn from the software world, including concepts such as, for example, orthogonality; modularity; R.O.O.F.; D.R.Y. [Don't Repeat Yourself]; graceful failure; and the benefit of sometimes explaining why things are done the way they are, to help future readers understand.)
- 7y ago
jedieaston 7y ago
It seems like the system for sending out packages should've been designed to be unless this is being sent out as a critical security patch (and signed off by a security manager or something in the log), the package is deployed to 5% of the fleet per day (unless someone opts in to get it early for some reason on a package-by-package basis). In a big fleet, anything else would still be chaos if someone clicked the wrong button.
paxys 7y ago
Large-scale deploy systems for a distributed environment are ridiculously hard to build. We had to set one up quickly and on a budget, so in lieu of fancy engineering we set up three rules: - Any change, whether it's a one-line config update or an entirely new backend, can only go out to a maximum of x% of hosts at a time - Hosts are taken fully out of rotation before getting an update - Every change has to be backwards compatible, and there can be no side-effects of random rollbacks, sometimes even across several versions The first two are easy to enforce programmatically, and the third one relies on the team. While the trade-off is that it's harder to make relatively simple changes, it has served us well so far.
- seanwilson 7y ago
  > Every change has to be backwards compatible, and there can be no side-effects of random rollbacks, sometimes even across several versions How does that rule work out in practice? If you're doing staged rollouts of new builds because you think there's a decent chance those new builds have bugs in them, shouldn't you realistically expect that rollbacks could have unexpected side-effects?
  - tantalor 7y ago
    > expect that rollbacks could have unexpected Expect the unexpected? Sure anything can happen. But paxys is specifically referring to expected (known) side-effects of rollbacks. For example if Foo and Bar are on versions Foo.3 and Bar.3, and we know rolling back to Bar.2 breaks Foo.3, then you broke backwards compatibility. Generally all combinations of possible rollbacks must continue to work, e.g., {Foo.2, Foo.3} x {Bar.2, Bar.3}. The number of rollbacks is limited to 1 or 2 versions, e.g., Foo.1 and Bar.1 cannot be used anymore.
- gaogao 7y ago
  The third one is not too bad to test programmatically, if you write a test harness that spins up a mixed version system.
- hedora 7y ago
  Downgrade should “just work” (with help from the team) for stateless services, since the old and new have to coexist during upgrade. However, if you’re pushing out new storage systems, supporting downgrade ends up lowering overall reliability in practice. It takes many times more effort to implement and test the downgrade of every disk layout change than it does to ensure that upgrades will always succeed in practice.
aahortwwy 7y ago
It's funny that the big tech companies have such high hiring bars and yet when you get inside there are large teams of people actively working to protect the company from the stupidity of their colleagues.
- hinkley 7y ago
  I’m mostly trying to protect myself from my own stupidity. That other people fall prey to the same issues in many scenarios just makes the pill go down easier for them.
  - EdwardDiego 7y ago
    That's the story of my development life! I know I'm human, and as such, I will err, but I really don't want to be erring as humans do in production. I feel like my "capacity to err over time" graph is U shaped with a bit of an M tendency towards the end - as a newbie, lots of ways to make mistakes without realising, then as my skills increased, realised the "known unknowns", and coded and released very cautiously, but when I blurred into a senior, ran the risk of thinking "I know this" without checking that all my assumptions and prior knowledge held true, and learned the caution of the intermediate developer again.
    - hinkley 7y ago
      I had a boss who insisted on calling the root account stupiduser, because when a server gets properly fucked up it's invariably because some stupid user did the wrong thing...
  - aahortwwy 7y ago
    When I say stupidity I don't mean mistakes. From the article: > At that point, it was decided that something easier had to be done. The barrier to entry for safe and sane rollouts had to be lowered even more to bring more teams on board. Why does something have to be easy for a team of professional software engineers to do it correctly? Make things easier to improve productivity, sure, but in my opinion awkward/difficult/complex tooling isn't a license to cut corners.
greendave 7y ago
The joke about building idiot-proof software comes to mind, but it's surprising to me how many systems really don't do a good job of making the right thing easy and the wrong hard. Dangerous defaults have consequences! Of course sometimes there's no substitute for learning things the hard way. Where practical, I've found having the responsible team on the hook for cleaning up their mess tends to work wonders in terms of getting people to be appropriately careful. That's not to say mistakes can't still happen, but it cuts down a lot.
csours 7y ago
> "This was one of those times when we needed to build a system that made it stupidly easy to do the right thing, so that it would actually be MORE work to do the wrong thing" This is basically the core of UX. It's kind of amazing to me that UX is so poorly represented in management tools.
diebeforei485 7y ago
This lesson, but in a UX context - Apple generally does a good job of making it hard for users to do the wrong thing (for their definition of wrong).
primitivesuave 7y ago
Really interesting post. Can anyone comment on how serverless deployments help and/or exacerbate the deployment problem? To give a theoretical starting point - Acme corporation has a serverless API deployed at api.acme.com powering website acme.com. The staging system is deployed at staging.api.acme.com, and the corresponding web app (e.g. staging.acme.com) targets that API. Staging gets deployed to production via a pull request that triggers CI/CD, which has no ability to do an X% rollout (i.e. the underlying serverless function is updated). Developer John Smith is writing a feature and merging to his personal branch "jsmith" - he can test his branch with jsmith.staging.api.acme.com and a corresponding development web app. When he's finished the feature, he does a PR to the staging branch, which is packaged into a PR to live. You may be able to see I'm trying to get input on our own release system :) It's worked quite well so far but I'm still paranoid about not having gradual roll-out - in my experience in bigger companies, there was always some A/B testing in place so bad deploys could be reverted quickly. If the sysop gurus of HN gathering on this thread can pass any comments/critique it would be greatly appreciated.
- mandelbrotwurst 7y ago
  Why would the ability to roll back deploys be tied to the ability to A/B test?
  - icebraining 7y ago
    I'm guessing the A/B functionality was being used as a Blue-Green deployment system.
- hibbelig 7y ago
  The post was talking about fleets of machines, where just restricting the number of machines that do things seems to be useful. But you only seem to have one machine in production (or one for the API plus one for the website), so it doesn't make sense to deploy to 10% of the machines. What you can do is to arrange things so that each individual upgrade can be rolled back. Let's say you want to rename a parameter in the API. You have to make a backward-compatible change to the API so that the existing website keeps working. In other words, you provide two calls, one with the old parameter name and a different call with the new parameter name. Then you deploy that. If there are problems, you can roll back to the previous API version. The next step is to change the website to invoke the new API call with the new parameter name. If that fails, you can go back to the previous website. Let's say it succeeds. Now you have a website using the new API with the new parameter name, and an API providing both the old and the new parameter name. The next step is to update the API to remove the old parameter name. If it turns out you forgot a place in the website where the old parameter is used, you can roll back to the previous API. If it goes well, you now have a website that uses the new parameter name, and also an API that provides the new parameter name, and the old parameter name is no longer used. It took a long time, but now it's mission accomplished and problem solved.
hackeryogi 7y ago
This is quite the representative case between Infra teams (or PaaS or SRE - based on where you're located) and application teams. Being part of the platform team, I've been in the exact same situation before at an ex employer. We assumed that _infra problems_ were _everyone's problems_ and how can anyone who cares about their application _not_ adopt to the latest-and-greatest-platform (or tool) we released. A lot of time was spent on figuring out the 'best approach' to getting everyone onboard quickly - we tried the carrot and the stick. We failed miserably at both. The trouble is, as someone else mentioned here - we didn't really go and _talk to the customer_. Internal customers are customers too - and 'product' thinking should be equally applied to all internal platforms. The general rule applies - talk to the customer and understand what they're going through. They have their own sets of problems and priorities. (a) Is the latest way actually _saving them a lot of time_ (b) Can we take another 2 weeks and make it even easier for them to come on board? (c) If they're really busy and/or lack the expertise, can we carve out some time to give enough training sessions? Yet, this never happens and we invariably end up blaming the application teams for _not doing this seemingly simple_ change. It may seem downright stupid from the platform engineer's PoV, but we are partly to blame. This also leads me to think that the cloud wars will be won by the <cloud-provider> that does this job better - of understanding the customer needs on a day to day basis and building platforms for them.
user5994461 7y ago
Been there, done that. The next rant will be: Massive outage because of a bug that was fixed last month. But the fix only shipped to 50% of the hosts, because the deployment system doesn't deploy everywhere by default.
- draw_down 7y ago
  And it will somehow be the users’ fault, not the deploy system’s fault.
hinkley 7y ago
Sounds like a setup for immutable servers, and I think I'm okay with that. If new copies of your service don't come up cleanly, you do not power down the old ones. If old servers can't talk to the new service, you get alarm bells going off while most of the old copies are still up and running. You've also partially serialized the deployment so you get some peak shaving. I wonder sometimes if the problem is that despite our protestations (and things like naming servers by unpronounceable names), we still think of servers running, instead of instances running. So the fact that the 20 copies of my service keep bouncing from machine to machine is at the very least a logistics issue, and possibly unsettling.

172 points by guiambros 7y ago

On the one hand, there are decent technical tools to deal with this kind of thing; I'm directly familiar with Ansible's support for controlled rollouts, most container management layers happily support blue/green deploys, etc. On the other hand - > Some people obviously couldn't be bothered to step their release along over the course of a couple of days, and we had to find some way to protect the production environment from them. That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod.

jameskraus 7y ago
> That's easy: HR Unfortunately, that approach doesn't scale well. In a larger company, there are always new users discovering the systems and how they work (to varying degrees). If doing the right thing isn't by-and-far the easiest path, then it's highly likely someone will misuse the tool – no matter how smart they are. It's far easier to make the interface really easy to use than it is to educate users on proper usage.
- thanksforfish 7y ago
  > there are always new users discovering the systems Software engineering turnover rates can be high, like 13%/year [1]. Add to that engineers who internally switch teams. New users is a real concern. In this case, it sounds like a custom deployment tool, so even engineers with good experience at other companies may not find the feature. Good team on-boarding and documentation is important. I think code review should catch something like this. Even with the high rate of new users, there should be enough experienced code reviewers to catch the issue. [1] https://www.techrepublic.com/article/software-had-the-highest-job-turnover-rate-of-any-industry-in-2017/ https://www.techrepublic.com/article/software-had-the-highes...
  - asveikau 7y ago
    The problem is when the distance between the team writing the package manager and random team trying to work on some feature is high. That entire team might not be aware of the team that wants them to deploy gradually and in a specific way. To them, maybe even it's a lot of noise and useless complaining and they don't see it as at all connected to getting their feature done. A common large company problem. People who should be talking are not. New hires are also hired into the "local" subculture of their team and something very, very important elsewhere in the company is not on their radar.
paxys 7y ago
HR is exactly the worst approach to deal with organizational and process problems like this. What do you do? Fire the entire dev team involved with the decision? Fire their managers who failed to set the right expectations? Fire the product team for setting deadlines that were too aggressive? Fire the CEO because hey, ultimately they are the one responsible for everything? Even if you do all that, you're not going to go back in time and fix things for your customers, and you're definitely not going to prevent the replacement developers from making the exact same mistakes. It takes really good leadership to build an engineering culture where good practices are valued and people learn from mistakes, which is why filling those positions is so hard and expensive. A company whose solution for every outage is to fire people isn't going to last very long.
- deleted 7y ago
  [deleted]
- AnimalMuppet 7y ago
  GP said: >> That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod. If people refuse to be careful. If they make a mistake once, teach them. If they make a mistake again, that happens. But if they make a mistake regularly, warn them, then warn them sternly, then fire them.
  - compiler-guy 7y ago
    Maybe so, but then you still have the problem, because in a big organization, some newbie is always present and going to make that first mistake. So you still need to make it hard for people to make mistakes.
    - csours 7y ago
      To amplify: think about road signs when you are trying to get somewhere new, especially in places with weird parking or entrances. After you've figured it out, it's easy, but everyone is a first timer at some point.
    - 7y ago
greenyoda 7y ago
HR is not an organization that should be arbitrating technical issues that they don't know anything about. The solution is for technical managers - at all levels, from team leads up to the CEO - to actually provide leadership to their teams. This can consist of training people about why it's important to do things right and setting up incentives to do things right (e.g., don't promote people who get things done quickly by cutting corners). To me, this story sounds like a classical management failure (been there, done that myself).
geofft 7y ago
But I don't want them to burn down someone else's prod, I want them to keep building features for my company. (Worse, if the other company has a misuse-resistant deployment system like the one the author alludes to, they might not burn down that company's prod and then beat us in the market while we're busy interviewing replacements!)
triceratops 7y ago
That's dumb. If your system makes it possible for someone to screw up, someone will eventually screw up. And if screwing up (rather than being malicious or stupid) results in people getting fired, then no work will ever get done for fear of getting fired. You have to fix the system so it's really hard to screw up.
csours 7y ago
"A person is smart. People are dumb, panicky dangerous animals and you know it." When I was first hired into a manufacturing plant I wondered why it was so hard to get people to make a quality product. It turns out that people are people, and they will try to do the job you tell them to do, with the tools and instruction you give them. The better the tools are, the less instruction is needed. Output rate and quality is also improved. The poorer the tools are, more instruction is needed; output rate and quality suffer, and at some point safety as well. "A poor workman blames their tools"; A poor manager blames their workman after providing them with poor tools. The very first thing you need to do is assess if the worker is following process and if not why not. If they are not following process, is it because of a mistake or a slip-up? Did they intentionally decide not to follow a process because it was difficult? Disaster documentaries are quick to point out where workers failed to follow written procedures, but they VERY RARELY point out how common those contraventions are with no negative impact. If "it's stupid but it works" it may not be stupid, or you may have gotten lucky. https://www.youtube.com/watch?v=WPMMNvYTEyI https://www.youtube.com/watch?v=WPMMNvYTEyI

hpoe 7y ago

I work at a very large IT org (1000+) and the only way I have ever seen anything change substantially and persistently was when the change was introduced and made it far easier to do the right thing. Ultimately if you want people to do the right thing you can't force them into it, you have to make them want it. It seems to me there are lots of opinions on how to change an organization or introduce improvements but ultimately if you aren't making people's lives easier, making them want the change you are trying to bring about, you'll find it almost impossible to succeed, human inertia is just too great.

pjc50 7y ago
This is the "poka-yoke" philosophy.
- deleted 7y ago
  [deleted]
Cthulhu_ 7y ago
The only major changes I've seen in IT organizations of companies was where the existing IT department had declined or was let go over time and any development was in slumber mode. Then an excited manager comes around and erects a new IT organization from nothing (usually leaning heavily on consultants and freelancers), having sold a Digital Transformation to the company. The new IT / developer department spends millions a year in conjuring up a new server stack (cloud native) and a new software architecture (microservices woo), often building on top of existing ancient software (probably running on a mainframe three further abstraction levels down), with the intent of eventually replacing them. Eventually. Maybe. Usually those things got bogged down in the end because after 1-2 years, the first wave of developers get bored and move on to other opportunities, and the architecture they made is poorly abstracted microservices communicating via undocumented and unstructured REST APIs, with not nearly enough end-to-end tests or traceability to reliably make changes or add features. [/rant]
- DerArzt 7y ago
  That got a little to real for me, I started thinking about work.

erulabs 7y ago

Really good post - "making it hard to do the wrong thing" is more or less my entire design ethos when it comes to software, particularly developer tools. Somewhat on-topic, instead of re-inventing the wheel on this one, I've recently been using Flagger (https://flagger.app/ https://flagger.app/) and have been exceptionally happy with it. Automated rollback on error rates, metrics or integration tests. It's so nice to finally be able to stop writing this code at every company I work for!

sebcat 7y ago
> making it hard to do the wrong thing" is more or less my entire design ethos when it comes to software, particularly developer tools. See also poka-yoke: https://en.m.wikipedia.org/wiki/Poka-yoke https://en.m.wikipedia.org/wiki/Poka-yoke
- xyzzy_plugh 7y ago
  That's lovely, thank you!
- Someone 7y ago
  I found it strange that that page doesn’t mention the example I immediately thought of after reading the first sentence: large dangerous machinery such as presses or lasers that require two hands to operate to prevent the case where the operator, in a hurry, still has one hand in the danger area when he presses the “do it” button with the other.
- dctoedt 7y ago
  I'm going to add "poka-yoke" to the list of things that I make students commit to memory in my law school contract-drafting course; it'll be a synonym for "R.O.O.F" — Root Out Opportunities for [Foul]-ups. (This is another example of how legal-contract drafters can learn from the software world, including concepts such as, for example, orthogonality; modularity; R.O.O.F.; D.R.Y. [Don't Repeat Yourself]; graceful failure; and the benefit of sometimes explaining why things are done the way they are, to help future readers understand.)
7y ago

jedieaston 7y ago

It seems like the system for sending out packages should've been designed to be unless this is being sent out as a critical security patch (and signed off by a security manager or something in the log), the package is deployed to 5% of the fleet per day (unless someone opts in to get it early for some reason on a package-by-package basis). In a big fleet, anything else would still be chaos if someone clicked the wrong button.

paxys 7y ago

Large-scale deploy systems for a distributed environment are ridiculously hard to build. We had to set one up quickly and on a budget, so in lieu of fancy engineering we set up three rules: - Any change, whether it's a one-line config update or an entirely new backend, can only go out to a maximum of x% of hosts at a time - Hosts are taken fully out of rotation before getting an update - Every change has to be backwards compatible, and there can be no side-effects of random rollbacks, sometimes even across several versions The first two are easy to enforce programmatically, and the third one relies on the team. While the trade-off is that it's harder to make relatively simple changes, it has served us well so far.

seanwilson 7y ago
> Every change has to be backwards compatible, and there can be no side-effects of random rollbacks, sometimes even across several versions How does that rule work out in practice? If you're doing staged rollouts of new builds because you think there's a decent chance those new builds have bugs in them, shouldn't you realistically expect that rollbacks could have unexpected side-effects?
- tantalor 7y ago
  > expect that rollbacks could have unexpected Expect the unexpected? Sure anything can happen. But paxys is specifically referring to expected (known) side-effects of rollbacks. For example if Foo and Bar are on versions Foo.3 and Bar.3, and we know rolling back to Bar.2 breaks Foo.3, then you broke backwards compatibility. Generally all combinations of possible rollbacks must continue to work, e.g., {Foo.2, Foo.3} x {Bar.2, Bar.3}. The number of rollbacks is limited to 1 or 2 versions, e.g., Foo.1 and Bar.1 cannot be used anymore.
gaogao 7y ago
The third one is not too bad to test programmatically, if you write a test harness that spins up a mixed version system.
hedora 7y ago
Downgrade should “just work” (with help from the team) for stateless services, since the old and new have to coexist during upgrade. However, if you’re pushing out new storage systems, supporting downgrade ends up lowering overall reliability in practice. It takes many times more effort to implement and test the downgrade of every disk layout change than it does to ensure that upgrades will always succeed in practice.

aahortwwy 7y ago

It's funny that the big tech companies have such high hiring bars and yet when you get inside there are large teams of people actively working to protect the company from the stupidity of their colleagues.

hinkley 7y ago
I’m mostly trying to protect myself from my own stupidity. That other people fall prey to the same issues in many scenarios just makes the pill go down easier for them.
- EdwardDiego 7y ago
  That's the story of my development life! I know I'm human, and as such, I will err, but I really don't want to be erring as humans do in production. I feel like my "capacity to err over time" graph is U shaped with a bit of an M tendency towards the end - as a newbie, lots of ways to make mistakes without realising, then as my skills increased, realised the "known unknowns", and coded and released very cautiously, but when I blurred into a senior, ran the risk of thinking "I know this" without checking that all my assumptions and prior knowledge held true, and learned the caution of the intermediate developer again.
  - hinkley 7y ago
    I had a boss who insisted on calling the root account stupiduser, because when a server gets properly fucked up it's invariably because some stupid user did the wrong thing...
- aahortwwy 7y ago
  When I say stupidity I don't mean mistakes. From the article: > At that point, it was decided that something easier had to be done. The barrier to entry for safe and sane rollouts had to be lowered even more to bring more teams on board. Why does something have to be easy for a team of professional software engineers to do it correctly? Make things easier to improve productivity, sure, but in my opinion awkward/difficult/complex tooling isn't a license to cut corners.

greendave 7y ago

The joke about building idiot-proof software comes to mind, but it's surprising to me how many systems really don't do a good job of making the right thing easy and the wrong hard. Dangerous defaults have consequences! Of course sometimes there's no substitute for learning things the hard way. Where practical, I've found having the responsible team on the hook for cleaning up their mess tends to work wonders in terms of getting people to be appropriately careful. That's not to say mistakes can't still happen, but it cuts down a lot.

csours 7y ago

> "This was one of those times when we needed to build a system that made it stupidly easy to do the right thing, so that it would actually be MORE work to do the wrong thing" This is basically the core of UX. It's kind of amazing to me that UX is so poorly represented in management tools.

diebeforei485 7y ago

This lesson, but in a UX context - Apple generally does a good job of making it hard for users to do the wrong thing (for their definition of wrong).

primitivesuave 7y ago

Really interesting post. Can anyone comment on how serverless deployments help and/or exacerbate the deployment problem? To give a theoretical starting point - Acme corporation has a serverless API deployed at api.acme.com powering website acme.com. The staging system is deployed at staging.api.acme.com, and the corresponding web app (e.g. staging.acme.com) targets that API. Staging gets deployed to production via a pull request that triggers CI/CD, which has no ability to do an X% rollout (i.e. the underlying serverless function is updated). Developer John Smith is writing a feature and merging to his personal branch "jsmith" - he can test his branch with jsmith.staging.api.acme.com and a corresponding development web app. When he's finished the feature, he does a PR to the staging branch, which is packaged into a PR to live. You may be able to see I'm trying to get input on our own release system :) It's worked quite well so far but I'm still paranoid about not having gradual roll-out - in my experience in bigger companies, there was always some A/B testing in place so bad deploys could be reverted quickly. If the sysop gurus of HN gathering on this thread can pass any comments/critique it would be greatly appreciated.

mandelbrotwurst 7y ago
Why would the ability to roll back deploys be tied to the ability to A/B test?
- icebraining 7y ago
  I'm guessing the A/B functionality was being used as a Blue-Green deployment system.
hibbelig 7y ago
The post was talking about fleets of machines, where just restricting the number of machines that do things seems to be useful. But you only seem to have one machine in production (or one for the API plus one for the website), so it doesn't make sense to deploy to 10% of the machines. What you can do is to arrange things so that each individual upgrade can be rolled back. Let's say you want to rename a parameter in the API. You have to make a backward-compatible change to the API so that the existing website keeps working. In other words, you provide two calls, one with the old parameter name and a different call with the new parameter name. Then you deploy that. If there are problems, you can roll back to the previous API version. The next step is to change the website to invoke the new API call with the new parameter name. If that fails, you can go back to the previous website. Let's say it succeeds. Now you have a website using the new API with the new parameter name, and an API providing both the old and the new parameter name. The next step is to update the API to remove the old parameter name. If it turns out you forgot a place in the website where the old parameter is used, you can roll back to the previous API. If it goes well, you now have a website that uses the new parameter name, and also an API that provides the new parameter name, and the old parameter name is no longer used. It took a long time, but now it's mission accomplished and problem solved.

hackeryogi 7y ago

This is quite the representative case between Infra teams (or PaaS or SRE - based on where you're located) and application teams. Being part of the platform team, I've been in the exact same situation before at an ex employer. We assumed that _infra problems_ were _everyone's problems_ and how can anyone who cares about their application _not_ adopt to the latest-and-greatest-platform (or tool) we released. A lot of time was spent on figuring out the 'best approach' to getting everyone onboard quickly - we tried the carrot and the stick. We failed miserably at both. The trouble is, as someone else mentioned here - we didn't really go and _talk to the customer_. Internal customers are customers too - and 'product' thinking should be equally applied to all internal platforms. The general rule applies - talk to the customer and understand what they're going through. They have their own sets of problems and priorities. (a) Is the latest way actually _saving them a lot of time_ (b) Can we take another 2 weeks and make it even easier for them to come on board? (c) If they're really busy and/or lack the expertise, can we carve out some time to give enough training sessions? Yet, this never happens and we invariably end up blaming the application teams for _not doing this seemingly simple_ change. It may seem downright stupid from the platform engineer's PoV, but we are partly to blame. This also leads me to think that the cloud wars will be won by the <cloud-provider> that does this job better - of understanding the customer needs on a day to day basis and building platforms for them.

user5994461 7y ago

Been there, done that. The next rant will be: Massive outage because of a bug that was fixed last month. But the fix only shipped to 50% of the hosts, because the deployment system doesn't deploy everywhere by default.

draw_down 7y ago
And it will somehow be the users’ fault, not the deploy system’s fault.

hinkley 7y ago

Sounds like a setup for immutable servers, and I think I'm okay with that. If new copies of your service don't come up cleanly, you do not power down the old ones. If old servers can't talk to the new service, you get alarm bells going off while most of the old copies are still up and running. You've also partially serialized the deployment so you get some peak shaving. I wonder sometimes if the problem is that despite our protestations (and things like naming servers by unpronounceable names), we still think of servers running, instead of instances running. So the fact that the 20 copies of my service keep bouncing from machine to machine is at the very least a logistics issue, and possibly unsettling.