6 ms·

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Sub

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, and the impact happens even faster. The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way.

ledauphin 3y ago
The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."
- SoftTalker 3y ago
  Or at least not by a developer who has made that sort of mistake in the past. I don't know what software engineering programs teach these days, but in the 1980s there was very little inclusion of case studies of things that went wrong. This was unlike the courses in the business school (my undergrad was CS major + business minor) nor I would presume what real engineering disciplines teach. My first exposure to a fuckup in production was a fuckup in production on my first job.
- lopkeny12ko 3y ago
  I doubt any manager or VP cares or knows enough about the technical details of the code to dictate the name that should be used for a feature flag, of all things.
- andersa 3y ago
  I wonder if this code was written in c++ or similar, the flags were actually a bitfield, and they repurposed it because they ran out of bits. Need a space here? Oh, let's throw out this junk nobody used in 8 years and there we go...
  - CraigRo 3y ago
    It is very hard to change the overall size of the messages, and there's a lot of pressure to keep them short. So it could have been a bitfield or several similar things... e.g a value in a char field
  - bluelightning2k 3y ago
    This sounds particularly plausible with it being high frequency trading. Those presumably have optimisations few other applications have
  - paradox242 3y ago
    This is the first thing I thought of because otherwise this story doesn't make a lot of sense.
- Nekhrimah 3y ago
  >whoever decided that reusing an old flag was a good idea. My understanding is that in high frequency trading, minimizing the size of the transmission is paramount. Hence re-purposing an existing flag, rather than adding size to the packet makes some sense.
- manicennui 3y ago
  9 times out of 10, I see developers making the mistakes that everyone seems to want to blame on non-technical people. There is a massive amount of software being written by people with a wide range of capabilities, and a large number of developers never master the basics. It doesn't help that some of the worst tools "win" and offer little protection against many basic mistakes.
  - hinkley 3y ago
    For a group who so thoroughly despises bosses that operate on 'blame allocation', we spend a lot of time shopping around for permission to engage in reckless behavior. Most people would call that being a hypocrite. Whereas I would call it... no, hypocrite works just fine.
    - PH95VuimJjqBqy 3y ago
      At the company I work, we have a team that took 3 weeks and multiple tries to get an API response (JSON) capitalized properly (camelCase to PascalCase). When I tried to talk to the tech lead about it, his response is that SAFe would have prevented the issue (it was discovered by another team who consumes their API). Throughout the entire thing this tech lead maintained his team didn't do anything and that the problem was the process. yeah, no. I have 25+ years of experience as a developer, it doesn't take 3+ weeks to fix the casing of a JSON property name. I eventually had to be the bad guy and tell them their work was unacceptable because they themselves couldn't recognize it. Only when I did it, I ran it up the chain because if the tech lead doesn't see the problem then I need someone who can help them see the problem. For some people there's a "responsibility shield" that's so strong you can never get through to them.
  - xupybd 3y ago
    You have to assume people will make mistakes. A great book on that is this https://www.thenile.co.nz/books/sidney-dekker/the-field-guide-to-understanding-human-error/9781472439055?gad=1&gclid=Cj0KCQjw0vWnBhC6ARIsAJpJM6fCSM35CckKh3lq15AyUpaXP-APy_buuxkJnszBeiFagUzbtDugVWUaAoUzEALw_wcB https://www.thenile.co.nz/books/sidney-dekker/the-field-guid...
  - 3y ago
- mijoharas 3y ago
  At the very least have a two deploys - actually removing the old code that relies on it and then repurposing it. Giant foot gun to do it all in one especially without any automated deploys.
  - bluelightning2k 3y ago
    Good point. Actually I think I'll treat this as a best practice in general when there's a transition
  - wink 3y ago
    That assumes that you have a stable, reliable, quick process to roll out updates. Sounds like they didn't, so maybe they worked on the "oh better add this feature, it's our only chance this month" pattern.
- hinkley 3y ago
  Flag recycling is a task that should be measured in months to quarters, and from what I recall of the postmortem they tried to achieve it in weeks, which is just criminally stupid. It's this detail of the story which flips me from sympathy to schadenfreude. You dumb motherfuckers fucked around and found out.
dmurray 3y ago
> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill switch because of the opportunity cost (perhaps pulling the switch at the wrong time costs $500k in opportunity). I worked for a competitor to Knight at the time, and we deployed terrible bugs to production all the time, and during post mortems we couldn't fathom the same thing happening to us. A dozen automated systems would have kicked in to stop individual trades, and any senior trader or operations person could have got a kill switch pulled with 60 seconds of dialogue, and not feared the repercussions. Actually, we made way less of Knight's $400m than we could have because our risk systems kept shutting strategies down because what was happening was "too good to be true".
- mpeg 3y ago
  It’s nice to see your perspective as someone familiar with better systems. I have always found this story fascinating; in my junior days I worked at a relatively big adtech platform (ie billions of impressions per day) and as cowboy as we were about lots of things, all our systems always had kill switches that could stop spending money and I could have pulled them with minimal red tape if I suspected something was wrong. And this was for a platform where our max loss for an hour would have hurt but not killed the business (maybe a six figure loss), I can’t imagine not having layers of risk management systems in HFT software.
- deleted 3y ago
  [deleted]
- pxmpxm 3y ago
  They were asleep at the wheel, not unlike all the random brokerages that blew up when swiss central bank pulled the CHF peg in 2015. This is a culture problem - as soon as you load up your trading firm with a bunch of software industry hires, you end up with jiras and change management workflows instead of people on deck that have context for what they're doing. That's the only way to explain reverse scalping for 45 mins straight.
moeris 3y ago
Automated deployments would have allowed you to review the deployment before it happened. A failed deployment could be configured to allow automatic rollbacks. Automated deployments should also handle experiment flags, which could have been toggled to reduce impact. There are a bunch of places where it could have intervened and mitigated/prevented this whole situation.
stusmall 3y ago
I think the big improvement would be consistency. Either all servers would be correct or all servers would be incorrect. The step where "Since they were unable to determine what was causing the erroneous orders they reacted by uninstalling the new code from the servers it was deployed to correctly" wouldn't have had a negative impact. They could have even instantly rolled back. Also if they were using the same automated deployment processes for their test environment they might have even caught this in QA.
lwhi 3y ago
Automated deployments require planning before the time they're executed. If code is involved, someone likely reviews and approves it. There are naturally far more safeguards in place than there would be for a manual deployment.
- justinclift 3y ago
  In an ideal world, sure. In the current one, we have Facebook's "Move fast and break things" being applied to many things where it has no business being. Banking and communications infrastructure comes to my mind, but there are definitely others. :)
  - lwhi 3y ago
    I think the benefits from automated deployments are things that are just par for the course to be honest. Sure, you can mess these things up .. but doing so would involve willful negligence rather than someone's absent mindedness. Basically, I think the takeaway from the article is probably worth taking.
schneems 3y ago
I see this as a problem of not investing enough in the deploy process. (Disclosure: I maintain an open source deploy tool for a living). Charity Majors gave a talk in Euruko that talked a lot about this. Deploy tooling shouldn’t be a bunch of bash scripts in a trench coat, it should be fully staffed, fully tested, and automated within an inch of its life. If you have a deploy process that has some king of immutable architecture, tooling to monitor (failed/stuck/incomplete) rollouts, and the ability to quickly rollback to a prior known good stage then you have layers of protection and an easy course of action for when things do go sideways. It might not have made this problem impossible, but it would have made it harder to happen.
- hooverd 3y ago
  > bash scripts in a trench coat That's an amazing turn of phrase.
- hinkley 3y ago
  I wrote a tool to automate our hotfix process, and people were somewhat surprised that you could kill the process at any step and start over and it would almost always do the right thing. Like how did you expect it to work? Why replace an error prone process with an error prone and opaque one that you can't restart?
- bluelightning2k 3y ago
  Great reply. Counterpoint though: that automation in and of itself is more failure area. I can imagine a similar story where the deployment pipeline incorrectly rolled back due to some change in metric format and caused the infinite loss, for example. The thing with these being a 1 in a million chance is that there's thousands of different hypothetical causes. The more parts the harder to predict an interaction and we've all been blindsided by something. I would personally hate the stress of working on such high stakes releases.
  - schneems 3y ago
    Test test test. If that’s not enough, pick better tools. I’m rewriting bash scripts in rust at work because it gives me the ability to make many invalid states impossible to represent in code. Is it overkill? Maybe, but it is such a huge quality of life improvement. Automated things can fail. Sure. But consider that playbooks are just crappy automation run by unreliable meat computers. Also you can take an iterative approach to automation: - manual playbook only - automate one step of the playbook - if it goes well, move to another. If not, run a retro to figure out out how you can improve it and try again. Stress of failure at a job responsible for deployment architecture is manageable if you have a team and culture built around respecting that stress. There are some areas of code people are more careful around, but largely we make safety a product of our tools and processes and not some heroic “try harder not to screw up” attitude. I find the impact of helping so many developers and their companies rewarding.
Waterluvian 3y ago
Which really means is a failure of leadership for being so incompetent as to allow such an intensely risky situation to exist.
- michaelt 3y ago
  Well, yes and no. Imagine a company where the engineering culture hires macho programmers who love bitmasked flags and manual memory management, and who think memory safe languages and json are for sissies. Imagine they hire lots of graduates who've never worked elsewhere and teach them that their way is best, and everyone who says otherwise doesn't understand real performance. Those 'industry best practices' are written by javascript folks who think a 2 second pageload is fast, and a 200ms pageload is instant, don't you know? And imagine when they make experienced hires, they look for people who have experience with bitmasked flags and manual memory management - which is reasonable enough, they gotta be able to code review that stuff and coach junior employees on working with it. But has the side effect experienced hires won't rock the boat. Now, the nontechnical bosses can ask anyone from the highest engineering leadership to the must junior of peons, and they'll all agree that bitmasked flags are the right way of doing things. Is it so unreasonable for nontechnical bosses to trust the consensus of their engineers on matters of engineering?
hinkley 3y ago
The goal with automation is that the number of unidentified corner cases reduces over time. A manual runbook is a game of, "I did step 12, I think I did step 13, so the next step is 14." that plays out every single time you do server work. The thing with the human brain is that when you interrupt a task you've done a million times in the middle, most people can't reliably discern between this iteration and false memories from the last time they did it. So unless there are interlocks that prevent skipping a step, it's a gamble every single time. And the effort involved in creating interlocks is a large fraction of the cost of automating.
- WJW 3y ago
  1. Print the checklist/runbook out on paper with actual empty boxes next to the steps. 2. Laminate the printed checklist and put it in a big folder. 3. Every time you run the checklist, use a sharpie to mark the checkbox after you've done the step. 4. When you are done with the entire process, use whiteboard cleaner to wipe out the checks again and put the checklist back in the big folder with all the other checklists. This is how every safety critical profession (aviation, shipping, medical, power generation, etc) has worked for decades and unless people are willingly being obtuse it is extremely hard to do it wrong. You just need people to turn off their ego and follow the process instead of trying to show off by doing it from memory. This last part might be more difficult in software settings.
  - MiyamotoAkira 3y ago
    Regarding checklist people like Hollnagel, Wears, Braithwaite, Dekker, ... have done a bit on investigation (Hollnagel mostly on Healthcare, Dekker started on air industry but spread from there). Read the "Safety-I vs Safety-II" paper or the "When a checklist is not enough: How to improve them and what else is needed" paper
    - WJW 3y ago
      Great point and sometimes checklists are indeed not enough. My previous post was triggered more by the "I think I did step 13," part of the post I was responding to. That is not a flaw with checklists but a flaw in the safety culture of the operator team. It should never happen that you lose track of where you are in the process because human memory is unreliable, rather you should fix that through better processes and outsource the memorizing to paper.
rcpt 3y ago
It seems like the kind of thing that would be canaried which is the kind of thing that you'd typically build alongside automated deployment
thrashh 3y ago
I agree. It doesn’t matter if you give an inexperienced person a hammer or a saw — they’ll still screw it up. My biggest pet peeve is they NO ONE ever does failure modeling. I swear everyone builds things assuming it will work perfectly. Then when you mention if one part fails, it will completely bring down everything, they’ll say that it’s a 1 in a million chance. Yeah, the problem isn’t that it’s unlikely, it’s that when it does happen, you’ve perfectly designed your system to destroy itself.
- TheAlchemist 3y ago
  No one, is quite a bold assumption ! It's actually quite routine stuff now in finance at least - to perform some kind of 'fire test' on a regular basis - you shut down some components during the day, and switch to backups solutions, to test everything works smoothly.
movedx 3y ago
CI/CD would have have solved this 100%: > ... one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Read this part again: > ... one of Knight’s technicians did not *copy the new code to one of the eight SMARS computer servers*. Yes, of course a CI/CD pipeline can fail midway through and only partially deploy the code to a partial number of servers, but I doubt it. And even if that were the case, just off the top of my head I can guarantee an Ansible Playbook would have not only stopped the moment that particular transfer failed, the whole Playbook would have therefore failed, and none of the services would have been restarted (because that would be a final step that wouldn't be reached.) This was due to human error and is the very reason CI/CD/automation is a thing. > Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server CI/CD would have have solved this 100%. A "Pull Request" made against a repository of Ansible code (or whatever you flavour is) would have *PREVENTED* the first technician from ever being able to merge the code into master/main (because you have master/main protected right... right?), completely preventing the entire process from ever rolling out without a review, which would have hopefully caught the misaligned configuration. DevOps, which is mostly underpinned by CI/CD, would have solved this 100%. I'm very certain of this.
- SoftTalker 3y ago
  Ansible in my experience will stop trying to run subsequent tasks on a server once one of them fails, but it will go ahead with other servers that match the inventory pattern. So it very well could have successfully updated 7 out of 8 hosts. Maybe there is a switch that will stop everything if any task on any host fails but it's not the default behavior. At least it would have logged an error that hopefully would have been looked at.
  - BoorishBears 3y ago
    I think this is an example of hindsight not always being 20/20 If you replace each step of the post mortem with a CI/CD based alternative, you miss out on the fact CI/CD trivializes designs where this wouldn't have happened. The "easy default" wouldn't be to run a play against 8 hosts manually in your terminal, it'd be run a playbook with them all baked in, and that would fail correctly by default: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_strategies.html https://docs.ansible.com/ansible/latest/playbook_guide/playb... The key here is CI/CD makes it so its actually less work to run that one play than it is to shoot yourself in the foot with 8 separate invocations. Even in the fact of incompetence/laziness/oversight, the general framework makes the right choice
itsthecourier 3y ago
Also, api versioning. They weren't running api versioning on it, they called an old method with a new set of parameters, that shouldn't have been possible in first place
emmelaich 3y ago
Technically, a flag re-use was the most impactful error, code wise. A flag of such importance should not be just on/off; the ON should require a positive response / receipt containing the name and version of the code being turned on. [edit - don't mean for each trade, I mean validation on startup]
devjab 3y ago
> the deployment agent errored while downloading the new binary/code onto the server In that case the build would never be pushed to production. The worst it would accomplish, and this is if your systems fail, is that it will break your staging area. Sure this is in the the ideal world where people actually know how to set up their deployment pipelines correctly, so you’re likely still right in many cases, but you shouldn’t be.
aiunboxed 3y ago
I think the blame is not on either the devops or the developers, it is on the process. If a bug occurs than there should be atleast 5-6 different metrics / alerts that should be able to catch the bug.

ledauphin 3y ago

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

SoftTalker 3y ago
Or at least not by a developer who has made that sort of mistake in the past. I don't know what software engineering programs teach these days, but in the 1980s there was very little inclusion of case studies of things that went wrong. This was unlike the courses in the business school (my undergrad was CS major + business minor) nor I would presume what real engineering disciplines teach. My first exposure to a fuckup in production was a fuckup in production on my first job.
lopkeny12ko 3y ago
I doubt any manager or VP cares or knows enough about the technical details of the code to dictate the name that should be used for a feature flag, of all things.
andersa 3y ago
I wonder if this code was written in c++ or similar, the flags were actually a bitfield, and they repurposed it because they ran out of bits. Need a space here? Oh, let's throw out this junk nobody used in 8 years and there we go...
- CraigRo 3y ago
  It is very hard to change the overall size of the messages, and there's a lot of pressure to keep them short. So it could have been a bitfield or several similar things... e.g a value in a char field
- bluelightning2k 3y ago
  This sounds particularly plausible with it being high frequency trading. Those presumably have optimisations few other applications have
- paradox242 3y ago
  This is the first thing I thought of because otherwise this story doesn't make a lot of sense.
Nekhrimah 3y ago
>whoever decided that reusing an old flag was a good idea. My understanding is that in high frequency trading, minimizing the size of the transmission is paramount. Hence re-purposing an existing flag, rather than adding size to the packet makes some sense.
manicennui 3y ago
9 times out of 10, I see developers making the mistakes that everyone seems to want to blame on non-technical people. There is a massive amount of software being written by people with a wide range of capabilities, and a large number of developers never master the basics. It doesn't help that some of the worst tools "win" and offer little protection against many basic mistakes.
- hinkley 3y ago
  For a group who so thoroughly despises bosses that operate on 'blame allocation', we spend a lot of time shopping around for permission to engage in reckless behavior. Most people would call that being a hypocrite. Whereas I would call it... no, hypocrite works just fine.
  - PH95VuimJjqBqy 3y ago
    At the company I work, we have a team that took 3 weeks and multiple tries to get an API response (JSON) capitalized properly (camelCase to PascalCase). When I tried to talk to the tech lead about it, his response is that SAFe would have prevented the issue (it was discovered by another team who consumes their API). Throughout the entire thing this tech lead maintained his team didn't do anything and that the problem was the process. yeah, no. I have 25+ years of experience as a developer, it doesn't take 3+ weeks to fix the casing of a JSON property name. I eventually had to be the bad guy and tell them their work was unacceptable because they themselves couldn't recognize it. Only when I did it, I ran it up the chain because if the tech lead doesn't see the problem then I need someone who can help them see the problem. For some people there's a "responsibility shield" that's so strong you can never get through to them.
- xupybd 3y ago
  You have to assume people will make mistakes. A great book on that is this https://www.thenile.co.nz/books/sidney-dekker/the-field-guide-to-understanding-human-error/9781472439055?gad=1&gclid=Cj0KCQjw0vWnBhC6ARIsAJpJM6fCSM35CckKh3lq15AyUpaXP-APy_buuxkJnszBeiFagUzbtDugVWUaAoUzEALw_wcB https://www.thenile.co.nz/books/sidney-dekker/the-field-guid...
- 3y ago
mijoharas 3y ago
At the very least have a two deploys - actually removing the old code that relies on it and then repurposing it. Giant foot gun to do it all in one especially without any automated deploys.
- bluelightning2k 3y ago
  Good point. Actually I think I'll treat this as a best practice in general when there's a transition
- wink 3y ago
  That assumes that you have a stable, reliable, quick process to roll out updates. Sounds like they didn't, so maybe they worked on the "oh better add this feature, it's our only chance this month" pattern.
hinkley 3y ago
Flag recycling is a task that should be measured in months to quarters, and from what I recall of the postmortem they tried to achieve it in weeks, which is just criminally stupid. It's this detail of the story which flips me from sympathy to schadenfreude. You dumb motherfuckers fucked around and found out.

dmurray 3y ago

> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill switch because of the opportunity cost (perhaps pulling the switch at the wrong time costs $500k in opportunity). I worked for a competitor to Knight at the time, and we deployed terrible bugs to production all the time, and during post mortems we couldn't fathom the same thing happening to us. A dozen automated systems would have kicked in to stop individual trades, and any senior trader or operations person could have got a kill switch pulled with 60 seconds of dialogue, and not feared the repercussions. Actually, we made way less of Knight's $400m than we could have because our risk systems kept shutting strategies down because what was happening was "too good to be true".

mpeg 3y ago
It’s nice to see your perspective as someone familiar with better systems. I have always found this story fascinating; in my junior days I worked at a relatively big adtech platform (ie billions of impressions per day) and as cowboy as we were about lots of things, all our systems always had kill switches that could stop spending money and I could have pulled them with minimal red tape if I suspected something was wrong. And this was for a platform where our max loss for an hour would have hurt but not killed the business (maybe a six figure loss), I can’t imagine not having layers of risk management systems in HFT software.
deleted 3y ago
[deleted]
pxmpxm 3y ago
They were asleep at the wheel, not unlike all the random brokerages that blew up when swiss central bank pulled the CHF peg in 2015. This is a culture problem - as soon as you load up your trading firm with a bunch of software industry hires, you end up with jiras and change management workflows instead of people on deck that have context for what they're doing. That's the only way to explain reverse scalping for 45 mins straight.

moeris 3y ago

Automated deployments would have allowed you to review the deployment before it happened. A failed deployment could be configured to allow automatic rollbacks. Automated deployments should also handle experiment flags, which could have been toggled to reduce impact. There are a bunch of places where it could have intervened and mitigated/prevented this whole situation.

stusmall 3y ago

I think the big improvement would be consistency. Either all servers would be correct or all servers would be incorrect. The step where "Since they were unable to determine what was causing the erroneous orders they reacted by uninstalling the new code from the servers it was deployed to correctly" wouldn't have had a negative impact. They could have even instantly rolled back. Also if they were using the same automated deployment processes for their test environment they might have even caught this in QA.

lwhi 3y ago

Automated deployments require planning before the time they're executed. If code is involved, someone likely reviews and approves it. There are naturally far more safeguards in place than there would be for a manual deployment.

justinclift 3y ago
In an ideal world, sure. In the current one, we have Facebook's "Move fast and break things" being applied to many things where it has no business being. Banking and communications infrastructure comes to my mind, but there are definitely others. :)
- lwhi 3y ago
  I think the benefits from automated deployments are things that are just par for the course to be honest. Sure, you can mess these things up .. but doing so would involve willful negligence rather than someone's absent mindedness. Basically, I think the takeaway from the article is probably worth taking.

schneems 3y ago

I see this as a problem of not investing enough in the deploy process. (Disclosure: I maintain an open source deploy tool for a living). Charity Majors gave a talk in Euruko that talked a lot about this. Deploy tooling shouldn’t be a bunch of bash scripts in a trench coat, it should be fully staffed, fully tested, and automated within an inch of its life. If you have a deploy process that has some king of immutable architecture, tooling to monitor (failed/stuck/incomplete) rollouts, and the ability to quickly rollback to a prior known good stage then you have layers of protection and an easy course of action for when things do go sideways. It might not have made this problem impossible, but it would have made it harder to happen.

hooverd 3y ago
> bash scripts in a trench coat That's an amazing turn of phrase.
hinkley 3y ago
I wrote a tool to automate our hotfix process, and people were somewhat surprised that you could kill the process at any step and start over and it would almost always do the right thing. Like how did you expect it to work? Why replace an error prone process with an error prone and opaque one that you can't restart?
bluelightning2k 3y ago
Great reply. Counterpoint though: that automation in and of itself is more failure area. I can imagine a similar story where the deployment pipeline incorrectly rolled back due to some change in metric format and caused the infinite loss, for example. The thing with these being a 1 in a million chance is that there's thousands of different hypothetical causes. The more parts the harder to predict an interaction and we've all been blindsided by something. I would personally hate the stress of working on such high stakes releases.
- schneems 3y ago
  Test test test. If that’s not enough, pick better tools. I’m rewriting bash scripts in rust at work because it gives me the ability to make many invalid states impossible to represent in code. Is it overkill? Maybe, but it is such a huge quality of life improvement. Automated things can fail. Sure. But consider that playbooks are just crappy automation run by unreliable meat computers. Also you can take an iterative approach to automation: - manual playbook only - automate one step of the playbook - if it goes well, move to another. If not, run a retro to figure out out how you can improve it and try again. Stress of failure at a job responsible for deployment architecture is manageable if you have a team and culture built around respecting that stress. There are some areas of code people are more careful around, but largely we make safety a product of our tools and processes and not some heroic “try harder not to screw up” attitude. I find the impact of helping so many developers and their companies rewarding.

Waterluvian 3y ago

Which really means is a failure of leadership for being so incompetent as to allow such an intensely risky situation to exist.

michaelt 3y ago
Well, yes and no. Imagine a company where the engineering culture hires macho programmers who love bitmasked flags and manual memory management, and who think memory safe languages and json are for sissies. Imagine they hire lots of graduates who've never worked elsewhere and teach them that their way is best, and everyone who says otherwise doesn't understand real performance. Those 'industry best practices' are written by javascript folks who think a 2 second pageload is fast, and a 200ms pageload is instant, don't you know? And imagine when they make experienced hires, they look for people who have experience with bitmasked flags and manual memory management - which is reasonable enough, they gotta be able to code review that stuff and coach junior employees on working with it. But has the side effect experienced hires won't rock the boat. Now, the nontechnical bosses can ask anyone from the highest engineering leadership to the must junior of peons, and they'll all agree that bitmasked flags are the right way of doing things. Is it so unreasonable for nontechnical bosses to trust the consensus of their engineers on matters of engineering?

hinkley 3y ago

The goal with automation is that the number of unidentified corner cases reduces over time. A manual runbook is a game of, "I did step 12, I think I did step 13, so the next step is 14." that plays out every single time you do server work. The thing with the human brain is that when you interrupt a task you've done a million times in the middle, most people can't reliably discern between this iteration and false memories from the last time they did it. So unless there are interlocks that prevent skipping a step, it's a gamble every single time. And the effort involved in creating interlocks is a large fraction of the cost of automating.

WJW 3y ago
1. Print the checklist/runbook out on paper with actual empty boxes next to the steps. 2. Laminate the printed checklist and put it in a big folder. 3. Every time you run the checklist, use a sharpie to mark the checkbox after you've done the step. 4. When you are done with the entire process, use whiteboard cleaner to wipe out the checks again and put the checklist back in the big folder with all the other checklists. This is how every safety critical profession (aviation, shipping, medical, power generation, etc) has worked for decades and unless people are willingly being obtuse it is extremely hard to do it wrong. You just need people to turn off their ego and follow the process instead of trying to show off by doing it from memory. This last part might be more difficult in software settings.
- MiyamotoAkira 3y ago
  Regarding checklist people like Hollnagel, Wears, Braithwaite, Dekker, ... have done a bit on investigation (Hollnagel mostly on Healthcare, Dekker started on air industry but spread from there). Read the "Safety-I vs Safety-II" paper or the "When a checklist is not enough: How to improve them and what else is needed" paper
  - WJW 3y ago
    Great point and sometimes checklists are indeed not enough. My previous post was triggered more by the "I think I did step 13," part of the post I was responding to. That is not a flaw with checklists but a flaw in the safety culture of the operator team. It should never happen that you lose track of where you are in the process because human memory is unreliable, rather you should fix that through better processes and outsource the memorizing to paper.

rcpt 3y ago

It seems like the kind of thing that would be canaried which is the kind of thing that you'd typically build alongside automated deployment

thrashh 3y ago

I agree. It doesn’t matter if you give an inexperienced person a hammer or a saw — they’ll still screw it up. My biggest pet peeve is they NO ONE ever does failure modeling. I swear everyone builds things assuming it will work perfectly. Then when you mention if one part fails, it will completely bring down everything, they’ll say that it’s a 1 in a million chance. Yeah, the problem isn’t that it’s unlikely, it’s that when it does happen, you’ve perfectly designed your system to destroy itself.

TheAlchemist 3y ago
No one, is quite a bold assumption ! It's actually quite routine stuff now in finance at least - to perform some kind of 'fire test' on a regular basis - you shut down some components during the day, and switch to backups solutions, to test everything works smoothly.

movedx 3y ago

CI/CD would have have solved this 100%: > ... one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Read this part again: > ... one of Knight’s technicians did not *copy the new code to one of the eight SMARS computer servers*. Yes, of course a CI/CD pipeline can fail midway through and only partially deploy the code to a partial number of servers, but I doubt it. And even if that were the case, just off the top of my head I can guarantee an Ansible Playbook would have not only stopped the moment that particular transfer failed, the whole Playbook would have therefore failed, and none of the services would have been restarted (because that would be a final step that wouldn't be reached.) This was due to human error and is the very reason CI/CD/automation is a thing. > Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server CI/CD would have have solved this 100%. A "Pull Request" made against a repository of Ansible code (or whatever you flavour is) would have *PREVENTED* the first technician from ever being able to merge the code into master/main (because you have master/main protected right... right?), completely preventing the entire process from ever rolling out without a review, which would have hopefully caught the misaligned configuration. DevOps, which is mostly underpinned by CI/CD, would have solved this 100%. I'm very certain of this.

SoftTalker 3y ago
Ansible in my experience will stop trying to run subsequent tasks on a server once one of them fails, but it will go ahead with other servers that match the inventory pattern. So it very well could have successfully updated 7 out of 8 hosts. Maybe there is a switch that will stop everything if any task on any host fails but it's not the default behavior. At least it would have logged an error that hopefully would have been looked at.
- BoorishBears 3y ago
  I think this is an example of hindsight not always being 20/20 If you replace each step of the post mortem with a CI/CD based alternative, you miss out on the fact CI/CD trivializes designs where this wouldn't have happened. The "easy default" wouldn't be to run a play against 8 hosts manually in your terminal, it'd be run a playbook with them all baked in, and that would fail correctly by default: https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_strategies.html https://docs.ansible.com/ansible/latest/playbook_guide/playb... The key here is CI/CD makes it so its actually less work to run that one play than it is to shoot yourself in the foot with 8 separate invocations. Even in the fact of incompetence/laziness/oversight, the general framework makes the right choice

itsthecourier 3y ago

Also, api versioning. They weren't running api versioning on it, they called an old method with a new set of parameters, that shouldn't have been possible in first place

emmelaich 3y ago

Technically, a flag re-use was the most impactful error, code wise. A flag of such importance should not be just on/off; the ON should require a positive response / receipt containing the name and version of the code being turned on. [edit - don't mean for each trade, I mean validation on startup]

devjab 3y ago

> the deployment agent errored while downloading the new binary/code onto the server In that case the build would never be pushed to production. The worst it would accomplish, and this is if your systems fail, is that it will break your staging area. Sure this is in the the ideal world where people actually know how to set up their deployment pipelines correctly, so you’re likely still right in many cases, but you shouldn’t be.

aiunboxed 3y ago

I think the blame is not on either the devops or the developers, it is on the process. If a bug occurs than there should be atleast 5-6 different metrics / alerts that should be able to catch the bug.