9 ms·

Knightmare: A DevOps Cautionary Tale (2014)

207 points by strzalek 12y ago

Mandatum 12y ago
I remember reading a summary of this when it occurred in 2012. It's obvious to everyone here what SHOULD have been done, and I find this pretty surprising in the finance sector.. Also your submission should probably have (2014) in the title.
- johansch 12y ago
  The art of rehashing old stories for exposure. Without mentioning where you got your material of course.
  - Mandatum 12y ago
    https://news.ycombinator.com/item?id=4333089 https://news.ycombinator.com/item?id=4333089 I believe it was this, as it was a Guardian article - which has since been subject to the digital graveyard.
- bsdpython 12y ago
  "I find this pretty surprising in the finance sector.." ...not to anyone who has worked in financial software
recursive 12y ago
"Power Peg"? More like powder keg.
solarmist 12y ago
Why would they repurpose an old flag at all? That seems crazy to me unless it was something hardware bound.
- vertex-four 12y ago
  To keep the messages as short as possible, to reduce the time-costs of transmitting and processing them. It's HFT, they do things like that.
  - jemfinch 12y ago
    I assume "flag" in this context means something akin to a command-line flag.
    - socceroos 12y ago
      I assumed it was a single bit in a string of them that describes a message.
      - MattHeard 12y ago
        You're correct. Single bits representing boolean values are often called "flags".
    - matteotom 12y ago
      Except it's probably a byte or less.
      - solarmist 12y ago
        For highly efficient messages I'll bet they did bit packing, so it was probably only a single bit, which is why they'd want to reuse it rather than completely change the message format.
  - mason240 12y ago
    Which is why I don't feel bad at all. Live by the sword, die by the sword.
    - otakucode 12y ago
      I was honestly surprised the NYSE/NASDAQ didn't step in and reverse the trades. They've done so in the past when automated trading systems went off the rails. I am glad they didn't do so, as such favoritism is completely unfair and sends a terrible message to HFT companies, but it was still surprising.
- bsdpython 12y ago
  I've seen it done on a regular basis at a large financial institution. Like, so often that almost half the fields had a different meaning now or have dual purpose on some old systems.
ratsbane 12y ago
I can't read something like this without feeling really bad for everyone involved and taking a quick mental inventory of things I've screwed up in the past or potentially might in the future. Pressing the enter key on anything that affects a big-dollar production system is (and should be) slightly terrifying.
- click170 12y ago
  I have the same fear, but I wonder if that fear stems from lack of training and or documentation and or time. When I ask myself why I am afraid of deploying to production servers its always because I don't fully understand what the deployment process does. If I had to deploy manually I would be lost at sea. Is it just nievete thinking that enough documentation and training makes that fear go away?
  - vinceguidry 12y ago
    It is, ultimately, a management problem. Computer systems are rarely useful on their own. They need to be attached to some other process in order to derive any value from them. Sometimes it's a network, somethings it's a factory, sometimes it's just process run by humans. In most domains that you attach computers to, it's possible for one person to understand both the computer side of it and the other side of it. Web development is like this, you can easily understand both the web and computer systems, as one is really just another version of the other. If you're attaching the computer system to another very hard-to-understand system, like, say, a surgical robot, then the person that understands both domains well enough to avoid problems like this is a unicorn, there might well be only one person in the world that's built up expertise in both fields. To get around this you need careful, effective management of the two pools of labor, and robust dialogue between the two teams. The second this starts breaking down, is the second you start marching down the road to disaster. In the aforementioned example, everyone would be well-aware of the failure modes. So it's a bit easier to manage. In finance, failure modes can be so subtle, particularly as the system grows more complex, that they can escape detection by both teams unless they're both checking each other's work and keeping each other honest. Institutional knowledge transfer has to constantly be happening, areas of ignorance on both sides have to constantly be appraised and plans undertaken to reduce said ignorance. The more everyone knows about the entire system, the more likely it will be that critical defects like this can be discovered before they strike. This kind of effective interaction of those at the bottom can only be organized and directed at the top. It's very much a "captains winning the war"-type situation, but captains can't lead without support from the generals.
NhanH 12y ago
Everytime I'm reading the story, there is one question that I've never understood: why can't the just shutdown the servers itself? There ought to be some mechanism to do that. I mean, $400 millions is a lot of money to not just bash the server with a hammer. It seems like they realized the issue early on and was debugging for at least part of the 45 minutes. I know they might not have physical access to the server, but wouldn't there be any way to do a hard reboot?
- jpatokal 12y ago
  Violently shutting down trading also exposes the bank to significant risk, eg. massively leveraged trades on stocks that were meant to be held for 3 milliseconds suddenly hanging on until the system's back up. In hindsight, this would still have been preferable to losing $400 million, but quite obviously nobody at the time realized just how catastrophic this was going be.
  - xyzzy123 12y ago
    With the benefit of hindsight I think company wide realtime dashboard would be high priority. I guess actual deployment procedures would be higher though.
    - geocar 12y ago
      They had a wire-protocol they were incrementally updating without a version-byte. That is caused by bad engineering, and they needed to do less of that. "Realtime dashboard" and "deployment procedures" are more engineering, not less.
    - zhte415 12y ago
      If the system was not counting its trades, would they show up in the dashboard? A reconciliation would be necessary, which would come from where the orders were being sent to (the exchange), but with millisecond delays a realtime dashboard seems only necessary for this case (not that it is a bad case) and while End Of Day reconciliations are needed, I'd be interested if anyone knew of exchange requirements for intra-day trading reconciliations?
serve_yay 12y ago
If you fill the basement with oily rags for ten years, when the building goes up in flames, is it the fault of the guy who lit a cigarette?
- wingerlang 12y ago
  Did the article specify that some one person was at fault?
  - serve_yay 12y ago
    This is a story about the importance of DevOps, i.e., this company went bankrupt because of poor DevOps practices.
- dredmorbius 12y ago
  In the case of wildfires, why yes, yes indeed we do.
  - serve_yay 12y ago
    The conditions for wildfire do not arise out of neglect and laziness.
    - dredmorbius 12y ago
      The conditions for risk from wildfire do arise out of a multitude of factors. Including discounting hazards, poor construction, insufficient warning or evacuation capabilities, and more. But more to the point: the conditions in which wildfires are common involve such insanely high risk of conflagration that the least spark can set them off. And do. When you've got to tell people not to mow lawns or trim brush for fear of stray sparks setting off tinder-dry brush, you're simply sitting on top of (or in the midst of) a bomb. And that is the reality in much of the world -- Australia, the western US, Greece, and elsewhere. Or a hot car exhaust from a parked vehicle. Or broken glass. Or, yes, flame in the form of cigarettes, a campfire, or barbecue. But except for the case of deliberate arson (which does happen), I find the practice of convicting people for what's essentially a mistake waiting to happen to be quite distasteful.
hcarvalhoalves 12y ago
As usual in catastrophic failures, a series of bad decisions had to occur: - They had dead code in the system - They repurposed a flag for a previous functionality - They (apparently) didn't had code reviews - They didn't had a staging environment - They didn't had a tested deployment process - They didn't had a contingency plan to revert the deploy It could be minimized or avoided altogether by fixing just one of the points. Incredible.
- carrotleads 12y ago
  and if you do it all well, you are paid the avg dev salary. The value of a good dev is a realised only when someone screws up.
- UnoriginalGuy 12y ago
  > They (apparently) didn't had code reviews I don't get that. There was no code issue. The old and new code both worked as intended, it was a deployment and deployment-verification problem. > They didn't had a staging environment Yes they did. They staged the new code and tested it. They did a slow deployment also. > They didn't had a contingency plan to revert the deploy They did revert the deploy within the 45 minutes. It made it worse. I think you need to re-read the article. Your assessment is strange given the event.
  - paulrademacher 12y ago
    Code review could have been another set of eyes to predict the problem of reusing a flag.
    - solarmist 12y ago
      If the message was as compact and low level as possible it was probably a bit flag, so in that context it makes sense to repurpose it. Being so removed from binary and bit level interactions it can be easy to forget things like this.
    - saalweachter 12y ago
aosmith 12y ago
Wasn't Knight in trouble for some other things as well?
bevacqua 12y ago
Ah yes, this story is legendary. I discuss it in my JavaScript Application Design book[1]. Chaos-monkey server-ball-wrecking sounds like a reasonable way to mitigate this kind of issues (and sane development/deployment processes, obviously) [1]: http://bevacqua.io/bf http://bevacqua.io/bf
danbruc 12y ago
What really looks broken to me in this story is the financial system. It has become an completely artificial and lunatic system that has almost nothing to do with the real - goods and services producing - economy.
- raincom 12y ago
  Trickle down economics = financialization of capital. Those who influence the policy are beneficiaries of the financialization of capital: think of grads from elite law schools and b shools!! If there is no financialization of capital, these grads from elite law/b schools wont get fat bonus.
narrator 12y ago
Somebody was on the other side of all those trades and they made a lot of money that day. That's finance. Nobody loses money, no physical damage gets done and somebody on the other side of the poker table gets all the money somebody else lost.
- noonespecial 12y ago
  Its probably weirder than that. As soon as their system "lost containment" all of that money probably just diffused into the market like smoke amid hundreds of thousands of trades. There's no "big winner" on the other side of the table grumble at. HFT systems aren't like a bunch of poker players slyly eying one another across a table, they're more like electro-chemical gold miners concentrating gold from the rivers of cyanide sludge pumping up from the financial system mines. Heaven help them if they depolarize.
- SixSigma 12y ago
  Also - if they had accidentally made a $450m profit, none of this would ever be public.
otakucode 12y ago
While articles like this are very interesting for explaining the technical side of things, I am always left wondering about the organizational/managerial side of things. Had anyone at Knight Capital Group argued for the need of an automated and verifiable deployment process? If so, why were their concerns ignored? Was it seen as a worthless expenditure of resources? Given how common automated deployment is, I think it would be unlikely that none of the engineers involved ever recommended moving to a more automated system. I encountered something like this about a year ago at work. We were deploying an extremely large new system to replace a legacy one. The portion of the system which I work on required a great deal of DBA involvement for deployment. We, of course, practiced the deployment. We ran it more than 20 times against multiple different non-production environments. Not once in any of those attempts was the DBA portion of the deployment completed without error. There were around 130 steps involved and some of them would always get skipped. We also had the issue that the production environment contained some significant differences from the non-production environments (over the past decade we had, for example, delivered software fixes/enhancements which required database columns to be dropped... this was done on the non-production systems, but was not done on the production environment because dropping the columns would take a great deal of time). Myself and others tried to raise concerns about this, but in the end we were left to simply expect to do cleanup after problems were encountered. Luckily we were able to do the cleanup and the errors (of which there were a few) were able to be fixed in a timely manner. We also benefitted from other portions of the system having more severe issues, giving us some cover while we fixed up the new system. The result, however, could have been very bad. And since it wasn't, management is growing increasingly enamored with the idea of by-the-seat-of-your-pants development, hotfixes, etc. When it eventually bites us as I expect it will, I fear that no one will realize it was these practices that put us in danger.
- bd_at_rivenhill 12y ago
  You should read Charles Perrow's "Normal Accidents" and all will be revealed. This is hardly a new problem. http://www.amazon.com/Normal-Accidents-Living-High-Risk-Technologies/dp/0691004129/ref=sr_1_1?s=books&ie=UTF8&qid=1423026671&sr=1-1&keywords=normal+accidents http://www.amazon.com/Normal-Accidents-Living-High-Risk-Tech...
__abc 12y ago
This must be an old wives tale. I live in Chicago and a trading form on the floor beneath us went bankrupt, in roughly the same time, with a similar "repurposed bit" story. Maybe it's the same one .....
- fnordfnordfnord 12y ago
  Knight Capital. It was a big deal. Not an old wive's tale. http://www.nanex.net/aqck2/3522.html http://www.nanex.net/aqck2/3522.html
  - easytiger 12y ago
    HE was joking.
    - fnordfnordfnord 12y ago
      In that case, "whoosh".
ooOOoo 12y ago
The post is quite poor and suffer a lot from hindsight bias. Following article is so much better: http://www.kitchensoap.com/2013/10/29/counterfactuals-knight-capital/ http://www.kitchensoap.com/2013/10/29/counterfactuals-knight...
- jackgavigan 12y ago
  Great link. Thanks for posting.
rgj 12y ago
Repurposing a flag should be spread over two deployments. First remove the code using the old flag, then verify, then introduce code reusing the flag. Even if the deployment was done correctly, during the deployment there would be old and new code in the system.
beat 12y ago
It's nice to see a more detailed technical explanation of this. I've used the story of Knight Capital is part of my pitching for my own startup, which addresses (among other things) consistency between server configurations. This isn't just a deployment problem. It's a monitoring problem. What mechanism did they have to tell if the servers were out of sync? Manual review is the recommended approach. Seriously? You're going to trust human eyeballs for the thousands of different configuration parameters? Have computers do what computers do well - like compare complex system configurations to find things that are out of sync. Have humans do what humans do well - deciding what to do when things don't look right.
gunnark01 12y ago
I used to work in the HFT, and I don´t understand is why there was no risk controls. They way we did it was to have explicit shutdown/pause rules (pause meaning that the strategy will only try to get flat). The rules where things like: - Too many trades in one direction (AKA. big pos) - P/L down by X over Y - P/L up by X over Y - Orders way off the current price When ever there was a shutdown/pause a human/trader would need to assess the situation and decide to continue or not.
- neomantra 12y ago
  To add insult to injury, Knight got fined for not having appropriate risk controls after this incident.
vijucat 12y ago
I once shut down an algorithmic trading server by hastily typing (in bash): - Ctrl-r for reverse-search through history - typing 'ps' to find the process status utility (of course) - pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh) I got a call from the head Sales Trader within 5 seconds asking why his GUI showed that all Orders were paused. Luckily, our recovery code was robust and I could restart the server and resume trading in half a minute or so. That's $250 million to $400 million of orders on pause for half a minute. Not to mention my heartbeat. Renamed stopserver.sh to stop_server.sh after that incident :| P.S. typing speed is not merely overrated, but dangerous in some contexts. Haste makes waste.
- Beltiras 12y ago
  Repercussions?
  - vijucat 12y ago
    Luckily, and just that one time, there was nothing but disapproving glares.
- phaemon 12y ago
  Might be better having something like the following at the top of your script: echo "This will STOP THE SERVER. Are you sure you want to do this?" echo "Type 'yes' to continue:" read response if [[ $response != "yes" ]] then echo "You must type 'yes' to continue. Aborting." exit 1 fi echo "Stopping server ..." It barely takes any time to type 'yes' but it makes you stop and think.
  - marcosdumay 12y ago
    That'll make it impossible to use in scripts. This remember me the time I tried to fix my "rm -r * .o" with a CLI trash system, instead of doing proper backups. Might be better to start reviewing EVERY command one sends to important servers, and testing them if viable. What probably is the line that vijucat took... that's the line that everybody ends up taking, the only thing that changes is the number of accidents needed.

207 points by strzalek 12y ago

I remember reading a summary of this when it occurred in 2012. It's obvious to everyone here what SHOULD have been done, and I find this pretty surprising in the finance sector.. Also your submission should probably have (2014) in the title.

johansch 12y ago
The art of rehashing old stories for exposure. Without mentioning where you got your material of course.
- Mandatum 12y ago
  https://news.ycombinator.com/item?id=4333089 https://news.ycombinator.com/item?id=4333089 I believe it was this, as it was a Guardian article - which has since been subject to the digital graveyard.
bsdpython 12y ago
"I find this pretty surprising in the finance sector.." ...not to anyone who has worked in financial software

recursive 12y ago

"Power Peg"? More like powder keg.

solarmist 12y ago

Why would they repurpose an old flag at all? That seems crazy to me unless it was something hardware bound.

vertex-four 12y ago
To keep the messages as short as possible, to reduce the time-costs of transmitting and processing them. It's HFT, they do things like that.
- jemfinch 12y ago
  I assume "flag" in this context means something akin to a command-line flag.
  - socceroos 12y ago
    I assumed it was a single bit in a string of them that describes a message.
    - MattHeard 12y ago
      You're correct. Single bits representing boolean values are often called "flags".
  - matteotom 12y ago
    Except it's probably a byte or less.
    - solarmist 12y ago
      For highly efficient messages I'll bet they did bit packing, so it was probably only a single bit, which is why they'd want to reuse it rather than completely change the message format.
- mason240 12y ago
  Which is why I don't feel bad at all. Live by the sword, die by the sword.
  - otakucode 12y ago
    I was honestly surprised the NYSE/NASDAQ didn't step in and reverse the trades. They've done so in the past when automated trading systems went off the rails. I am glad they didn't do so, as such favoritism is completely unfair and sends a terrible message to HFT companies, but it was still surprising.
bsdpython 12y ago
I've seen it done on a regular basis at a large financial institution. Like, so often that almost half the fields had a different meaning now or have dual purpose on some old systems.

ratsbane 12y ago

I can't read something like this without feeling really bad for everyone involved and taking a quick mental inventory of things I've screwed up in the past or potentially might in the future. Pressing the enter key on anything that affects a big-dollar production system is (and should be) slightly terrifying.

click170 12y ago
I have the same fear, but I wonder if that fear stems from lack of training and or documentation and or time. When I ask myself why I am afraid of deploying to production servers its always because I don't fully understand what the deployment process does. If I had to deploy manually I would be lost at sea. Is it just nievete thinking that enough documentation and training makes that fear go away?
- vinceguidry 12y ago
  It is, ultimately, a management problem. Computer systems are rarely useful on their own. They need to be attached to some other process in order to derive any value from them. Sometimes it's a network, somethings it's a factory, sometimes it's just process run by humans. In most domains that you attach computers to, it's possible for one person to understand both the computer side of it and the other side of it. Web development is like this, you can easily understand both the web and computer systems, as one is really just another version of the other. If you're attaching the computer system to another very hard-to-understand system, like, say, a surgical robot, then the person that understands both domains well enough to avoid problems like this is a unicorn, there might well be only one person in the world that's built up expertise in both fields. To get around this you need careful, effective management of the two pools of labor, and robust dialogue between the two teams. The second this starts breaking down, is the second you start marching down the road to disaster. In the aforementioned example, everyone would be well-aware of the failure modes. So it's a bit easier to manage. In finance, failure modes can be so subtle, particularly as the system grows more complex, that they can escape detection by both teams unless they're both checking each other's work and keeping each other honest. Institutional knowledge transfer has to constantly be happening, areas of ignorance on both sides have to constantly be appraised and plans undertaken to reduce said ignorance. The more everyone knows about the entire system, the more likely it will be that critical defects like this can be discovered before they strike. This kind of effective interaction of those at the bottom can only be organized and directed at the top. It's very much a "captains winning the war"-type situation, but captains can't lead without support from the generals.

NhanH 12y ago

Everytime I'm reading the story, there is one question that I've never understood: why can't the just shutdown the servers itself? There ought to be some mechanism to do that. I mean, $400 millions is a lot of money to not just bash the server with a hammer. It seems like they realized the issue early on and was debugging for at least part of the 45 minutes. I know they might not have physical access to the server, but wouldn't there be any way to do a hard reboot?

jpatokal 12y ago
Violently shutting down trading also exposes the bank to significant risk, eg. massively leveraged trades on stocks that were meant to be held for 3 milliseconds suddenly hanging on until the system's back up. In hindsight, this would still have been preferable to losing $400 million, but quite obviously nobody at the time realized just how catastrophic this was going be.
- xyzzy123 12y ago
  With the benefit of hindsight I think company wide realtime dashboard would be high priority. I guess actual deployment procedures would be higher though.
  - geocar 12y ago
    They had a wire-protocol they were incrementally updating without a version-byte. That is caused by bad engineering, and they needed to do less of that. "Realtime dashboard" and "deployment procedures" are more engineering, not less.
  - zhte415 12y ago
    If the system was not counting its trades, would they show up in the dashboard? A reconciliation would be necessary, which would come from where the orders were being sent to (the exchange), but with millisecond delays a realtime dashboard seems only necessary for this case (not that it is a bad case) and while End Of Day reconciliations are needed, I'd be interested if anyone knew of exchange requirements for intra-day trading reconciliations?

serve_yay 12y ago

If you fill the basement with oily rags for ten years, when the building goes up in flames, is it the fault of the guy who lit a cigarette?

wingerlang 12y ago
Did the article specify that some one person was at fault?
- serve_yay 12y ago
  This is a story about the importance of DevOps, i.e., this company went bankrupt because of poor DevOps practices.
dredmorbius 12y ago
In the case of wildfires, why yes, yes indeed we do.
- serve_yay 12y ago
  The conditions for wildfire do not arise out of neglect and laziness.
  - dredmorbius 12y ago
    The conditions for risk from wildfire do arise out of a multitude of factors. Including discounting hazards, poor construction, insufficient warning or evacuation capabilities, and more. But more to the point: the conditions in which wildfires are common involve such insanely high risk of conflagration that the least spark can set them off. And do. When you've got to tell people not to mow lawns or trim brush for fear of stray sparks setting off tinder-dry brush, you're simply sitting on top of (or in the midst of) a bomb. And that is the reality in much of the world -- Australia, the western US, Greece, and elsewhere. Or a hot car exhaust from a parked vehicle. Or broken glass. Or, yes, flame in the form of cigarettes, a campfire, or barbecue. But except for the case of deliberate arson (which does happen), I find the practice of convicting people for what's essentially a mistake waiting to happen to be quite distasteful.

hcarvalhoalves 12y ago

As usual in catastrophic failures, a series of bad decisions had to occur: - They had dead code in the system - They repurposed a flag for a previous functionality - They (apparently) didn't had code reviews - They didn't had a staging environment - They didn't had a tested deployment process - They didn't had a contingency plan to revert the deploy It could be minimized or avoided altogether by fixing just one of the points. Incredible.

carrotleads 12y ago
and if you do it all well, you are paid the avg dev salary. The value of a good dev is a realised only when someone screws up.
UnoriginalGuy 12y ago
> They (apparently) didn't had code reviews I don't get that. There was no code issue. The old and new code both worked as intended, it was a deployment and deployment-verification problem. > They didn't had a staging environment Yes they did. They staged the new code and tested it. They did a slow deployment also. > They didn't had a contingency plan to revert the deploy They did revert the deploy within the 45 minutes. It made it worse. I think you need to re-read the article. Your assessment is strange given the event.
- paulrademacher 12y ago
  Code review could have been another set of eyes to predict the problem of reusing a flag.
  - solarmist 12y ago
    If the message was as compact and low level as possible it was probably a bit flag, so in that context it makes sense to repurpose it. Being so removed from binary and bit level interactions it can be easy to forget things like this.
  - saalweachter 12y ago

aosmith 12y ago

Wasn't Knight in trouble for some other things as well?

bevacqua 12y ago

Ah yes, this story is legendary. I discuss it in my JavaScript Application Design book[1]. Chaos-monkey server-ball-wrecking sounds like a reasonable way to mitigate this kind of issues (and sane development/deployment processes, obviously) [1]: http://bevacqua.io/bf http://bevacqua.io/bf

danbruc 12y ago

What really looks broken to me in this story is the financial system. It has become an completely artificial and lunatic system that has almost nothing to do with the real - goods and services producing - economy.

raincom 12y ago
Trickle down economics = financialization of capital. Those who influence the policy are beneficiaries of the financialization of capital: think of grads from elite law schools and b shools!! If there is no financialization of capital, these grads from elite law/b schools wont get fat bonus.

narrator 12y ago

Somebody was on the other side of all those trades and they made a lot of money that day. That's finance. Nobody loses money, no physical damage gets done and somebody on the other side of the poker table gets all the money somebody else lost.

noonespecial 12y ago
Its probably weirder than that. As soon as their system "lost containment" all of that money probably just diffused into the market like smoke amid hundreds of thousands of trades. There's no "big winner" on the other side of the table grumble at. HFT systems aren't like a bunch of poker players slyly eying one another across a table, they're more like electro-chemical gold miners concentrating gold from the rivers of cyanide sludge pumping up from the financial system mines. Heaven help them if they depolarize.
SixSigma 12y ago
Also - if they had accidentally made a $450m profit, none of this would ever be public.

otakucode 12y ago

While articles like this are very interesting for explaining the technical side of things, I am always left wondering about the organizational/managerial side of things. Had anyone at Knight Capital Group argued for the need of an automated and verifiable deployment process? If so, why were their concerns ignored? Was it seen as a worthless expenditure of resources? Given how common automated deployment is, I think it would be unlikely that none of the engineers involved ever recommended moving to a more automated system. I encountered something like this about a year ago at work. We were deploying an extremely large new system to replace a legacy one. The portion of the system which I work on required a great deal of DBA involvement for deployment. We, of course, practiced the deployment. We ran it more than 20 times against multiple different non-production environments. Not once in any of those attempts was the DBA portion of the deployment completed without error. There were around 130 steps involved and some of them would always get skipped. We also had the issue that the production environment contained some significant differences from the non-production environments (over the past decade we had, for example, delivered software fixes/enhancements which required database columns to be dropped... this was done on the non-production systems, but was not done on the production environment because dropping the columns would take a great deal of time). Myself and others tried to raise concerns about this, but in the end we were left to simply expect to do cleanup after problems were encountered. Luckily we were able to do the cleanup and the errors (of which there were a few) were able to be fixed in a timely manner. We also benefitted from other portions of the system having more severe issues, giving us some cover while we fixed up the new system. The result, however, could have been very bad. And since it wasn't, management is growing increasingly enamored with the idea of by-the-seat-of-your-pants development, hotfixes, etc. When it eventually bites us as I expect it will, I fear that no one will realize it was these practices that put us in danger.

bd_at_rivenhill 12y ago
You should read Charles Perrow's "Normal Accidents" and all will be revealed. This is hardly a new problem. http://www.amazon.com/Normal-Accidents-Living-High-Risk-Technologies/dp/0691004129/ref=sr_1_1?s=books&ie=UTF8&qid=1423026671&sr=1-1&keywords=normal+accidents http://www.amazon.com/Normal-Accidents-Living-High-Risk-Tech...

__abc 12y ago

This must be an old wives tale. I live in Chicago and a trading form on the floor beneath us went bankrupt, in roughly the same time, with a similar "repurposed bit" story. Maybe it's the same one .....

fnordfnordfnord 12y ago
Knight Capital. It was a big deal. Not an old wive's tale. http://www.nanex.net/aqck2/3522.html http://www.nanex.net/aqck2/3522.html
- easytiger 12y ago
  HE was joking.
  - fnordfnordfnord 12y ago
    In that case, "whoosh".

ooOOoo 12y ago

The post is quite poor and suffer a lot from hindsight bias. Following article is so much better: http://www.kitchensoap.com/2013/10/29/counterfactuals-knight-capital/ http://www.kitchensoap.com/2013/10/29/counterfactuals-knight...

jackgavigan 12y ago
Great link. Thanks for posting.

rgj 12y ago

Repurposing a flag should be spread over two deployments. First remove the code using the old flag, then verify, then introduce code reusing the flag. Even if the deployment was done correctly, during the deployment there would be old and new code in the system.

beat 12y ago

It's nice to see a more detailed technical explanation of this. I've used the story of Knight Capital is part of my pitching for my own startup, which addresses (among other things) consistency between server configurations. This isn't just a deployment problem. It's a monitoring problem. What mechanism did they have to tell if the servers were out of sync? Manual review is the recommended approach. Seriously? You're going to trust human eyeballs for the thousands of different configuration parameters? Have computers do what computers do well - like compare complex system configurations to find things that are out of sync. Have humans do what humans do well - deciding what to do when things don't look right.

gunnark01 12y ago

I used to work in the HFT, and I don´t understand is why there was no risk controls. They way we did it was to have explicit shutdown/pause rules (pause meaning that the strategy will only try to get flat). The rules where things like: - Too many trades in one direction (AKA. big pos) - P/L down by X over Y - P/L up by X over Y - Orders way off the current price When ever there was a shutdown/pause a human/trader would need to assess the situation and decide to continue or not.

neomantra 12y ago
To add insult to injury, Knight got fined for not having appropriate risk controls after this incident.

vijucat 12y ago

I once shut down an algorithmic trading server by hastily typing (in bash): - Ctrl-r for reverse-search through history - typing 'ps' to find the process status utility (of course) - pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh) I got a call from the head Sales Trader within 5 seconds asking why his GUI showed that all Orders were paused. Luckily, our recovery code was robust and I could restart the server and resume trading in half a minute or so. That's $250 million to $400 million of orders on pause for half a minute. Not to mention my heartbeat. Renamed stopserver.sh to stop_server.sh after that incident :| P.S. typing speed is not merely overrated, but dangerous in some contexts. Haste makes waste.

Beltiras 12y ago
Repercussions?
- vijucat 12y ago
  Luckily, and just that one time, there was nothing but disapproving glares.
phaemon 12y ago
Might be better having something like the following at the top of your script: echo "This will STOP THE SERVER. Are you sure you want to do this?" echo "Type 'yes' to continue:" read response if [[ $response != "yes" ]] then echo "You must type 'yes' to continue. Aborting." exit 1 fi echo "Stopping server ..." It barely takes any time to type 'yes' but it makes you stop and think.
- marcosdumay 12y ago
  That'll make it impossible to use in scripts. This remember me the time I tried to fix my "rm -r * .o" with a CLI trash system, instead of doing proper backups. Might be better to start reviewing EVERY command one sends to important servers, and testing them if viable. What probably is the line that vijucat took... that's the line that everybody ends up taking, the only thing that changes is the number of accidents needed.