17 ms·

Knightmare: A DevOps Cautionary Tale (2014)

• firesteelrain 3y ago
  Automation is not a silver bullet. Automation is still designed by humans. Peer reviews, acceptance test procedures, promotion procedures, etc all would have helped. And yes some of those things are manual. Sandbox environments, etc
• supportengineer 3y ago
  They were missing any kind of risk mitigation steps, in their deployment practice.
  • hyperhello 3y ago
    There’s no money for that.
    • earnesti 3y ago
      It is funny, but in one company I was working for, the more people they added the more they neglected all basics, such as backups. There were heavy processes for many things and they were followed very well, but for whatever reasons some really basic things went unnoticed for many years.
  • Gibbon1 3y ago
    I think Goldman Sachs or someone big like that had a similar oopsie. And what happened was the exchange reversed all their bad trades.
• taspeotis 3y ago
  Needs (2014) in the title.
• deleted 3y ago
  [deleted]
• hedora 3y ago
  No continuous deployment system I have worked with would have blocked this particular bug. They were in a situation where they were incrementally rolling out, but the code had a logic bug where the failure of one install within an incremental rollout step bankrupted the company. I’d guard against this with runtime checks that the software version (e.g. git sha) matches, and also add fault injection into tests that invoke the software rollout infrastructure.
  • kccqzy 3y ago
    No continuous deployment system worth its salt would allow configuration and code to be out of sync. They had a configuration change to turn on a flag that used to enable Power Peg but now enabled something else, plus a code change to reinterpret that flag differently.
    • chii 3y ago
      the situation is caused by a confluence of multiple issues. The biggest red-flag is that they chose to repurpose a flag! Why? Is it really difficult to add a new flag for a new feature? Even if the technician was careful not to let prod be out of sync, it is possible that the deployment isn't instantaneous, and that the old code could've ran when the repurposed flag was turned on.
      • atomicnumber3 3y ago
        Some of the trickiness that comes from HFT and adjacent things like this is you can be working on tremendously powerful hardware but still be miserly about bits because every extra byte you have to cram into a packet is extra latency. The HFT firm I worked for would "re-use flags" in the sense that each packet literally had an 8-bit section called "flags" (and further down in the message, another 8 bit section called flags2 because of course) and each bit in there was a Boolean that could be on or off - a flag. So we weren't reusing flags as much as we were re-allocating what a high bit at that index in flags meant. We were very conscious of this kind of error though and we managed them like Scrooge counting his farthings.
      • 3y ago
• reedf1 3y ago
  Gotta imagine the sinking feeling that guy felt.
• thorum 3y ago
  Honestly seems like the market itself should have safeguards against this kind of thing.
  • SoftTalker 3y ago
    The safeguard is "you go bankrupt if you fuck up" Imagine there was some way for a trading company to execute billions of dollars of trades and they say "ooops, sorry, that was all a mistake" can you not see how that would be abused? Now, the story also says that within a minute of the market opening, the experienced traders knew something was wrong. Do they bear any culpability for jumping on those trades, making their money off of something they knew couldn't be intentional?
    • bluelightning2k 3y ago
      A safeguard doesn't have to be a revert. A safeguard could be rate limiting. And or the ability to disable an API key or whatever the equivalent is. This actually seems pretty reasonable to me.
    • harerazer 3y ago
      This isn’t really correct. Typically exchanges have safety parameters which market makers can set according to how they wish to trade, and if you exceed those your orders will no longer be accepted and existing orders may also be pulled. Obviously there are false positives occasionally and there is typically communication between the exchange and the market maker to ensure those don’t reoccur.
  • alpark3 3y ago
    It's interesting to note that exchanges are adding "obvious error" rules that can slash trades under certain circumstances.
• realreality 3y ago
  [flagged]
  • eddtests 3y ago
    And remove easy/quick liquidity for the rest of the market? Edit: downvotes, any reason why? Or just HFT == Bad?
    • deleted 3y ago
      [deleted]
    • realreality 3y ago
      “The market” shouldn’t even exist.
      • TheAlchemist 3y ago
        And how would you invest in companies ?
      • eddtests 3y ago
        What should replace it?
  • tacker2000 3y ago
    How is high speed trading any more dubious than long term holding, or shorting, etc?
    • realreality 3y ago
      It’s all dubious.
  • alphanullmeric 3y ago
    good to know that you don’t consent to what other people do with their money.
  • dexwiz 3y ago
• tomp 3y ago
  Ah, Knight Capital. The warning story for every quant trader / engineer. This is what people don't realize when they say HFT (high frequency trading) is risk-free, leeching off people, etc. You make a million every day with very little volatility (the traditional way of quantifying "risk" in finance) but one little mistake, and you're gone. The technical term is "picking up pennies in front of a steamroller (train)". Selling options is also like that.
  • loa_in_ 3y ago
    I don't see how anything here goes against it leeching off people
    • Nevermark 3y ago
      Leeching implies someone has found a way to skim value from you without providing value. Someone taking on loads of risk to carry out your commands efficiently is providing value. You can argue whether they are doing so competently or not, or whether they are pricing optimally or not, but they are not just ”takers” or “leeches”.
      • UncleMeat 3y ago
        How are they taking on loads of risk? Risk has a particular meaning in investment and "well, a bug can blow up my company" isn't part of that meaning. Simply creating risky (in the colloquial meaning) things is not itself a reason to deserve money.
        • siftrics 3y ago
          When you make markets you are literally paid the spread to assume the risk of holding the position.
          • piyh 3y ago
            Can you break this down a bit more? I've heard about making markets in relation to FTX but didn't really get the full picture.
• hyperhopper 3y ago
  Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.
  • notnmeyer 3y ago
    this is what stood out to me reading the story. i wonder if there was a reason why they opted for this, however half-baked. it reads less to me like a case for devops as it does a case for better practices at every stage of development. how arrogant or willfully ignorant do you have to be to operate like this considering what’s at stake?
    • SoftTalker 3y ago
      They probably already had a bitfield of feature flags, maybe it was a 16-bit integer and full, and someone notices "hey this one is old, we can reuse it and not have to change the datatype"
      • notnmeyer 3y ago
        ah, yeah—hadn’t considered that!
  • Neil44 3y ago
    I can only think that it was some kind of fixed binary blob of 1/0 flags where all the positions had been used umpteen times over the years and nobody wanted to mess with the system to replace it with something better.
  • amluto 3y ago
    I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.
    • distortionfield 3y ago
• lopkeny12ko 3y ago
  I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, and the impact happens even faster. The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way.
  • ledauphin 3y ago
    The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."
    • SoftTalker 3y ago
      Or at least not by a developer who has made that sort of mistake in the past. I don't know what software engineering programs teach these days, but in the 1980s there was very little inclusion of case studies of things that went wrong. This was unlike the courses in the business school (my undergrad was CS major + business minor) nor I would presume what real engineering disciplines teach. My first exposure to a fuckup in production was a fuckup in production on my first job.
    • lopkeny12ko 3y ago
      I doubt any manager or VP cares or knows enough about the technical details of the code to dictate the name that should be used for a feature flag, of all things.
    • andersa 3y ago
      I wonder if this code was written in c++ or similar, the flags were actually a bitfield, and they repurposed it because they ran out of bits. Need a space here? Oh, let's throw out this junk nobody used in 8 years and there we go...
• gumballindie 3y ago
  > Had Knight implemented an automated deployment system – complete with configuration, deployment and test automation – the error that cause the Knightmare would have been avoided. Would it have been avoided though? Configuration, deployment and test automation mean nothing if they don't do what they are supposed to do. Regardless of how many tests you have, if you don't test for the right stuff it's all useless.
  • bluelightning2k 3y ago
    Yes. It would have. The specific part is configuration as code. So the config change (flag activation) and code change (flag calling) would have been synchronized. And there wouldn't have been one server of 8 with a different build for a meaningful time and also if it did fail to deploy on that one server it would have been obvious.
    • gumballindie 3y ago
      That's based on the assumption that someone would have thought about testing that particular flag for that particular scenario. In my view this would only have been caught by a deployment to an identical copy of production, with running, simulated transactions, and high level funtional testing. Testing for each individual config value and scenario of where it may be used is playing whack a mole. Basically, I'd make a clone of prod, simulate everything that happends externaly (APIs, etc) and observe transaction KPIs and other high level business indicators. Testing for tech is insuring that the tech works, and sometimes that means testing that it's broken.
• 40yearoldman 3y ago
  lol. No. Deployments were not the issue. At any given time an automated deployment system could have had a mistake introduced that resulted in bad code being sent to the system. It does not matter if it was old or new code. Any code could have had this bug. What the issue was, and it’s one that I see often. Firstly no vision into the system. Not even a dash board showing the softwares running version. How often i see people ship software without a banner posting its version and or an endpoint that simply reports the version. Secondly no god damn kill switch. You are working with money!! Shutting down has to be an option.
  • 40yearoldman 3y ago
    Oh god. I just realized this is a PM. A plight on software engineering. People who play technical, and “take the requirements from the customer to the engineer”. What’s worse is when they play engineer too.
  • INTPenis 3y ago
    I mean it makes no sense, without even reading the article, just by working in IT I can tell you that if you're one deployment away from being bankrupt then you're either doing it wrong, or in the wrong business.
  • deleted 3y ago
    [deleted]
• nickdothutton 3y ago
  “The code that that was updated repurposed an old flag…” Was as far as I needed to read. Never do this.
• valdiorn 3y ago
  Literally everyone in quant finance knows about knight capital. It even has its own phrase; "pulling a knight capital" (meaning; cutting corners on mission critical systems, even ones that can bankrupt the company in an instant, and experiencing the consequences)
  • shric 3y ago
    Indeed, it's used in onboarding material at my employer.
    • mxz3000 3y ago
      Yeap, it's used as a case study for us as to the worst case scenario in trading incidents. Definitely humbling.
    • atomicnumber3 3y ago
      Same, we knew of it as the Knightmare too.
• daft_pink 3y ago
  I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.
  • hgomersall 3y ago
    I'm so glad I'm not wasting my life working in finance.
    • envsubst 3y ago
      [flagged]
      • goldinfra 3y ago
        Most jobs do in fact contribute to the well being of humanity, however little. It's few jobs, like most in financial trading, that actively reduce the well being of humanity. Never will you meet a more self-deluded and pathetic set of humans. Desperate money addicts that often become other kinds of addicts. Whole thing should be abolished. Source: I worked in finance when I was young and dumb.
        • yieldcrv 3y ago
          I, for one, am so glad to “own” the “compose” button at a democracy destabilizing adtech-conglomerate
        • meiraleal 3y ago
          > Most jobs do in fact contribute to the well being of humanity, however little. No, they don't. A lot of jobs hold os back, actually. Salespeople selling things people don't wanna buy, finance and tech bros vampirizing third world countries without the safeguards that western countries have on their capital markets, etc.
      • dang 3y ago
        Hey, can you please review the site guidelines and stick to them when posting here? We have to ban accounts that won't. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
• pavas 3y ago
  My team's systems play a critical role for several $100M of sales per day, such that if our systems go down for long enough, these sales will be lost. Long enough means at least several hours and in this time frame we can get things back to a good state, often without much external impact. We too have manual processes in place, but for any manual process we document the rollback steps (before starting) and monitor the deployment. We also separate deployment of code with deployment of features (which is done gradually behind feature flags). We insist that any new features (or modification of code) requires a new feature flag; while this is painful and slow, it has helped us avoid risky situations and panic and alleviated our ops and on-call burden considerably. For something to go horribly wrong, it would have to fail many "filters" of defects: 1. code review--accidentally introducing a behavioral change without a feature flag (this can happen, e.g. updating dependencies), 2. manual and devo testing (which is hit or miss), 3. something in our deployment fails (luckily this is mostly automated, though as with all distributed systems there are edge cases), 4. Rollback fails or is done incorrectly 5. Missing monitoring to alert us that issue still hasn't been fixed. 5. Fail to escalate the issue in time to higher-levels. 6. Enough time passes that we miss out on ability to meet our SLA, etc. For any riskier manual changes we can also require two people to make the change (one points out what's being changed over a video call, the other verifies). If you're dealing with a system where your SLA is in minutes, and changes are irreversible, you need to know how to practically monitor and rollback within minutes, and if you're doing something new and manually, you need to quadruple check everything and have someone else watching you make the change, or its only a matter of time before enough things go wrong in a row and you can't fix it. It doesn't matter how good or smart you are, mistakes will always happen when people have to manually make or initiate a change, and that chance of making mistakes needs to be built into your change management process.
  • coldtea 3y ago
    >My team's systems play a critical role for several $100M of sales per day, such that if our systems go down for long enough, these sales will be lost. Would they? Or would they just happen later? In a lot of cases in regular commerce, or even B2B, the same sales can often be attempted again by the client for a little later, it's not "now or never". As a user I have retried things I wanted to buy when a vendor was down (usually because of a new announcement and big demand breaking their servers) or when my bank had some maintainance issue, and so on.
• gumby 3y ago
  > (why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point). Actually it's a big part of the point: they have a system that works with dead code in it. If you remove that dead code perhaps it unwittingly breaks something else. That kinds of chesterson's fence is a good practice.
  • rkuykendall-com 3y ago
    Leaving dead code in is not good practice?? I would love more explanation here because that sounds like crazy talk to me.
    • gumby 3y ago
      You'll have to ask the author of the article.
      • fphhotchips 3y ago
        Your original comment is somewhat unclear. Are you advocating for leaving old code in because the system works and it's more stable that way, or taking it out to force the necessary refactoring steps and understanding that will bring?
        • gumby 3y ago
          I'm sorry I wasn't clear: I re-read my comment and couldn't think of a decent edit. It was the author whom I was quoting as saying "why would someone have old code lying around." It seems obvious why that's a good idea and it seems commenters in this thread (including you) agree with me and not the author. Sorry again if I was unclear.
    • deleted 3y ago
      [deleted]
    • lionkor 3y ago
      It may not be obvious that it's dead code - in a lot of popular interpreted languages, it's impossible to tell if a given function can be called or not
• rvz 3y ago
  But ChatGPT would have fixed the issue faster in 45 mins than a human would. /s A high risk situation like this would make the idea of using LLMs for this as not an option; before someone puts out a 'use-case' for a LLM to fix this issue. I'm sorry to preempt the thought of this in advance, but it would not.
  • uxp8u61q 3y ago
    Who are you replying to? Nobody but you talked about chatbots in this thread. Are you talking to yourself?
    • bsagdiyev 3y ago
      No they're preempting someone coming along and claiming this. Haven't seen it in the replies yet but there's typically one (or a lot in some cases) person(s) claiming ChatGPT will bring Jesus back from the dead sort of thing.
  • cwkoss 3y ago
    That's an odd nonsequitor strawman you constructed to knock down. Did someone suggest LLMs as the solution or are you just asserting superiority over an imaginary guy?
    • rvz 3y ago
      As expected it seems many here, even you couldn't figure out what the '/s' means even as I preempted it in advance before anyone comes and tries to claim it anyway. So even putting the '/s' to denote sarcastic intent doesn't work on HN. Can't even take a joke here.
• dkarl 3y ago
  > why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point It's not the worst mistake in the story, but it's not "not the point." A proactive approach to pruning dead functionality would have resulted in a less complex, better-understood piece of software with less potential to go haywire. Driving relentlessly forward without doing this kind of maintenance work is a risk, calculated or otherwise.
• foota 3y ago
  The real issue here (sorry for true Scotsman-ing) is that they were using an untested combination of configuration and binary release. Configuration and binaries can be rolled out in lockstep, preventing this class of issues. Of course there were other mistakes here etc., but the issue wouldn't have been possible if this weren't the case.
• xyst 3y ago
  Having worked in some Fortune 500 financial firms and low rent “fintech” upstarts, I am not surprised this happened. Decades of bandaid fixes, years of rotating out different consultants/contractors, and software rot. Plus years of emphasizing mid level management over software quality. As other have mentioned, I don’t think “automation of deployment” would have prevented this company’s inevitable downfall. If it wasn’t this one incident in 2014, then it would have been another incident later on.
  • jonplackett 3y ago
    The thing I was surprised about is that they survived! After all that they still got a $400 million cash bailout!
    • m_0x 3y ago
      It wasn't a bailout, it was investment money. Bailout could imply government throwing a lifeguard
      • kaashif 3y ago
        > It wasn't a bailout, it was investment money. Almost all bailouts are investments, whether it's a government or private bailout. The investments are questionable sometimes, but they're still investments.
        • leoqa 3y ago
          The nuance is a) what happens to existing equity stakeholders and b) does the bailout have to be repaid. If the answer is nothing and no, then it’s a bailout philosophically. If the existing investors get diluted then they’re in part paying for the new capital injection.
    • deleted 3y ago
      [deleted]
    • ben-gy 3y ago
• m3kw9 3y ago
  Someone missed a blind spot
• codegeek 3y ago
  I refuse to believe that failed deployment can bring a company down. That is just a symptom. The root cause has to be a whole big collection of decisions and processes/systems built over years.
• codeulike 3y ago
  When I got to the memorable words "Power Peg" I remembered I'd heard all about this before.
• hinkley 3y ago
  > $400M in assets to bankrupt Was this Knight Capital? > Knight Capital Group Yep. Practically the canonical case study in deployment errors.
• jokoon 3y ago
  Oh no Anyways
• motoboi 3y ago
  Changes we make to software and hardware infrastructure are essentially hypotheses. They're backed by evidence suggesting that these modifications will achieve our intended objectives. What's crucial is to assess how accurately your hypothesis reflects the reality once it's been implemented. Above all, it's important to establish an instance that would definitively disprove your hypothesis - an event that wouldn't occur if your hypothesis holds true. Harnessing this viewpoint can help you sidestep a multitude of issues.
• dang 3y ago
  Related: Knightmare: A DevOps Cautionary Tale (2014) - https://news.ycombinator.com/item?id=22250847 https://news.ycombinator.com/item?id=22250847 - Feb 2020 (33 comments) Knightmare: A DevOps Cautionary Tale (2014) - https://news.ycombinator.com/item?id=8994701 https://news.ycombinator.com/item?id=8994701 - Feb 2015 (85 comments) Knightmare: A DevOps Cautionary Tale - https://news.ycombinator.com/item?id=7652036 https://news.ycombinator.com/item?id=7652036 - April 2014 (60 comments) Also: The $440M software error at Knight Capital (2019) - https://news.ycombinator.com/item?id=31239033 https://news.ycombinator.com/item?id=31239033 - May 2022 (172 comments) Bugs in trading software cost Knight Capital $440M - https://news.ycombinator.com/item?id=4329495 https://news.ycombinator.com/item?id=4329495 - Aug 2012 (1 comment) Knight Capital Says Trading Glitch Cost It $440 Million - https://news.ycombinator.com/item?id=4329101 https://news.ycombinator.com/item?id=4329101 - Aug 2012 (90 comments) Others?
  • SoftTalker 3y ago
    Early theory about the cause (incorrect, as it turns out): Nanex ~ 03-Aug-2012 ~ The Knightmare Explaned - https://news.ycombinator.com/item?id=4337359 https://news.ycombinator.com/item?id=4337359 (no comments)
• 0xFEE1DEAD 3y ago
  I don't exactly understand what this has to do with continuous delivery, but maybe I just don't know enough about this topic. Wouldn't it have been best to set up a 'shadow infrastructure' and route every trade into it for several weeks/months to verify the correctness of the system?
• dilyevsky 3y ago
  Their issue was neglecting an automated SCRAM system that would halt all the trading or any alerting with manual intervention. The article touches on that. There was no excuse why the system wasn’t halted by 9:32 which would’ve avoided most of the kerfuffle
• civilized 3y ago
  I see a lot of criticism of the deployment, but why did the developers "repurpose an old flag" that activates 8 years dead code that you haven't deleted and that has completely unknown current functionality? That seems like the strangest decision made in this debacle.
  • leaflets2 3y ago
    To save time, I guess. They deleted the inactive code, so, why not, they thought. But then they forgot to deploy that change (to one server). Bugs and configuration errors will happen from time to time, and might look silly in retrospect. But the real problem was, I think, that there was no kill switch (managers and tech leads should have decided to add long ago)
    • jundl77 3y ago
      I mean.. beyond that, that flag was used by a system from 8 years ago? Perhaps they simply didn't know they were re-using it.
• markus_zhang 3y ago
  Sometimes I think whether these events are more sinister than they appear to be. But then I heard that another MM is using Access applications to make markets for options and I think it's just incompetent.
• skizm 3y ago
  I feel like the first thing I would build into any automated trading system is a kill switch? then every single diff or pull request I add would have some sort of automated testing to ensure the kill switch still works. Also I'd manually flip it on/off once a day to make sure it works for real. That seems like the single most important thing to build and make sure works. Or is the system too complex for something like this and I don't understand the domain well enough?
  • slavboj 3y ago
    Depending on what you're doing, a straight "pull the plug and stop trading" could leave you with eg unhedged positions that blow through your risk limits. But when your ability to actually execute those trades sensibly is broken regardless, yeah, you're still going to want to hit that button.
  • atomicnumber3 3y ago
    Most systems do. At pastjob we had a few different levels: - halt - just stop trading - exit-only - only exit positions (but do so according to our alphas, no hurry) - flatten - exit in a hurry but obey certain limits (often if liquidity was thin we would just "journal" the shares - move them to a long-term-hold (meaning more than the current day) account to exit in the opening auction the next day - market exit - get the fuck out, now, no matter what the cost. I never saw us use that last one.
• brundolf 3y ago
  Much as I enjoy articles that reinforce my existing beliefs, high-frequency trading is a pretty extreme example when it comes to how how badly things can go in a short time
• siliconc0w 3y ago
  While nice automated deployment is the wrong lesson here, it's really not anticipating backwards incompatibility and poor altering and incident training. Flags should never be reused and should be retired after they're no longer useful.
  • ChrisMarshallNY 3y ago
    > Flags should never be reused and should be retired after they're no longer useful. That's such a "no-brainer," that I don't think it's even written down, anywhere. When I read that, I was like, "Whut?" In the Days of Yore, when we hammered programs directly into the iron as Machine Code, we would do stuff like that, but I can't even imagine doing that with any halfway modern language. They don't say, but it's probably C++. I know that's popular for HFT.
    • SoftTalker 3y ago
      Possibly Fortran.
• quickthrower2 3y ago
  Hold on. Are we blaming the plane crash on the pilot here? It seems there is so much other stuff wrong with this company first that such a deployment would tank it. No kill switch. Literally needs to be a power switch and a trader who runs to the room and flips it. Ridiculously small amount of cash for the trading volume, and no way to borrow more to stay in business (but that borrowing requiring manual intervention no accessible to the trading system). Obviously the decision to leave that code in there, and for there to be config setting to bring it back. Then the devops stuff - rollback plans, approvals, pairing on deployments, etc.
• TheAlchemist 3y ago
  Wild west times ! It's worth noting, that things changed a lot in trading systems since then. When I started working in this domain (2009), it was pretty crazy how unreliable those systems were, on all sides - banks, brokers, exchanges. Frequently you needed to make sure over the phone, what quantities got executed etc. I remember when the Italian exchange was rolling out their systems, at some point we did "tests" on a mix of production and UAT - if my memory is correct, we were just changing IPs to which to connect for order passing to test for the upcoming release, after the market closed. We couldn't just test in their UAT environment, since it was so bugged and half down most of the time. And let's not even talk about Excel spreadsheets with some VBA code that would make chatGPT swear, that were pricing instruments with volumes traded with a lot of zeros. It's very different nowadays, in part thanks to stories like this one. Most things are automated, and there is much less cowboy's attitude. There are mandatory kill switches, a lot of layers of risk / trading activity monitorings (on your side, on exchange side), and really a lot of hard learned lessons incorporated into the systems. That's also part of the reason why people sometime tend to be naive about how hard it is to build a good trading system - the strategies are sometimes now really smart - it's mostly about how to avoid getting killed by something that's outside of usual conditions.
• danielvaughn 3y ago
  I worked in fintech for a few years. I'll never again work on software that's responsible for trading, you could offer $1M/year and I wouldn't take it. By far the most stress I've ever experienced at a job.
  • CraigRo 3y ago
    We used to tell each other what stall we were going to. It is fun, like playing video games, but also super stressful.
• lolinder 3y ago
  > why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get resurrected, no rogue server. Maybe Knight Capital wasn't using version control and held onto this code "just in case", but I've seen this same resistance to deleting code in programmers working in repos that are completely under VCS, and it's flabbergasting. If you need it again, you can always bring it back from version control. If you need it again but forget it's there, you'd do the same with the dead code path. Leaving it in the source tree is pure liability. EDIT: Kevlin Henney gave an excellent talk at GOTO about software reliability and he touches on this, using Knight Capital as the example—he actually cites this very blog post [0]. The whole talk is excellent, but I've linked the three minutes where he talks about Knight Capital. > The problem is there is no code that is truly dead. It turns out all you need to do is make a small assumption, a change of an assumption and then suddenly it's no longer dead, it's zombie code. It has come back to life and the zombie apocalypse costs money. [0] https://youtu.be/IiGXq3yY70o?si=hZ9HB2dlfj0vHvNK&t=463 https://youtu.be/IiGXq3yY70o?si=hZ9HB2dlfj0vHvNK&t=463
  • datadrivenangel 3y ago
    Maybe they had engineering leadership mandates where creating new feature flags was more paperwork? I've seen that happen, where 'new' features took a lot of justification, but 'bugfixes' were free.
  • SamuelAdams 3y ago
    Version control isn’t bulletproof either. All your code history is one git rebase away from being abolished forever. I hope most orgs have processes around their main branches so this does not occur, but I’ve also been in smaller orgs and accidentally screwed up prod database tables, so the accidental git rebase isn’t impossible to consider…
    • chii 3y ago
• roughly 3y ago
  This is the Ur “devops fuckup” tale - I’ve told this to junior engineers who’ve bodged a deploy to make them feel better. I’ve been in this field for 20 years, and I can’t imagine I’ll ever have a day as bad as the engineers who got bit by this fuckup.
• cdchn 3y ago
  Good example of a blameless port mortem.
• piyh 3y ago
  This can be summarized as "terminally inadequate technical controls"
• _boffin_ 3y ago
  If you want to see how it looked like from the tick scale, take a look here: http://www.nanex.net/aqck2/3522.html http://www.nanex.net/aqck2/3522.html Ps. Anyone know of any other sites / places that does comparable level of research that's open to the public?
• KnuthIsGod 3y ago
  Not so simple. The company was then used as a building block to to create another entity, which was then acquired for over a billion dollars. "The company agreed to be acquired by Getco LLC in December 2012 after an August 2012 trading error lost $460 million. The merger was completed in July 2013, forming KCG Holdings. ...On April 20, 2017, KCG announced that it had agreed to be acquired by Virtu Financial for $20 per share in cash in a deal valued at approximately $1.4 billion."
• zsoltkacsandi 3y ago
  This has nothing to do with “DevOps”, and I am getting tired of this word. This mistake could have been prevented on multiple levels, and in my experience, deployments that involves major architectural changes rarely repeatable or can be fully automated.
• cratermoon 3y ago
  Not removing old code is akin to never throwing away food, even after it reaches its expiration date. Sure, you'll have it around next time you need it, but putting year-old yeast into your baguettes is, well, a recipe for disaster.
  • hollerith 3y ago
    Also on the tiny chance you really need the old code, you can dredge it up from your version-control system.
    • cratermoon 3y ago
      With git, yes. When I end up working with programmers who come from using other vcs, I find they are often the ones who don't delete code, or at best comment it out. I encourage them to trust git. It takes effort (barring a system failure out of git's control) to lose code. It can take some digging to find code in the git history, but it's there. Even if you run `git gc`, the objects are still there in the repo, by default. Even in extremis, if the central repo is gone, whoever has the most recent checkout still has history. With other, older VCS, I have less confidence.
• stevage 3y ago
  >They had 48-hours to raise the capital necessary to cover their losses (which they managed to do with a $400 million investment from around a half-dozen investors). I'm very curious about this bit. How exactly do you raise $400m of "investment" to cover such a massive footgun, in 48 hours, when you haven't even had time to understand what happened or whether it would happen again? Why are people stumping up hundreds of millions of cash here?
• mariusmg 3y ago
  First problem was repurposing flags used in the past for different functionality. Just dont do that.
• jammycakes 3y ago
  This incident highlights a problem that is often overlooked in the debate about feature branches versus feature toggles. I've worked with both feature branches and feature toggles, and while long lived feature branches can be painful to work with what with all the conflicts, they do have the advantage that problems tend to be uncovered and resolved in development before they hit production. When feature toggles go wrong, on the other hand, they go wrong in production -- sometimes, as was the case here, with catastrophic results. I've always been nervous about the fact that feature toggles and trunk based development means merging code into main that you know for a fact to be buggy, immature, insufficiently tested and in some cases knowingly broken. If the feature toggles themselves are buggy and don't cleanly separate out your production code from your development code, you're asking for trouble. This particular case had an additional problem: they were repurposing an existing feature toggle for something else. That's just asking for trouble.
  • MiyamotoAkira 3y ago
    That's interesting. Whenever I have an issue with a flag it gets picked up on dev/test/uat environments (all gets tested, especially around the code behaving the same as before with the flag off). The code change never reaches production. And if for some reason the code under the flag is wrong, and it has reached production (something unexpected, unseen), undoing the change is whatever long it takes to switch the flag back (and the cache to update if you have a cache).
    • jammycakes 3y ago
      That's a good approach if you can cleanly separate out the old code from the new code, and if you can make sure that you've got all the old functionality behind the switch. Unfortunately this can be difficult at times. Feature toggles involving UI elements, third party services or legacy code can be difficult to test automatically, for example. Another risk is accidental exposure: if a feature toggle gets switched on prematurely for whatever reason, you'll end up with broken code in production. The cases where I've experienced problems with feature toggles have been where we thought we were swapping out all the functionality but it later turned out that due to some subtleties or nuances with the system that we weren't familiar with, we had overlooked something or other. Feature toggles sound like a less painful way of managing changes, but you really need to have a disciplined team, a well architected codebase, comprehensive test coverage and a solid switching infrastructure to avoid getting into trouble with them. My personal recommendation is to ask the question, "What would be the damage that would happen if this feature were switched on prematurely?" and if it's not a risk you're prepared to take, that's when to move to a separate branch.
• belter 3y ago
  Knightmare: A DevOps Cautionary Tale (2014) - https://news.ycombinator.com/item?id=8994701 https://news.ycombinator.com/item?id=8994701 Knightmare: A DevOps Cautionary Tale (2014) - https://news.ycombinator.com/item?id=22250847 https://news.ycombinator.com/item?id=22250847 Knightmare: A DevOps Cautionary Tale - https://news.ycombinator.com/item?id=7652036 https://news.ycombinator.com/item?id=7652036
• praptak 3y ago
  Focusing on deployments is too narrow. Deployment can be automatic but still have a botched config. In this context it's more useful to think in terms of production principles. The principle that was poorly followed was defence in depth. There was no line of defence after the deployment.
• alexpotato 3y ago
  Couple fun facts/stories: 1. I signed my offer letter to work at Knight 5 days before this happened (and I still went to work there) You can read more about that here: https://twitter.com/alexpotato/status/1501174282969305093 https://twitter.com/alexpotato/status/1501174282969305093 2. As I mentioned above, I went to work at Knight as a DevOps on a team that deal directly with the team mentioned in the blog post. There are lots of stories around this but I will share this one: Late 2012 is when Apple rolled out the "emergency weather notification" function. I was in the office and the notification went off on multiple people's phones. Knight was also experimenting with call notifications. So when the alert goes off, someone yells "God damn it! Not again!!" (thinking there was another big outage) 3. People outside of finance have no idea of the different types of outage that can happen due to all sorts of factors. I have a LOT of stories here: https://twitter.com/alexpotato/status/1215876962809339904 https://twitter.com/alexpotato/status/1215876962809339904 4. In finance in general, the amount of legacy code that behaves in weird ways or was written by someone 10 years ago who is no longer with the firm is ASTOUNDING. Coupled with the billions of combinations of regulations, internal controls, multiple countries and jurisdictions etc makes accounting for every single edge case impossible. To use an infosec term the "attack surface" of possible user actions that could lead to bugs is enormous. Typical case: - User says they want to see reports for a couple days worth of trading for all securities - User also says they want to see FULL history for one security - User never says they might want to see FULL history for ALL securities at the same time - This being HN, someone will say "you should have thought of that" - Sure, but then they pull only some of the history for a Ukranian bond that has a 182 (not 180 like most) day bond. This is the only example of this type of bond. Ever. Did you think of that? What should the system have done? - An oh, btw, this system was pushed out quickly due to regulatory pressure etc
  • lukeasrodgers 3y ago
    I would be interested to read these stories, but the twitter links only show a single tweet ending in the phrase "A thread." Perhaps this is a new feature of X whereby only logged-in users can see a tweet and its replies.
    • alexpotato 3y ago
• terencegareth47 3y ago
  [dead]
• bettyfrank470 3y ago
  [dead]
• azimmm 3y ago
  [dead]