15 ms·

How to lose $172,222 a second for 45 minutes

• yogo 13y ago
  I remember when Knight was in the news regarding this but never the technical details about what took place. It's scary stuff especially given the money on the line, and it makes a good case study for devops. I understand the temptation to re-use a field but normally I'm for using new values in those fields.
• shtylman 13y ago
  Hindsight is 20/20
  • MBCook 13y ago
    Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years, is obviously bad. Deploying in such a way that all your servers are not running the same codebase is obviously bad. Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad. Not having anyone monitor things closely enough, including the hundreds of warning emails they got before the market opened, is obviously bad. There is no hindsight necessary here. You could look at what they were doing and predict a catastrophe.
    • coldcode 13y ago
      I wonder how you can have code sit on a server unused and then 8 years later have it be called? What language was this written in?
      • yareally 13y ago
        Why would the language matter? Dormant code is the same in any language.
        • twistedpair 13y ago
          Poster was probably thinking of directories of interpreted code (i.e. Python, PHP, Ruby), compared to compiled binaries (C++, C, C#, Java), since many compiled languages have dynamic deployment mechanisms like OSGi's hotswapping of individual modules.
      • yogo 13y ago
        It was a flag that was re-purposed. If the code is never deleted and the flag's use is discontinued then the code never runs. However, it seems that the flag was reused years later causing the old code to be inadvertently reactivated. It has nothing to do with the language it was written in.
      • deleted 13y ago
        [deleted]
      • frankc 13y ago
        It's most likely a giant C++ server or set of servers. It's not really that surprising that old code is in there. One way releases are handled in these kinds of places is to roll out new features on some kind of external flag system, perhaps shared memory flags or some other mechanism. This way if something goes wrong with that one feature, you can disable it by flipping the switch, rather than having to back out the software, which would likely cause downtime during market hours, a major problem. What sometimes happens when you want to decommission features is you just turn the flags off rather than remove the code. There's an obvious allure to this as you have already tested the on/off functionality of the switch when you did the original roll-out so you can avoid having to test whether you have removed the code correctly. It sounds like in this case they removed the code and repurposed the switch that disabled said code (it may really be a shared memory system and they are running out of flags), but they fucked it up. The old code was still there on some servers and the switch was turned back in intent to enable the new feature it was re-purposed for, re-enabling this old code.
    • thrownaway2424 13y ago
      I'm not sure I agree with "Deploying in such a way that all your servers are not running the same codebase is obviously bad." I have a lot of experience in large scale systems (although this incident with 8 machines does not qualify) and I would say there is _always_ a period of transition where versions X and Y are online in production simultaneously. How can it be otherwise? You'd need scheduled downtime to do it any other way. I think the main problem here is nobody at this company pushed back on this stupid development plan of reusing a flag for a different purpose. There's no excuse for that (or maybe there is, they had run out of fields in some fixed-width message format or something dumb like that). Also apparently the use of the flag was not tied hermetically to the binary in production; when they rolled back the binary the flag was still there but it meant something different to the old software. The correct way to roll this type of change out is for the new input (the "flag" in this case) to be totally inert for the old version of the software, and for the new version to have a config file or command line argument that disables it. So _first_ you start sending this new feature in the input, which is meaningless and ignored by the existing software, and then you roll out the new software to maybe 1% of your fleet, and see if it works. Then roll it out to maybe 10% and leave it that way for a week. Insist that your developers have created a a way to cross-check the correctness of the feature in the 10% test fleet (structured logging etc). If it looks good roll it to 100%. You now have three ways to disable it: turn it off in the input stream, turn it off in the new software with the config or argument, or roll back the software. Doesn't look like these guys really knew what they were doing.
      • randomfool 13y ago
        "How can it be otherwise? You'd need scheduled downtime to do it any other way." Trading floor is only open a few hours every day, the functionality being rolled out required the markets to be open. Furthermore, since the changes were all for new functionality they rolled it out in stages days ahead of time (good move).
      • thaumasiotes 13y ago
        > How can it be otherwise? You'd need scheduled downtime to do it any other way. Roll out the code in advance, and have the production machines switch to it at a defined, synchronized time? I mean, imagine you only have one production machine. If you're willing to admit that you can have it switch from version X to version Y with no downtime, then synchronization is the only barrier to doing the same on n machines. Why would you need scheduled downtime?
• sirsar 13y ago
  I'm shocked they didn't have a killswitch or automated stop-loss of some kind. A script that says "We just lost $5M in a few minutes; maybe there's a problem." Or, a guy paid minimum wage to watch the balance, with a button on his desk. $172,222 is a lot of minimum-wage years.
  • MBCook 13y ago
    How many millions in orders do they normally process per minute? Since there were no procedures in place, would you like to be the guy who pulled the plug on the (let's guess) $100 million/minute processing system? Do you think you could get another job after that? What would the costs be for violating contracts? You could single handedly sink the company (which, in the end, this issue basically did). I don't blame the guy for not killing all operations. He never should have been put in that situation. Proper QA, regression testing, monitoring, running a shadow copy and verifying it's output, there are tons of things that could have prevented/mitigated this.
    • candydance 13y ago
      Have them in pairs or teams, war games style. Guy1 "Sir, we're losing money much faster than predicted, turn your key" Guy2 "It could just be market variance" Guy1 pulls out gun "Turn your key, sir"
      • kops 13y ago
        Despite the gun brandishing, that conversation looks too polite to have ever taken place on a trading floor :-)
    • paganel 13y ago
      > running a shadow copy Honest question, it's that really doable at this scale?
      • babuskov 13y ago
        It's doable. The real question is how much it would cost and does it make sense then.
• OSButler 13y ago
  The title reminds me of hosting clients, who would complain about losing thousands of dollars per minute when their $10/month website was experiencing downtime.
• meepmorp 13y ago
  Powder Keg is a distinctly un-reassuring name for finance related functionality.
  • marincounty 13y ago
    funny!
  • frankc 13y ago
    It is peg, not keg. Peg refers to an order where the limit price is automatically adjusted to some benchmark. For instance, you always want to be 1 penny away from the best bid. I don't know specifically what "power peg" is, though.
    • meepmorp 13y ago
      I seem to have misread, but not in a way that markedly distorts things.
      • ycombobreaker 13y ago
        "Powder Keg" is a very marked distortion of "Power Peg".
        • JonnieCache 13y ago
          In the context of this story though, it's very apposite. I did the same thing. Quite disappointed after I read it correctly.
        • meepmorp 13y ago
          Considering how it blew up the company, it's an apt misreading.
      • mooted1 13y ago
        Ha, did the same thing. Had to do a double take.
    • 13y ago
• fiatmoney 13y ago
  "The best part is the fine: $12m, despite the resulting audit also revealing that the system was systematically sending naked shorts." Cool - all you have to do to get away with financial crimes is create a system with no protections against breaking the law.
  • judk 13y ago
    This connects back to the 'autonomous corporation' being discussed on the bitcoin discussion on HN today.
    • deleted 13y ago
      [deleted]
    • malandrew 13y ago
      Related link for others and for posterity: https://news.ycombinator.com/item?id=6589067 https://news.ycombinator.com/item?id=6589067
  • notaddicted 13y ago
    Yes, these criminal masterminds found a way to swindle honest hard-working themselves out of 440 million dollars and "get away with" a cool extra -12 million.
    • mcphilip 13y ago
      They also amassed over $3 billion net short positions spread across 75 stocks during those 45 minutes causing significant losses to investors with stop loss positions triggered that would not have happened without Knight's erroneous trades. They didn't just harm themselves...
      • notaddicted 13y ago
        Yeah, actually now that I'm reading the SEC report it appears pretty seriously reckless, also their lack of capital thresholds.
• adambratt 13y ago
  The week after this we had a trader in our office who had a meeting at Knight on the morning it happened. He said he saw the whole dev team just power off and go home at 11am, followed quickly by the rest of the employees. At that point, there was nothing they could do. The craziest thing is that it went on for so long. No one caught it until their own traders so it come across Bloomberg and CNBC. They actually thought it was a rival HFT and tried to play against it. The only people that came out of this ahead were aggressive algos on the other side and a few smart individual traders. A lot of retail guys had stop losses blown through that normally would never have been hit. After trading was halted they set the cap at 20% loss for rolling back trades. So if you lost 19% of your position in that short period of craziness, tough luck.
  • deleted 13y ago
    [deleted]
  • ycombobreaker 13y ago
    > After trading was halted they set the cap at 20% loss for rolling back trades. FYI, NYSE rolled back transactions based on predetermined price rules. There were no discretionary rollbacks associated with Knight's "big day". > The only people that came out of this ahead were aggressive algos on the other side Don't forget the other market makers; they took the other side of those Knight trades and made the spread every time Knight lost the spread.
    • donpdonp 13y ago
      is there more information on what rollbacks did happen and what the price rules are?
      • jzwinck 13y ago
        Search for "clearly erroneous" and the name of your favorite exchange. The policies are often spelled out fairly clearly, though most (all) will include an exchange discretion clause.
  • jbrechtel 13y ago
• malbs 13y ago
  Just one of the risks of automation, and a good reminder why human monitoring is necessary. Having said that, we deployed a system that was mostly automated, with the human operator to oversee investments and if any out-of-the-ordinary transactions (based on experience) were taking place, to shut it down. She happily sat there approving the recommendations even though the recommendations were absolutely outside of anything we'd ever generated in the past, and bled accounts dry in one evening, so sometimes even with a human observing you're still boned.
  • fiatmoney 13y ago
    You should read the linked PDF - they had systems that were 100% dependent on human monitoring, that no one was checking, or where no one recognized anything unusual. If anything, their failures were due to massive lack of automation in deployment, testing, and monitoring.
    • malbs 13y ago
      Yeah ours was monitoring just rubber stamped it. Afterwards everyone remarked, everyone could tell they were bad just looking at what was in front of them, our theory was she was too busy watching the breaking bad final episode or something.
      • 4hthth4 13y ago
        It makes me think you shouldn't allow only a single person to monitor something like that...
        • drill_sarge 13y ago
          Yes, the more people monitoring the better. But in this situation someone has to be in charge and make a decision. And nobody wants to be that guy.
          • comrade_ogilvy 13y ago
• Narkov 13y ago
  Out of interest, what would have been the outcome for Knight if their positions had caused them to be winners? $12m fine, keep the spoils and "carry on" ?
  • vinceguidry 13y ago
    It's not really possible to fail upwards this way. It would be like forgetting how to play chess in the middle of a game and then winning. Anomalies are universally negative in high-stakes environments, or if they're positive, only engender modest improvements.
  • ycombobreaker 13y ago
    Knight's systems were sending small, aggressive orders on both sides of the market, so their net positions were relatively small compared to the total volume traded. They were hemmhoraging money very quickly because they were crossing the spread repeatedly.
  • tedunangst 13y ago
    As others have noted, not really possible. You can lose a lot of money very fast by repeatedly buying high and selling low. Everybody will trade with you, happily. The opposite, placing low buy orders and high sell orders, will simply result in nothing happening.
  • koliber 13y ago
    In a big long-running error like this it is extremely unlikely that a company would have come out on top. However, it happened fairly regularly that smaller trading errors would cause a couple (or tens of) thousand dollar win or loss for the client. If it was a loss, the client universally complains and gets comped by the company. If it was a win and it is found, the client keeps mum and the company does not raise the issue.
• dror 13y ago
  Is there any benefit to the market as a whole to have these high speed transactions trying to game the system? Seems like as a rule, they're likely to cause instability, and I have a hard time seeing any benefits in them.
  • tempestn 13y ago
    The standard answer to this is that they provide liquidity. Whether that benefit outweighs the drawbacks is a subject of debate.
    • nhebb 13y ago
      A non-economist wants to know: if liquidity is beneficial to our economy and liquidity is a function of time, how how does the time-benefit curve look as t approaches 0? I don't know if you can quantify the benefit and map this curve, but if you could I don't imagine it would scale to infinity as time approached zero.
      • gd1 13y ago
        Liquidity isn't a function of time, it is a function of relative time between the predators (arbitrageurs) and the prey (market makers). If the predators are much faster than the prey, liquidity will disappear since the market makers can't survive (their prices are too stale and they are getting taken advantage of). It has always been this way, even since Nathan Rothschild used carrier pigeons to get news of the Battle of Waterloo.
  • Houshalter 13y ago
    I don't know how much of a benefit there is, but I doubt it's a problem. If they make a mistake, worst case is they lose money and everyone else gains from it.
  • lmm 13y ago
    In a sense the markets should be unstable. A perfect market would be so sensitive that any new information, every order from a fundamentals trader, should shift the price. That's what these high speed transactions get you - more accurate prices, and faster. And as a result of that they can offer much narrower spreads than you'd get elsewhere. Benefits? Knight gave a bunch of other market participants a better price than they could get anywhere else, and no-one traded at a price they didn't agree to.
• drill_sarge 13y ago
  I still find just the fact scary that at this moment automated systems are shoving billions of fake money back and forth around the world.
  • phyalow 13y ago
    How is it fake?
    • drill_sarge 13y ago
      Or better call it virtual money. (but we all know theres also "fake" money made and destroyed in that system, too)
    • rosser 13y ago
      As opposed to all the "real" money in the world?
      • qznc 13y ago
        How is it "real"?
  • shin_lao 13y ago
    Unless you barter your groceries you are using "fake money" every day. http://en.wikipedia.org/wiki/Fiat_money http://en.wikipedia.org/wiki/Fiat_money
• manishsharan 13y ago
  Don't humans also make similar large scale mistakes? Merill Lynch's infamous London whale comes to mind. Also. I could be wrong but aren't most of derivatives a zero sum game: don't I have to lose money on my puts for you to make money on your calls ? Didn't so many people lose money on securities because they misunderstood their exposure ? The Knight computer error was spectacular and catastrophic but us humans have a longer track record of making catastrophic financial decisions in the market.
  • adambratt 13y ago
    Options are complicated. At their most basic level they are no different than a bet, so yes zero-sum. However when used in a spread or as a hedge or any other way to avoid risk or when sold against stock you own as an income generator, it's tough to call them zero-sum. Puts and Calls are confusing as they are both something you buy. It's not like a sports bet where you're betting on the team to win so the other side will lose if they win. You can buy a put and a call in the same stock and profit on both if your strike prices are aligned properly. The opposite side of buying a call is selling a call. Just like shorting a stock, when selling an option the most you can make is 100% and your loss potential is infinite. As an option seller, you're hoping the option expires worthless and out of the money so that you can keep the initial outlay. Most brokerages require a high level of clearance to allow you to sell naked puts and calls as it's generally a bad idea if you can't cover the potential upside.
    • Tyrant505 13y ago
      Yea I always used options as a form of protection, not the rest.
    • bradleyjg 13y ago
      For most products there is a legal framework that forbids buying an insurance contract unless you have an insurable interest. However, under the Commodity Futures Modernization Act of 2000, designed by Summers, Greenspan, Levitt, and Rainer, state insurance regulators are forbidden from regulating OTC derivatives as insurance products. They were already forbidden from regulating exchange traded derivatives (e.g. options and futures).
• pallandt 13y ago
  Wow, this could have been prevented at so many 'checkpoints' that it reads like an almost cautionary, fake anecdote rather than a real story.
• jpatokal 13y ago
  Just another reminder of how systems that you'd think are rock solid often aren't. In my previous life working with telcos, I once tried to teach a particularly huge customer how to use CVS how to manage configurations across a 10+ machine cluster of machines. They didn't see any value in it, so they stuck to their good old process of SSHing into each machine individually, "cp config.xml config.xml.20131022", and then editing the configs by hand. Didn't take too long until a typo in a chmod command took down the whole thing (= the node couldn't take down a network interface anymore, so failover stopped working), and they spent several weeks flying in people from all over the planet to debug it... and they still didn't learn their lesson!
  • fosk 13y ago
    Ha, probably no one ever got fired for manually SSHing into each machine. It seems like a joke, but it really isn't.
    • rev_null 13y ago
      They can't fire that person. He's the only one who knows the state of the servers.
    • falcolas 13y ago
      Considering that I've watched a new automated deployment process nuke an entire network due to a mistake in the config... Sometimes slow and plodding and potentially losing a node (as opposed to 100 nodes) is seen as preferable to the PTBs. Of course intelligent, incremental deployment works too, but let's not confuse those poor suits...
    • mathattack 13y ago
      Is that job security?
    • gngeal 13y ago
      What about SSHing into each machine not manually, but automatically with OKing all the changes before reloading the configuration? SSH is great at least in the sense that it's configuration-format-agnostic, mostly-OS-agnostic, client-side-automation-agnostic, and many other things like that.
• mischanix 13y ago
  Well, this makes me 1000x more scared of working in a DevOps role.
  • hedwall 13y ago
    How come? DevOps isn't a role (to begin with) and a lot of the practices documented in the text is the opposite of good DevOps practices.
    • twic 13y ago
      AOL. True devops is about building systems that make this kind of disaster impossible. Or at least very hard. However, like agile before it, despite the fact that it really means something purposeful and rigorous, the word "devops" has become widely abused to camouflage undisciplined, thoughtless, cowboy behaviour. A handy way of telling the difference is to ask yourself "what would Devops Borat do?"; if it's something Devops Borat would do, it's the false devops.
      • nasalgoat 13y ago
        I like to think of DevOps as one of those cheap, 3-in-1 printers that you buy thinking that you'll be saving money and desk space. Then you discover it does a mediocre job of each of those tasks as compared to a dedicated printer, scanner and fax machine. Sure, they'll take up more desk space, but you'll get higher quality results.
  • Sanddancer 13y ago
    Look at this as a handy guide to how not to deploy in one single page. Pretty much any of the things in bold, had they not been done, would have avoided this from happening. Reused flags, letting dead code sit around in production for years, lack of change management, etc, all are signs of something gone horribly wrong. Also, regardless of if the deploy went good or bad, discuss the aftermath with co-workers. I'd almost guarantee that some of these problems had crept up before, and being able to ask the question of "how can this never happen again" is important, because otherwise, those problems will be forgotten and stumbled over again, in a future, perhaps more critical incident.
• Houshalter 13y ago
  They fined them for losing money? What?
  • lmm 13y ago
    They fined them for (accidentally) selling shorts they couldn't cover.
  • joshAg 13y ago
    No, the SEC fined them for losing money stupidly. In order to have access to the market like they did, they had to follow certain laws that are enforced by the SEC. When they were losing all that money they weren't following those laws. It's like if you cause an accident while you're driving by breaking the law; you get a traffic citation (and the accompanying fine), even if your car is totaled as a result of the accident, because you did something illegal.
    • Houshalter 13y ago
      Right but they didn't do anything except offer trades. The thing about selling shorts they couldn't cover makes sense, but just for "acting stupidly" seems silly.
      • comrade_ogilvy 13y ago
        These are pros who are paid very well to have a clue. It is not silly if "acting stupidly" is described in writing, such that all parties adequately understand when the hammer is likely to come down. I am sure a lot of traders push the envelope and "drive 71 in a 65 mph zone". But it still not silly to give the guy driving 81 a ticket.
      • joshAg 13y ago
        sorry, i wasn't sure how detailed to be, but by 'acting stupidly' I meant unintentionally selling shorts they couldn't cover in a way that made the action illegal.
  • chalst 13y ago
• telephonetemp 13y ago
  I assumed they had redundant servers with consensus algorithms in place in finance but apparently they don't. Would it be impractical?
  • clearly 13y ago
    Yes- latency is a big issue for this type of trading system.
    • telephonetemp 13y ago
      Perhaps you could do consensus checking retrospectively? I.e., out of N supposedly identical servers a random one gets to make any given decision in real time but then a separate system goes back and compares all servers' results and stops their operation if there's divergence?
      • clearly 13y ago
        I guess, but it's more typical to do something like cap the total trading volume, position, risk limits etc. It's a more fundamental check on what you are doing.
  • marincounty 13y ago
    Ya, I just assumed they would have at least 3 levels of redundancy, or at least a person there who could pull 8 cords?
  • sgt101 13y ago
    Maybe the extra step in the workflow prohibits HFT?
• vincie 13y ago
  I would love to hear from an ex-Knight tech. Wouldn't be surprised if they wrote something along the lines of: "Management just wanted this thing in ASAP!", or perhaps "Tests weren't part of the kpi's". I may sound biased against non-techs, but I have seen this time and time again. Testing is a barrier to quick deployment, and "How much money are we losing while doing all that stoopid testing?".
  • SideburnsOfDoom 13y ago
    > Testing is a barrier to quick deployment I really feel bad for people who think like that. A process where tests and deployments are automated and repeatable are vital to quick, robust deployment. Quick deployment without tests just isn't going to work well.
• tantalor 13y ago
  That explains how the deprecated "Power Peg" model was activated, but why was that model so flawed?
  • michaelt 13y ago
    You may have heard people with things like backups and emergency generators saying "you have to test this stuff weekly, in case someone has broken it so it'll fail the moment you call on it." Software is the same. Knight had code that hadn't been run in 8 years. Sure, the code worked 8 years ago, but things have changed around it since then. As the problem code never ran, no-one noticed it getting broken, or had any reason to fix it if it broke in testing. Most likely the code worked fine 8 years ago, broke in the intervening 8 years, and hence was broken when activated.
    • Pitarou 13y ago
      If I understand this correctly, this isn't like having untested code around. It's more like leaving highly toxic medicine in the bathroom cabinet when you no longer need it, or leaving an electrically powered band saw plugged in when it's not in use.
  • Dylan16807 13y ago
    They roughly explained it. The counters were in a different state because of refactoring moving their updates around, so the Power Peg code had an incorrect view of reality.
    • rurounijones 13y ago
      There is a part of me that is strangely impressed that the code ran at all after 8 years of things changing around it. I would have expected it to just segfault or error out in some way.
  • deleted 13y ago
    [deleted]
• scrrr 13y ago
  High Frequency Trading seems so abstract. There's no value created, it seems. It's like something in between imperfect systems, scraping off the margin created by that imperfection. It's fascinating, and interesting from an algorithmic point of view (like a computer game), but at the same time I don't feel sympathy for this company going out of business.
  • marijn 13y ago
    Looking at systems by considering whether they 'create value' in some generalized utilitarian sense is unproductive. Such systems survive by being able to extract energy somehow, in this case by exploiting properties of the stock trading system. I guess you could say that they create a lot of value—for the people doing it. Very few modern economic activities make sense in a broader perspective, they exist purely because they allow some energy to flow towards the people perpetuating them, on a more local level.
    • Peaker 13y ago
      I disagree completely. You seem to be confusing money/worth with value. If someone manages to redirect money towards themselves without creating a new economic resource, then no new value is made. Avoiding systems that don't create value would help us live in a world with more value in it. That's why considering this is very productive, and helps moral people avoid wasting precious resources on zero sum games.
      • marijn 13y ago
        What I'm trying to say is that systems don't exist because they are 'good', they exist because they survive. When trying to interpret the world, 'value' is a much less powerful explainer than survival.
        • Peaker 13y ago
          Who says he tried to interpret the world? He was giving his own value judgement about it. Also, I do think that knowing whether something actually creates value or not (is a positive-sum, negative-sum or zero-sum game) is definitely helpful in understanding large systems. i.e: It helps predicting whether a society will succeed or fail, by seeing how much of it is wasted on negative/zero sum games.
• mgav 13y ago
  Very interesting, though I was happy to see Knight Capital take the huge loss, since they were such complete scumbags who stole hundreds of millions of dollars by backing away from trades* during the dotcom boom and bust. *Backing away is when a market maker makes a firm offer to buy or sell shares, receives an order to execute that transaction (which they are ethically and legally obligated to do) and instead cancels the trade so they can trade those shares at a more favorable price (capturing enormous unethical profits in fast-moving markets while regulators did virtually nothing to enforce the rules in a meaningful way) Learn more: http://bit.ly/1ddUzWP http://bit.ly/1ddUzWP
  • simoncion 13y ago
    Would you kindly provide the un-minified version of your link? This isn't Twitter, and HN has its own method of shortening overlong links while also displaying the original link target.
    • Ricapar 13y ago
      Un-minified link: http://books.google.com/books?id=geCHWBx-e9EC&pg=PA126&lpg=PA126&dq=%22backing+away%22+market+makers&source=bl&ots=UeWYm3gjwc&sig=ZMul3CSEI-vr60iMbcH6GpERdV4&hl=en&sa=X&ei=m1NmUonyOZTs8ATdtoGoCA&ved=0CDkQ6AEwAzgK#v=onepage&q=%22backing%20away%22%20market%20makers&f=false http://books.google.com/books?id=geCHWBx-e9EC&pg=PA126&lpg=P...
      • mgav 13y ago
        Gracias Ricapar.
• zipfle 13y ago
  The original report is remarkably well-written. It's nice when you get someone with the domain knowledge to understand an issue and also the language skill to explain it clearly.
• at-fates-hands 13y ago
  >>>What kind of cowboy shop doesn’t even have monitoring to ensure a cluster is running a consistent software release!? I think you'd be surprised at what happens in large companies. I went through four, count em' four major releases with a company and each time the failure was on load balancing and not testing the capacity of the servers we had prior to release. Even after the second release was an unmitigated disaster, the CTO said we needed more time to do load testing and making sure the servers were configured to handle traffic spikes to the sites we were working on. It happened again, TWICE after he said we needed to do this. You would think something as basic as load testing would be at the top of the list of "to do's" for a major release, but it wasn't. It wasn't even close.
• protomyth 13y ago
  "During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review." That is just painful to read. How many times do we hear a company couldn't figure out how to migrate code properly? Do any software engineering programs teach proper code migration? Next time a manager questions money spent on integration or system testing, hand them a printout of this SEC document and explain how much the problem can cost.
  • HillRat 13y ago
    Look up the 1999 WorldCom outage due to a screwed-up load of Lucent's Jade platform upgrade. Fun times. Best I can say about that is that within a year WCOM execs had bigger problems than just pissing off CBOT....
    • protomyth 13y ago
      Oh my, that sounds like a serious Career Ending Event for someone.
• sitkack 13y ago
  Dead code takes down another system. A perfect storm of failures that they made themselves. My gut feeling is that most trading firms could suffer a similar loss. Having worked for a 3rd party accounting management firm that kept logs for smaller traders I really realized how borked the whole system is. 60s era pen and paper stuff moving at the speed of light. > Sadly, the primary cause was found to be a piece of software which had been retained from the previous launchers systems and which was not required during the flight of Ariane 5. http://www.vuw.ac.nz/staff/stephen_marshall/SE/Failures/SE_Ariane.html http://www.vuw.ac.nz/staff/stephen_marshall/SE/Failures/SE_A...
  • frankc 13y ago
    Actually, I would characterize this as the removal of dead code that brought down the system, which is basically the opposite.
    • sitkack 13y ago
      Dead code (latent functionality) in both systems was reactivated without proper testing or knowledge and both systems failed as result. We need to be very weary of latent functionality.
• avty 13y ago
  Someone made $172,222 a second for 45 minutes on the opposite side of these trades.