14 ms·

NYSE Tuesday opening mayhem traced to a staffer who left a backup system running

278 points by helsinkiandrew 4y ago

kube-system 4y ago
I spent an hour trying to figure out why my new stock purchase had disappeared from my account. I had an order placed for opening on Tuesday morning, and I guess I was affected by the trade cancellations. Which is totally weird, because they showed up in my account on Tuesday morning after opening.
halotrope 4y ago
That means some process is broken. Imagine reading “flight crashed because baggage handler place some suitcases wrongly in the cargo space”
helsinkiandrew 4y ago
https://archive.ph/UoMr9 https://archive.ph/UoMr9
bmitc 4y ago
Seems so weird to not have automated checks for something that seems to be described as "someone left the light on" and also not have the exchange automatically initiate itself. However, it still isn't that clear what the problem was. Were prices not "real" or correct? Stuff like this will happen more and more. We treat software driven systems rather recklessly.
- mrkeen 4y ago
  More automation -> more code -> more things to go wrong
  - bongobingo1 4y ago
    The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.
    - junon 4y ago
      I've not seen it put so eloquently.
      - dsr3 4y ago
        To err is human, to really foul things up requires a computer - William E. Vaughan https://quoteinvestigator.com/2010/12/07/foul-computer/ https://quoteinvestigator.com/2010/12/07/foul-computer/
        erik_seaberg 4y ago
        “A computer lets you make more mistakes faster than any other invention, with the possible exceptions of handguns and tequila.”—Mitch Ratcliffe
    - WJW 4y ago
      Some submarine operations are deliberately not automated, because if eg a sensor is broken or miscalibrated it could sink the entire boat if a computer very rapidly acts on the wrong information. Rather they do those operations with one person operating the valve/machine/device/etc, another watching and confirming readings, and the whole thing is on a constant audio link with a third person in an engineering room who watches the readings through a centralized system. It's clearly not optimized for efficient use of personnel, but the personnel complement will have been designed to provide sufficient people at all times and the cost of getting it wrong can be very large indeed.
      - TeMPOraL 4y ago
        This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general).
        dragonwriter 4y ago
        > This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general The canon explanation is that automation-in-charge was experimented with and went really badly, though periodically they try something approaching it again. https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(episode) https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(... (AI, human genetic engineering, and a number of other areas of technology are affected by variants of this issue in the Trek canon.)
  - themitigating 4y ago
    This employee left the backup system running. There's obviously some automation but what is the solution? Process changes that people have to remember or more systems to prevent the issue. So I don't get your statement related to this article
    - nix23 4y ago
      >There's obviously some automation but what is the solution? Shutdown and wake-up time in bios of server and switch ;)
    - deleted 4y ago
      [deleted]
    - mrkeen 4y ago
      I don't imply that there is a solution. We will simply create a second system (B) to monitor the first system (A). Now we have two systems to maintain. System B will not be capable of steering A by itself. So we still need to know how to diagnose and repair A, and we also need to know about B too. Maybe system B can talk to a Prometheus/Grafana stack (if it's up). And that can put alerts into Slack (which we ignore because there's always alerts in Slack). And after standup we can take turns looking at graphs with consternation. > Stuff like this will happen more and more. We treat software driven systems rather recklessly. That sentence is where I go when I hear the word 'automation'.
  - nix23 4y ago
    >More automation -> more code -> more things to go wrong More people -> more entropy -> much more things to go wrong
    - wongarsu 4y ago
      One person might be inattentive or drunk, but it's less likely that two people are. So you institue a two-person rule. And if that's not good enough, add a third person to double-check. Maybe a supervisor to observe the people doing all of the above, to catch any mistakes or negligent behavior. Also have them write down the steps they have taken, and have somebody else read through that to verify. Just keep adding people until you are satisfied with your odds (and hope you are not making it worse through second-order effects)
      - nix23 4y ago
        So two people are more reliable as a automated system you wanna say? That's totally wrong....
        reaperducer 4y ago
        Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. I did last year, and my company is in the process of de-automating certain processes that can endanger that company if they go wrong. There are many things in tech that are too important to automate. I'd even posit that the more experience you have in tech, the more you've seen how things go wrong, and the more you realize that automation is a tool for humans to use, not a replacement for humans doing a task.
        nix23 4y ago
        >Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. Much much much more due to human error......but hey maybe you are the worst programmer ever..but even then i would say your programs are more reliable then a human.
- baxtr 4y ago
  [flagged]
- qrybam 4y ago
  The auction is meant to find a stable price and can have some wild prices coming in, because no matching happens at the point of entry, the market will naturally find a level before trading commences. In this case, those wild prices were matching, resulting in crazy trades no-one would have expected. You'd be surprised how many manual processes there are in places like this. It's a combination of legacy systems / processes, and a general paranoia around automation going wrong. I wouldn't be surprised if they always have someone there to shepherd the system along.
  - benjaminwootton 4y ago
    When I worked on a trading platform, I spent many a happy Sunday night waiting for the Australian market to open and watch the first orders go through successfully. We had hundreds of jobs and upgrades happening over each weekend. It definetly needed an eye casting over it regardless of the automation.
    - xwolfi 4y ago
      I am working at one now and nothing has changed. Automations are so many we recruit an army of people just checking if they ran, while knowing how to replace them if they didnt (or, more accurately, who to call at night to fix it asap). And the AU orders going through is a good sign, but it's far from guaranteeing a free monday, as Japan, Korea or Shanghai can fuck it up, each in their own little ways. Hong Kong is the best, low regulatory crap, invested regulator, high volume low latency traffic everyday (relative to the region), I cant recall a time it broke. Once, someone fat fingered an excel import at close, and we lost our trading license for that entire country for 18 months. And we're not small. But the amount mismatched at settlement was super tiny. High attack surface, low holistic understanding (it works despite us, we honestly have no clue sometimes), heavy consequences on screwup.
- helsinkiandrew 4y ago
  Matt Levine at Bloomberg gives a good explanation: https://www.bloomberg.com/opinion/articles/2023-01-25/nyse-forgot-to-open-yesterday https://www.bloomberg.com/opinion/articles/2023-01-25/nyse-f... Basically at the market open all the requests to buy and sell get matched at the same "open" auction price, then (a second later) the orders get sent to the order book, where the price can go up and down based on size. Because the system didn't think there was an opening, there wasn't the opening auction, the prices went straight to the book and there were large swings in price.
  - usefulcat 4y ago
    https://archive.is/LovhH https://archive.is/LovhH
- tgtweak 4y ago
  I don't think it's that they "left it running" like you would leave a backup app running... they literally left the entire disaster-recovery site up and running and live. Cermak (referred to in the article as the "backup") is an entire datacenter, hosting a running copy of the exchange to be used in a failover scenario. You'd have to have more than 1 person involved to forget that DR is still active when completing these failover exercises and tests off-hours.
  - Kon-Peki 4y ago
    > Cermak (referred to in the article as the "backup") is an entire datacenter Well, they are a tenent at the Cermak data center. It’s a truly massive building with huge amounts of connectivity and colo opportunities. Probably also the only 100+ year old data center building on the US register of historic places, lol (it’s a former catalog printing facility, built to hold insanely heavy printing presses on 8 or 9 really tall floors, so it has no problems with densely-packed server racks)
    - tgtweak 4y ago
      Yeah my point is only that it's a full DR site, not a "backup" that was left running as the article pointed out (and as a lot of commenters are insinuating).
      - Kon-Peki 4y ago
        Ah, I see. Some quick internet searching shows that from time to time (rarely), NYSE operates the Cermak site as the primary site for at least part of their operations - for up to a week at a time. It seems to have multiple purposes - DR, customer software testing, etc.
        wallstprog 4y ago
        Well, you do that to make sure it works. At one of my previous jobs, we'd switch the primary and secondary datacenters every six months just to make sure we could.
h3daz 4y ago
I am short a couple dozen naked Feb 3 calls on a lot of affected tickers and almost had a heart attack when looking at one of them on my phone. Thankfully I was not in front of my computer at the time because I have no idea how my broker was managing my margin at open.
- qeternity 4y ago
  The LULD breakers saved everyone from more chaos here because effectively as soon as the market opened, all these symbols were halted.
WiSaGaN 4y ago
A lot of stocks have minute bar of wide range prices in the first few minutes of continuous trading. This seems like the incident in what caused Knight Capital fiasco, in which the system repeatedly buy on the ask and sell on the bid very fast, thus pushing the high price high and low price low. In the opening usually the market maker will be more weary of the risk they cannot hedge directly and thus will be less willing to take on positions, leading to wild swings. Still this report (and the previous statement) does not give enough detail on why a backup system misoperation resulted this. Also, critical large systems like exchange rarely have a single point failure. Usually there will be a sequence of issues along the event chains leading to this. Thus one "failed to properly shutdown" caused all this is a bit incredible. We will need more explanation.
- itronitron 4y ago
  Doesn't each trade require both a buyer and a seller, who buy and sell at an agreed on price? Presumably both parties would be satisfied with the trade so it isn't clear to me what all the fuss is about.
  - pcl 4y ago
    Market price trades are executed at whatever the current price is. Presumably it’s those trades that caused havoc.
  - gpderetta 4y ago
    I haven't really researched anything, but both parties thought they were bid/offering into an auction with time to cancel or amend their orders.
  - WiSaGaN 4y ago
    I don't know what happened in current case. In the Knight Capital case, KC clearly didn't intend to send those erroneous orders. And if those trades were not annuled, KC would not be able to settle those, since the trade loss were larger than the collateral KC put up.
    - 4y ago
hksoftware 4y ago
[flagged]
- ideamotor 4y ago
  You suspect a breach?
  - avree 4y ago
    He is, undoubtedly, a meme-stock conspiracy theorist. Only those steeped in the cult of AMC, GME, or BBBY say things like that.
    - deleted 4y ago
      [deleted]
- shapefrog 4y ago
  I am curious what your "independent research" has turned up on the subject.
- deleted 4y ago
  [deleted]
lordnacho 4y ago
Backup system connected to prod, that somehow reminds of the Knight Trading debacle. Someone there apparently connected some test code to their prod and they blew up the company in under an hour.
- laurencei 4y ago
  I mean - isnt that how all DR is essentially configured? You need it "somehow" connected to Prod depending on the failover, config, system etc. And in many of these complex systems DR can be on a subsystem etc - not an "all or nothing" approach?
afhammad 4y ago
It seems that this wasn't as routine as these things aught to be but rarely are.
- H8crilA 4y ago
  Yes it could have also been a test of potential escalation from Solomon Islands.
mellosouls 4y ago
These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.
- wrldos 4y ago
  This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.
  - iancmceachern 4y ago
    Exactly, management shouldn't allow this kind of situation to occur by designing the xomoanies processes such that there are checks and balances
  - thisarticle 4y ago
    The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.
    - deleted 4y ago
      [deleted]
    - lotsofspots 4y ago
      Oh, they take full responsibility, it always says so in the mails they send out. It's just that taking responsibility doesn't appear to actually result in anything happening.
      - parsimo2010 4y ago
        A cyncical take might be that they are saying that they take responsibility (credit) for reducing the monthly payroll expenses. They may also have overhired in the past, but what's in the past was already paid for. The savings next month is how they justify a large paycheck.
herpderperator 4y ago
If the trades are being cancelled, are they going to correct the chart data? Right now it looks very misleading on the daily[0], weekly, monthly, quarterly, yearly etc for large caps that trade quite steadily otherwise. I do understand that this would be a challenging effort as that data already flowed to and was stored by all the broker-dealers, but I think it should be done. [0] https://www.dropbox.com/s/6jdmgkdyei9xqz0/mcd.png?dl=0 https://www.dropbox.com/s/6jdmgkdyei9xqz0/mcd.png?dl=0
- evanpw 4y ago
  The same feed that publishes trades also publishes trade busts, so it's up to whoever's consuming it downstream to take care of.
- anonu 4y ago
  Technically yes, historical market data feed needs to be cleaned up. Which will be a nightmare for every single person who maintains one... Which is also why exchanges are very reluctant to mass cancel trades. The knock on effect goes beyond just market data feeds
lvl102 4y ago
Remember the flash crash of 2015? They let those trades actually STAND. Including options. This week’s open was nothing in comparison.
- detaro 4y ago
  That wasn't an exchange "malfunction" in the sense that the exchange did not do what it was supposed to, was it?
  - lvl102 4y ago
    Do you really think they will take responsibility for billions lost/made that day? No, that's a big liability. Anyone trading that day knew it was a big glitch at open. Some names, BLUE CHIPS, were down 40-50%! What shocked us all is that they actually allowed the trades to stand. What was different about this week was that the SEC actually tried to do their jobs for once and the exchange had to address it ie come up with bullshit excuse.
    - spywaregorilla 4y ago
      So... what was wrong? Why should those people not have been allowed to make and lose money?
corobo 4y ago
Oh we're still doing scapegoats? If your system can be hosed by a single person the system is at fault. Start with the scapegoat's manager.
hknmtt 4y ago
there is no way a person, a single person, an unauthorized person, can have access into such system/functionality like this. utter BS.
- jterrys 4y ago
  my 2c: In reality what probably happened is previous market day and post-trading data encountered some kind of error, which triggered a cascade of problems overnight that they were unable to properly rectify. This caused delays up until market open. They were unable to fully resolve the issue, and forced with either delaying opening the market (which is a HUGE no-no) or opening with wrong data as is, they chose wrong data. All in all a lot of people didn't get much sleep Monday. More than likely they implemented some changes or updates over the weekend that were not properly done, or they encountered some errors, and didn't have adequate controls/time to roll-back Monday night. They made the right calls too late and there was a controls process up the chain that seriously fucked up. These are the kinds of problems that get the CEO woken up in the middle of the night.
kmac_ 4y ago
Weak blame game.
pedro2 4y ago
Successes are management's, failures are individual's :)
- chiefalchemist 4y ago
  True. But when that happens, that's a textbook sign of lack of leadership.
chiefalchemist 4y ago
While we're on the subject, if your company uses technology in an capacity, read the book The Phoenix Project. https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/1942788290 https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine...
- dogleash 4y ago
  I wanted to like that book but it's a big Just-so story. tl;dr: The brave knight implemented devops and everyone lived happily ever after!
  - chiefalchemist 4y ago
    Yeah. But it made clear many things that are otherwise too often cloudy. And despite its age it's still relevant. Some people need a story.
irthomasthomas 4y ago
Is it normal to simply accept the word of an anonymous source for something so important? I genuinely don't know, anymore, but it doesn't seem like a good idea. I'd rather wait for a more thorough investigation. Especially when the story from these sources boils down to "Kevin was in charge of booting the NYSE App that morning, but he was late for work. He had a good excuse, though, he flaked! We'll have the chap straight up for lunch, no question". Edit: I also note that this piece is lacking the traditional "The NYSE did not respond to a request for comment".
- itsoktocry 4y ago
  >Is it normal to simply accept the word of an anonymous source for something so important? Anonymous means they aren't revealing the source, not that Bloomberg doesn't know who the sources is, or what they do.
  - irthomasthomas 4y ago
    I know that. I am referring to you and I, the reader accepting the word of the anonymous source. Combined with the fact that they apparently did not ask for a comment from the NYSE before publishing this. Or if they did, they neglected to mention it.
    - maronato 4y ago
      We aren’t accepting the word of the anonymous source. We’re accepting Bloomberg’s word that the source is reliable.
      - adolph 4y ago
        Bloomberg's word: https://news.ycombinator.com/item?id=19526348 https://news.ycombinator.com/item?id=19526348
      - bink 4y ago
        Some of us aren't "accepting" anything. We're just reading about a potential cause of an incident and speculating about how it could happen to us or could have been prevented. Just because we're reading this article and commenting here doesn't mean we just believe everything that we read. The post-mortem will come out soon enough and we'll read that and comment again.
nickdothutton 4y ago
If a single staffer SNAFU can send your exchange into chaos then you dun goofed at risk management and probably a whole lot of other management discipline.
h2odragon 4y ago
Tuesday news blackout; Thursday "it was all Jim's fault"... Right. Smells like horse shit. Most of what comes out of the profession of "journalism" does too, lately; but this smells strongly.
- yakubin 4y ago
  Why couldn’t have been Jim’s fault? The world is run by Jims.
  - h2odragon 4y ago
    As others have said, if Jim's fuckup can have such large consequences, then there should have been backstops for Jim who would have shared the blame. Nothing against Jim, it's just he's being used as a distraction to avoid talking about the real issues.
    - throwawaaarrgh 4y ago
      Manager: "I hope nobody asks why Jim was allowed to have so much power with no oversight or validation"
- epistemer 4y ago
  and all the trades are being reversed basically. The whole thing stinks.
  - tgtweak 4y ago
    Not all of them... only the most egregious.
anonu 4y ago
Tech can be so fragile. You do everything right, trade millions of shares everyday and handle billions of dollars. But you forget to run one script to shutdown a backup system and everything comes crashing down: your reputation in tatters, millions in costs to settle bad trades, barbarians at the gates.
lr1970 4y ago
From the article: > Meanwhile, market professionals and day traders are rattled and waiting for the exchange to elaborate on what it publicly called a “manual error” involving its “disaster recovery configuration”. Oh, I love it -- a disaster caused by "disaster recovery configuration" :-)
- gjvc 4y ago
  Oh, I love it -- a disaster caused by "disaster recovery configuration" :-) People install failover configurations to minimise time-to-repair or time-to-resume service (and some customers' contracts will demand this). This is at the expense of another layer of stuff to go wrong, and raising the possibility that it fails over when it shouldn't, causing brief but embarrassing outages. It's possible in some such situations that, on the balance of probabilities, introducing mechanisms like this cause more disruption over time than they were intended to protect against, and that this is more widespread than often considered. Still, their operational cost must be borne in order to satisfy the clause in the customers' contracts.
twawaaay 4y ago
Oh, yes. It is the staffer who made the mistake. What about people who designed it this way?
jrochkind1 4y ago
> That misled the exchange’s computers to treat the 9:30 a.m. opening bell as a continuation of trading, and so they skipped the day’s opening auctions that neatly set initial prices. I didn't even know about this process. I don't know much about trading, but it surprises me that there is a separate process for setting prices at the start of trading, and that if it's missed, chaotic prices result. Is this related to how stock markets aren't really ever open 24 hours? Do they need that reset to function in stable way?
- xyzelement 4y ago
  That's right. In short it's something like this: stocks trade on their primary exchanges during specific hours. For example 9:30 to 4 in the US. Part of it is legacy from when trading was done by actual humans being at the exchange physically to trade during those times and part of it (I would guess is still the case) is to allow plenty of non-trading hours for back-office jobs and settlement. So yes there's a special start of day process that runs at 9:30 that runs through all the orders on the books at that time and determines a price at which some optimal set of those orders can trade, trades them at that price, and also posts that price as the Open price for the day. The process is different during continuous trading since orders are one by one matched against the order book. Source: ran one of the world's largest equity platforms for 5 years.
  - ajoseps 4y ago
    isn't there another component to the NYSE auction where the DMM has some input into what the closing/opening price actually is?
    - khold_stare 4y ago
      Yes. In my reply to the first comment I mentioned setting the opening price is "more complicated". Every exchange has their own system for the opening auction which you buy into when you list with a particular exchange. Most exchanges have an algorithmic way of calculating the price. For NYSE, it's again more historical. A Designated Market Maker (DMM) for a stock technically determines the opening price. There is a person physically on the NYSE trading floor who represents the DMM firm who technically opens the different stocks. They have a weird custom keyboard from NYSE for this purpose... The price is usually calculated algorithmically by the DMM firm and sent to the person at NYSE to approve. Pretty arcane. Also somewhat shady, as the DMM firm can be and is part of the auction themselves. DMM firms can analyze the order book to see what the imbalance is in the overlapping region, and place an order of their own to correct the imbalance and then set the opening price. I can see how one can profit from this in certain situations
papito 4y ago
I sweat over "idiot-proofing" the smallest systems, while multi-billion dollar operations don't seem to care enough. Like the S3 being blown away with a simple change in the early days, or GitHub running a test suite with production settings. It's like the FIRST thing I think about when starting a project. https://github.blog/2010-11-15-today-s-outage/ https://github.blog/2010-11-15-today-s-outage/
ynniv 4y ago
It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or so there will be a comprehensive internal post mortem, and every engineer in the company will read it because that's why they work there. "The staffer" will not be named, nor will they be fired. The process will be changed. The systems will be changed. You probably haven't heard of Pillar, but the NYSE in your head was replaced by some pretty amazing, distributed, low latency systems. The culture is to over-engineer, over-provision, plan for black swans. And test. That it works. Test that it scales. Test that backups work. Test, test, test. firmitatis, utilitatis, venustatis. This failure was due to daily testing. Sometimes things still fail. That's true anywhere. In most places your failures don't make the papers, and accidents are swept under the rug. That doesn't happen at NYSE for obvious reasons. They're not building large language models (that I know of), or self driving cars (pretty sure on this one), but they're a modern, cutting edge, "soft" real-time engineering shop. If you haven't looked already, you might find something interesting there: https://www.ice.com/careers https://www.ice.com/careers
- jacquesm 4y ago
  Just naming a 'staffer' though seems to already be a way to apportion blame to a segment of the employees, insulating management from what was done. Named or not doesn't really matter, clearly blame is being assigned.
  - bostonsre 4y ago
    Yea, it sounds like an issue with process and automation. It shouldn't have been possible for the staffer to make a mistake that would cause this.
    - jacquesm 4y ago
      Precisely. It is never just one error. At a minimum two and if you really stare at this sort of thing long enough it isn't rare at all to discover a whole chain of them. The only difference with all the times that it went right is that this time everything was aligned 'just so'.
throwawaaarrgh 4y ago
There is so much stupidity in the process they describe, I have no faith it will be fixed. a manual daily DR test that clearly wasn't followed by a test or checklist or double checked by another person, and leaving the DR up broke prod?? literally none of those things should have happened. I know the world is held together with duct tape, but it's embarrassing when you see the tape fall off.
- tgtweak 4y ago
  The way their DR is setup is that clients of NYSE (brokerages, OTC systems, firms, banks) all have IP (not dns) connections to the primary NYSE production datacenter and a full second set of IPs for the DR site. It's not a "dns and load balancers" setup where the service itself can just route the traffic somewhere else. The clients themselves determine where to connect to consume trade data and execute trades. There is likely some modus operandi given to clients on how to connect to primary and DR sites based on some specific logic. The NYSE DR guide [1] says that if DR is active, production is not. It's not a distant reach to consider that some of these clients have a deadman switch doing a healthcheck poll on DR and switching to it when it see's that it is "up". If they've built their systems in such a way that when it detects the DR site active it uses that, then it makes sense that having both "online" would cause some havoc. I'm sure the complexity of the entire exchange is fairly significant, and having "two" copies of it running in parallel with both able to accept and execute trades would be a scenario that can cause some unintended consequences. Fundamentally, an exchange is "atomic" and transactional and cannot be meaningfully distributed to two sites that are that far away. The replication in place is likely master/slave with a switch to make the slave primary. Anyone who has toyed with master-master replication on less complicated databases knows the issues that can come up with split writes. Imagine that at the scale of a system as large as the NYSE. [1] https://www.nyse.com/publicdocs/support/DisasterRecoveryFAQs.pdf https://www.nyse.com/publicdocs/support/DisasterRecoveryFAQs...
jonpo 4y ago
Its nearly always the damn humans. I find it awful that we are often assigning blame to some "technology error" when its the damn humans pulling the strings all the time. all those times the market shut accidentally. or the market opens at the wrong time. or that time someone accidentally deletes all the GTC orders in order to "save some disk space". that time someone tests opening the market at the weekend and puts in the wrong date. Sometimes we are just trying to test that the things work and so we take awful risks like adding test orders, or failing over to test that backup versions of the trading infrastructure still work. All these things add human execution risk. That said I find the US market structure is unfair Charles Schwab does protest too much. Retail orders never seem to get near the central order book. there is no direct market access. brokers just sell your order to whomever MM pays them for the spread in return for a kickback. this should be a fantastic fair multiplayer game, but instead its pay to win mobile crap with vested interests milking their customers.
psychlops 4y ago
This and other exchanges need to be running 24/7, in large part to level the trading field for retail investors. The backend should be handled invisibly behind the scenes. There should further be an exchange version of a Netflix chaos monkey running constantly to ensure such a critical infrastructure is robust. The fact that these systems do not exist is an exchange problem, not a "staffer".
- bagacrap 4y ago
  why do retail traders "need" to exist? Retail traders realistically have only luck to rely on to beat hedge funds and banks. What they do is akin to gambling, which is on net quite negative for those who participate in it and heavily regulated. Retail traders don't serve any purpose in our society. They don't help with efficient allocation of capital and anyone who might be an actual savant in trading can join or start a firm rather than staying independent and unlicensed.
tgtweak 4y ago
https://www.nyse.com/publicdocs/support/DisasterRecoveryFAQs.pdf https://www.nyse.com/publicdocs/support/DisasterRecoveryFAQs... > Question: Can I connect to both the production and the DR site at the same time? Answer: No, only one site is available at a time. When the primary site is up, the DR site is down; and when the DR site is activated, the primary site is down. I think they need to update these docs to say /should/ be down
jmount 4y ago
Galileo's Principle bites hard, great explanation here: https://99percentinvisible.org/episode/cautionary-tales/transcript/ https://99percentinvisible.org/episode/cautionary-tales/tran...
PaulHoule 4y ago
See https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-...
sabujp 4y ago
yeah absolutely nothing to do the SPX hitting a target 4k level and messing up everything just as it hit that level

I spent an hour trying to figure out why my new stock purchase had disappeared from my account. I had an order placed for opening on Tuesday morning, and I guess I was affected by the trade cancellations. Which is totally weird, because they showed up in my account on Tuesday morning after opening.

halotrope 4y ago

That means some process is broken. Imagine reading “flight crashed because baggage handler place some suitcases wrongly in the cargo space”

helsinkiandrew 4y ago

https://archive.ph/UoMr9 https://archive.ph/UoMr9

bmitc 4y ago

Seems so weird to not have automated checks for something that seems to be described as "someone left the light on" and also not have the exchange automatically initiate itself. However, it still isn't that clear what the problem was. Were prices not "real" or correct? Stuff like this will happen more and more. We treat software driven systems rather recklessly.

mrkeen 4y ago
More automation -> more code -> more things to go wrong
- bongobingo1 4y ago
  The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.
  - junon 4y ago
    I've not seen it put so eloquently.
    - dsr3 4y ago
      To err is human, to really foul things up requires a computer - William E. Vaughan https://quoteinvestigator.com/2010/12/07/foul-computer/ https://quoteinvestigator.com/2010/12/07/foul-computer/
      - erik_seaberg 4y ago
        “A computer lets you make more mistakes faster than any other invention, with the possible exceptions of handguns and tequila.”—Mitch Ratcliffe
  - WJW 4y ago
    Some submarine operations are deliberately not automated, because if eg a sensor is broken or miscalibrated it could sink the entire boat if a computer very rapidly acts on the wrong information. Rather they do those operations with one person operating the valve/machine/device/etc, another watching and confirming readings, and the whole thing is on a constant audio link with a third person in an engineering room who watches the readings through a centralized system. It's clearly not optimized for efficient use of personnel, but the personnel complement will have been designed to provide sufficient people at all times and the cost of getting it wrong can be very large indeed.
    - TeMPOraL 4y ago
      This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general).
      - dragonwriter 4y ago
        > This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general The canon explanation is that automation-in-charge was experimented with and went really badly, though periodically they try something approaching it again. https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(episode) https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(... (AI, human genetic engineering, and a number of other areas of technology are affected by variants of this issue in the Trek canon.)
- themitigating 4y ago
  This employee left the backup system running. There's obviously some automation but what is the solution? Process changes that people have to remember or more systems to prevent the issue. So I don't get your statement related to this article
  - nix23 4y ago
    >There's obviously some automation but what is the solution? Shutdown and wake-up time in bios of server and switch ;)
  - deleted 4y ago
    [deleted]
  - mrkeen 4y ago
    I don't imply that there is a solution. We will simply create a second system (B) to monitor the first system (A). Now we have two systems to maintain. System B will not be capable of steering A by itself. So we still need to know how to diagnose and repair A, and we also need to know about B too. Maybe system B can talk to a Prometheus/Grafana stack (if it's up). And that can put alerts into Slack (which we ignore because there's always alerts in Slack). And after standup we can take turns looking at graphs with consternation. > Stuff like this will happen more and more. We treat software driven systems rather recklessly. That sentence is where I go when I hear the word 'automation'.
- nix23 4y ago
  >More automation -> more code -> more things to go wrong More people -> more entropy -> much more things to go wrong
  - wongarsu 4y ago
    One person might be inattentive or drunk, but it's less likely that two people are. So you institue a two-person rule. And if that's not good enough, add a third person to double-check. Maybe a supervisor to observe the people doing all of the above, to catch any mistakes or negligent behavior. Also have them write down the steps they have taken, and have somebody else read through that to verify. Just keep adding people until you are satisfied with your odds (and hope you are not making it worse through second-order effects)
    - nix23 4y ago
      So two people are more reliable as a automated system you wanna say? That's totally wrong....
      - reaperducer 4y ago
        Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. I did last year, and my company is in the process of de-automating certain processes that can endanger that company if they go wrong. There are many things in tech that are too important to automate. I'd even posit that the more experience you have in tech, the more you've seen how things go wrong, and the more you realize that automation is a tool for humans to use, not a replacement for humans doing a task.
        nix23 4y ago
        >Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. Much much much more due to human error......but hey maybe you are the worst programmer ever..but even then i would say your programs are more reliable then a human.
baxtr 4y ago
[flagged]
qrybam 4y ago
The auction is meant to find a stable price and can have some wild prices coming in, because no matching happens at the point of entry, the market will naturally find a level before trading commences. In this case, those wild prices were matching, resulting in crazy trades no-one would have expected. You'd be surprised how many manual processes there are in places like this. It's a combination of legacy systems / processes, and a general paranoia around automation going wrong. I wouldn't be surprised if they always have someone there to shepherd the system along.
- benjaminwootton 4y ago
  When I worked on a trading platform, I spent many a happy Sunday night waiting for the Australian market to open and watch the first orders go through successfully. We had hundreds of jobs and upgrades happening over each weekend. It definetly needed an eye casting over it regardless of the automation.
  - xwolfi 4y ago
    I am working at one now and nothing has changed. Automations are so many we recruit an army of people just checking if they ran, while knowing how to replace them if they didnt (or, more accurately, who to call at night to fix it asap). And the AU orders going through is a good sign, but it's far from guaranteeing a free monday, as Japan, Korea or Shanghai can fuck it up, each in their own little ways. Hong Kong is the best, low regulatory crap, invested regulator, high volume low latency traffic everyday (relative to the region), I cant recall a time it broke. Once, someone fat fingered an excel import at close, and we lost our trading license for that entire country for 18 months. And we're not small. But the amount mismatched at settlement was super tiny. High attack surface, low holistic understanding (it works despite us, we honestly have no clue sometimes), heavy consequences on screwup.
helsinkiandrew 4y ago
Matt Levine at Bloomberg gives a good explanation: https://www.bloomberg.com/opinion/articles/2023-01-25/nyse-forgot-to-open-yesterday https://www.bloomberg.com/opinion/articles/2023-01-25/nyse-f... Basically at the market open all the requests to buy and sell get matched at the same "open" auction price, then (a second later) the orders get sent to the order book, where the price can go up and down based on size. Because the system didn't think there was an opening, there wasn't the opening auction, the prices went straight to the book and there were large swings in price.
- usefulcat 4y ago
  https://archive.is/LovhH https://archive.is/LovhH
tgtweak 4y ago
I don't think it's that they "left it running" like you would leave a backup app running... they literally left the entire disaster-recovery site up and running and live. Cermak (referred to in the article as the "backup") is an entire datacenter, hosting a running copy of the exchange to be used in a failover scenario. You'd have to have more than 1 person involved to forget that DR is still active when completing these failover exercises and tests off-hours.
- Kon-Peki 4y ago
  > Cermak (referred to in the article as the "backup") is an entire datacenter Well, they are a tenent at the Cermak data center. It’s a truly massive building with huge amounts of connectivity and colo opportunities. Probably also the only 100+ year old data center building on the US register of historic places, lol (it’s a former catalog printing facility, built to hold insanely heavy printing presses on 8 or 9 really tall floors, so it has no problems with densely-packed server racks)
  - tgtweak 4y ago
    Yeah my point is only that it's a full DR site, not a "backup" that was left running as the article pointed out (and as a lot of commenters are insinuating).
    - Kon-Peki 4y ago
      Ah, I see. Some quick internet searching shows that from time to time (rarely), NYSE operates the Cermak site as the primary site for at least part of their operations - for up to a week at a time. It seems to have multiple purposes - DR, customer software testing, etc.
      - wallstprog 4y ago
        Well, you do that to make sure it works. At one of my previous jobs, we'd switch the primary and secondary datacenters every six months just to make sure we could.

h3daz 4y ago

I am short a couple dozen naked Feb 3 calls on a lot of affected tickers and almost had a heart attack when looking at one of them on my phone. Thankfully I was not in front of my computer at the time because I have no idea how my broker was managing my margin at open.

qeternity 4y ago
The LULD breakers saved everyone from more chaos here because effectively as soon as the market opened, all these symbols were halted.

WiSaGaN 4y ago

A lot of stocks have minute bar of wide range prices in the first few minutes of continuous trading. This seems like the incident in what caused Knight Capital fiasco, in which the system repeatedly buy on the ask and sell on the bid very fast, thus pushing the high price high and low price low. In the opening usually the market maker will be more weary of the risk they cannot hedge directly and thus will be less willing to take on positions, leading to wild swings. Still this report (and the previous statement) does not give enough detail on why a backup system misoperation resulted this. Also, critical large systems like exchange rarely have a single point failure. Usually there will be a sequence of issues along the event chains leading to this. Thus one "failed to properly shutdown" caused all this is a bit incredible. We will need more explanation.

itronitron 4y ago
Doesn't each trade require both a buyer and a seller, who buy and sell at an agreed on price? Presumably both parties would be satisfied with the trade so it isn't clear to me what all the fuss is about.
- pcl 4y ago
  Market price trades are executed at whatever the current price is. Presumably it’s those trades that caused havoc.
- gpderetta 4y ago
  I haven't really researched anything, but both parties thought they were bid/offering into an auction with time to cancel or amend their orders.
- WiSaGaN 4y ago
  I don't know what happened in current case. In the Knight Capital case, KC clearly didn't intend to send those erroneous orders. And if those trades were not annuled, KC would not be able to settle those, since the trade loss were larger than the collateral KC put up.
  - 4y ago

hksoftware 4y ago

[flagged]

ideamotor 4y ago
You suspect a breach?
- avree 4y ago
  He is, undoubtedly, a meme-stock conspiracy theorist. Only those steeped in the cult of AMC, GME, or BBBY say things like that.
  - deleted 4y ago
    [deleted]
shapefrog 4y ago
I am curious what your "independent research" has turned up on the subject.
deleted 4y ago
[deleted]

lordnacho 4y ago

Backup system connected to prod, that somehow reminds of the Knight Trading debacle. Someone there apparently connected some test code to their prod and they blew up the company in under an hour.

laurencei 4y ago
I mean - isnt that how all DR is essentially configured? You need it "somehow" connected to Prod depending on the failover, config, system etc. And in many of these complex systems DR can be on a subsystem etc - not an "all or nothing" approach?

afhammad 4y ago

It seems that this wasn't as routine as these things aught to be but rarely are.

H8crilA 4y ago
Yes it could have also been a test of potential escalation from Solomon Islands.

mellosouls 4y ago

These "issue traced to staffer" stories sound like management cover up for management/system shortcomings to me. Systems with such significant potential impact, and in industries where lack of financial investment in their continuity is a deliberate choice have very little excuse to be passing the buck to grunts for basic process flaws that can be triggered by individual error.

wrldos 4y ago
This. If an individual's mistake can take out your business you have a process control problem and that is owned by management.
- iancmceachern 4y ago
  Exactly, management shouldn't allow this kind of situation to occur by designing the xomoanies processes such that there are checks and balances
- thisarticle 4y ago
  The days of management taking responsibility for anything are over. See: not a single CEO stepping down for over hiring.
  - deleted 4y ago
    [deleted]
  - lotsofspots 4y ago
    Oh, they take full responsibility, it always says so in the mails they send out. It's just that taking responsibility doesn't appear to actually result in anything happening.
    - parsimo2010 4y ago
      A cyncical take might be that they are saying that they take responsibility (credit) for reducing the monthly payroll expenses. They may also have overhired in the past, but what's in the past was already paid for. The savings next month is how they justify a large paycheck.

herpderperator 4y ago

If the trades are being cancelled, are they going to correct the chart data? Right now it looks very misleading on the daily[0], weekly, monthly, quarterly, yearly etc for large caps that trade quite steadily otherwise. I do understand that this would be a challenging effort as that data already flowed to and was stored by all the broker-dealers, but I think it should be done. [0] https://www.dropbox.com/s/6jdmgkdyei9xqz0/mcd.png?dl=0 https://www.dropbox.com/s/6jdmgkdyei9xqz0/mcd.png?dl=0

evanpw 4y ago
The same feed that publishes trades also publishes trade busts, so it's up to whoever's consuming it downstream to take care of.
anonu 4y ago
Technically yes, historical market data feed needs to be cleaned up. Which will be a nightmare for every single person who maintains one... Which is also why exchanges are very reluctant to mass cancel trades. The knock on effect goes beyond just market data feeds

lvl102 4y ago

Remember the flash crash of 2015? They let those trades actually STAND. Including options. This week’s open was nothing in comparison.

detaro 4y ago
That wasn't an exchange "malfunction" in the sense that the exchange did not do what it was supposed to, was it?
- lvl102 4y ago
  Do you really think they will take responsibility for billions lost/made that day? No, that's a big liability. Anyone trading that day knew it was a big glitch at open. Some names, BLUE CHIPS, were down 40-50%! What shocked us all is that they actually allowed the trades to stand. What was different about this week was that the SEC actually tried to do their jobs for once and the exchange had to address it ie come up with bullshit excuse.
  - spywaregorilla 4y ago
    So... what was wrong? Why should those people not have been allowed to make and lose money?

corobo 4y ago

Oh we're still doing scapegoats? If your system can be hosed by a single person the system is at fault. Start with the scapegoat's manager.

hknmtt 4y ago

there is no way a person, a single person, an unauthorized person, can have access into such system/functionality like this. utter BS.

jterrys 4y ago
my 2c: In reality what probably happened is previous market day and post-trading data encountered some kind of error, which triggered a cascade of problems overnight that they were unable to properly rectify. This caused delays up until market open. They were unable to fully resolve the issue, and forced with either delaying opening the market (which is a HUGE no-no) or opening with wrong data as is, they chose wrong data. All in all a lot of people didn't get much sleep Monday. More than likely they implemented some changes or updates over the weekend that were not properly done, or they encountered some errors, and didn't have adequate controls/time to roll-back Monday night. They made the right calls too late and there was a controls process up the chain that seriously fucked up. These are the kinds of problems that get the CEO woken up in the middle of the night.

kmac_ 4y ago

Weak blame game.

pedro2 4y ago

Successes are management's, failures are individual's :)

chiefalchemist 4y ago
True. But when that happens, that's a textbook sign of lack of leadership.

chiefalchemist 4y ago

While we're on the subject, if your company uses technology in an capacity, read the book The Phoenix Project. https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/1942788290 https://www.amazon.com/Phoenix-Project-DevOps-Helping-Busine...

dogleash 4y ago
I wanted to like that book but it's a big Just-so story. tl;dr: The brave knight implemented devops and everyone lived happily ever after!
- chiefalchemist 4y ago
  Yeah. But it made clear many things that are otherwise too often cloudy. And despite its age it's still relevant. Some people need a story.

irthomasthomas 4y ago

Is it normal to simply accept the word of an anonymous source for something so important? I genuinely don't know, anymore, but it doesn't seem like a good idea. I'd rather wait for a more thorough investigation. Especially when the story from these sources boils down to "Kevin was in charge of booting the NYSE App that morning, but he was late for work. He had a good excuse, though, he flaked! We'll have the chap straight up for lunch, no question". Edit: I also note that this piece is lacking the traditional "The NYSE did not respond to a request for comment".

itsoktocry 4y ago
>Is it normal to simply accept the word of an anonymous source for something so important? Anonymous means they aren't revealing the source, not that Bloomberg doesn't know who the sources is, or what they do.
- irthomasthomas 4y ago
  I know that. I am referring to you and I, the reader accepting the word of the anonymous source. Combined with the fact that they apparently did not ask for a comment from the NYSE before publishing this. Or if they did, they neglected to mention it.
  - maronato 4y ago
    We aren’t accepting the word of the anonymous source. We’re accepting Bloomberg’s word that the source is reliable.
    - adolph 4y ago
      Bloomberg's word: https://news.ycombinator.com/item?id=19526348 https://news.ycombinator.com/item?id=19526348
    - bink 4y ago
      Some of us aren't "accepting" anything. We're just reading about a potential cause of an incident and speculating about how it could happen to us or could have been prevented. Just because we're reading this article and commenting here doesn't mean we just believe everything that we read. The post-mortem will come out soon enough and we'll read that and comment again.

nickdothutton 4y ago

If a single staffer SNAFU can send your exchange into chaos then you dun goofed at risk management and probably a whole lot of other management discipline.

h2odragon 4y ago

Tuesday news blackout; Thursday "it was all Jim's fault"... Right. Smells like horse shit. Most of what comes out of the profession of "journalism" does too, lately; but this smells strongly.

yakubin 4y ago
Why couldn’t have been Jim’s fault? The world is run by Jims.
- h2odragon 4y ago
  As others have said, if Jim's fuckup can have such large consequences, then there should have been backstops for Jim who would have shared the blame. Nothing against Jim, it's just he's being used as a distraction to avoid talking about the real issues.
  - throwawaaarrgh 4y ago
    Manager: "I hope nobody asks why Jim was allowed to have so much power with no oversight or validation"
epistemer 4y ago
and all the trades are being reversed basically. The whole thing stinks.
- tgtweak 4y ago
  Not all of them... only the most egregious.

anonu 4y ago

Tech can be so fragile. You do everything right, trade millions of shares everyday and handle billions of dollars. But you forget to run one script to shutdown a backup system and everything comes crashing down: your reputation in tatters, millions in costs to settle bad trades, barbarians at the gates.

lr1970 4y ago

From the article: > Meanwhile, market professionals and day traders are rattled and waiting for the exchange to elaborate on what it publicly called a “manual error” involving its “disaster recovery configuration”. Oh, I love it -- a disaster caused by "disaster recovery configuration" :-)

gjvc 4y ago
Oh, I love it -- a disaster caused by "disaster recovery configuration" :-) People install failover configurations to minimise time-to-repair or time-to-resume service (and some customers' contracts will demand this). This is at the expense of another layer of stuff to go wrong, and raising the possibility that it fails over when it shouldn't, causing brief but embarrassing outages. It's possible in some such situations that, on the balance of probabilities, introducing mechanisms like this cause more disruption over time than they were intended to protect against, and that this is more widespread than often considered. Still, their operational cost must be borne in order to satisfy the clause in the customers' contracts.

twawaaay 4y ago

Oh, yes. It is the staffer who made the mistake. What about people who designed it this way?

jrochkind1 4y ago

> That misled the exchange’s computers to treat the 9:30 a.m. opening bell as a continuation of trading, and so they skipped the day’s opening auctions that neatly set initial prices. I didn't even know about this process. I don't know much about trading, but it surprises me that there is a separate process for setting prices at the start of trading, and that if it's missed, chaotic prices result. Is this related to how stock markets aren't really ever open 24 hours? Do they need that reset to function in stable way?

xyzelement 4y ago
That's right. In short it's something like this: stocks trade on their primary exchanges during specific hours. For example 9:30 to 4 in the US. Part of it is legacy from when trading was done by actual humans being at the exchange physically to trade during those times and part of it (I would guess is still the case) is to allow plenty of non-trading hours for back-office jobs and settlement. So yes there's a special start of day process that runs at 9:30 that runs through all the orders on the books at that time and determines a price at which some optimal set of those orders can trade, trades them at that price, and also posts that price as the Open price for the day. The process is different during continuous trading since orders are one by one matched against the order book. Source: ran one of the world's largest equity platforms for 5 years.
- ajoseps 4y ago
  isn't there another component to the NYSE auction where the DMM has some input into what the closing/opening price actually is?
  - khold_stare 4y ago
    Yes. In my reply to the first comment I mentioned setting the opening price is "more complicated". Every exchange has their own system for the opening auction which you buy into when you list with a particular exchange. Most exchanges have an algorithmic way of calculating the price. For NYSE, it's again more historical. A Designated Market Maker (DMM) for a stock technically determines the opening price. There is a person physically on the NYSE trading floor who represents the DMM firm who technically opens the different stocks. They have a weird custom keyboard from NYSE for this purpose... The price is usually calculated algorithmically by the DMM firm and sent to the person at NYSE to approve. Pretty arcane. Also somewhat shady, as the DMM firm can be and is part of the auction themselves. DMM firms can analyze the order book to see what the imbalance is in the overlapping region, and place an order of their own to correct the imbalance and then set the opening price. I can see how one can profit from this in certain situations

papito 4y ago

I sweat over "idiot-proofing" the smallest systems, while multi-billion dollar operations don't seem to care enough. Like the S3 being blown away with a simple change in the early days, or GitHub running a test suite with production settings. It's like the FIRST thing I think about when starting a project. https://github.blog/2010-11-15-today-s-outage/ https://github.blog/2010-11-15-today-s-outage/

ynniv 4y ago

It's easy to throw shade when Bloomberg writes an article that puts the blame on "a staffer". Having worked near some of these systems, the engineering and process are actually quite good. How many companies publish their private network topology, service p99.9 in microseconds, and detailed pricing on the open web? They're in a painfully competitive global market that's ambivalent to names on buildings. In a week or so there will be a comprehensive internal post mortem, and every engineer in the company will read it because that's why they work there. "The staffer" will not be named, nor will they be fired. The process will be changed. The systems will be changed. You probably haven't heard of Pillar, but the NYSE in your head was replaced by some pretty amazing, distributed, low latency systems. The culture is to over-engineer, over-provision, plan for black swans. And test. That it works. Test that it scales. Test that backups work. Test, test, test. firmitatis, utilitatis, venustatis. This failure was due to daily testing. Sometimes things still fail. That's true anywhere. In most places your failures don't make the papers, and accidents are swept under the rug. That doesn't happen at NYSE for obvious reasons. They're not building large language models (that I know of), or self driving cars (pretty sure on this one), but they're a modern, cutting edge, "soft" real-time engineering shop. If you haven't looked already, you might find something interesting there: https://www.ice.com/careers https://www.ice.com/careers

jacquesm 4y ago
Just naming a 'staffer' though seems to already be a way to apportion blame to a segment of the employees, insulating management from what was done. Named or not doesn't really matter, clearly blame is being assigned.
- bostonsre 4y ago
  Yea, it sounds like an issue with process and automation. It shouldn't have been possible for the staffer to make a mistake that would cause this.
  - jacquesm 4y ago
    Precisely. It is never just one error. At a minimum two and if you really stare at this sort of thing long enough it isn't rare at all to discover a whole chain of them. The only difference with all the times that it went right is that this time everything was aligned 'just so'.

throwawaaarrgh 4y ago

There is so much stupidity in the process they describe, I have no faith it will be fixed. a manual daily DR test that clearly wasn't followed by a test or checklist or double checked by another person, and leaving the DR up broke prod?? literally none of those things should have happened. I know the world is held together with duct tape, but it's embarrassing when you see the tape fall off.

tgtweak 4y ago
The way their DR is setup is that clients of NYSE (brokerages, OTC systems, firms, banks) all have IP (not dns) connections to the primary NYSE production datacenter and a full second set of IPs for the DR site. It's not a "dns and load balancers" setup where the service itself can just route the traffic somewhere else. The clients themselves determine where to connect to consume trade data and execute trades. There is likely some modus operandi given to clients on how to connect to primary and DR sites based on some specific logic. The NYSE DR guide [1] says that if DR is active, production is not. It's not a distant reach to consider that some of these clients have a deadman switch doing a healthcheck poll on DR and switching to it when it see's that it is "up". If they've built their systems in such a way that when it detects the DR site active it uses that, then it makes sense that having both "online" would cause some havoc. I'm sure the complexity of the entire exchange is fairly significant, and having "two" copies of it running in parallel with both able to accept and execute trades would be a scenario that can cause some unintended consequences. Fundamentally, an exchange is "atomic" and transactional and cannot be meaningfully distributed to two sites that are that far away. The replication in place is likely master/slave with a switch to make the slave primary. Anyone who has toyed with master-master replication on less complicated databases knows the issues that can come up with split writes. Imagine that at the scale of a system as large as the NYSE. [1] https://www.nyse.com/publicdocs/support/DisasterRecoveryFAQs.pdf https://www.nyse.com/publicdocs/support/DisasterRecoveryFAQs...

jonpo 4y ago

Its nearly always the damn humans. I find it awful that we are often assigning blame to some "technology error" when its the damn humans pulling the strings all the time. all those times the market shut accidentally. or the market opens at the wrong time. or that time someone accidentally deletes all the GTC orders in order to "save some disk space". that time someone tests opening the market at the weekend and puts in the wrong date. Sometimes we are just trying to test that the things work and so we take awful risks like adding test orders, or failing over to test that backup versions of the trading infrastructure still work. All these things add human execution risk. That said I find the US market structure is unfair Charles Schwab does protest too much. Retail orders never seem to get near the central order book. there is no direct market access. brokers just sell your order to whomever MM pays them for the spread in return for a kickback. this should be a fantastic fair multiplayer game, but instead its pay to win mobile crap with vested interests milking their customers.

psychlops 4y ago

This and other exchanges need to be running 24/7, in large part to level the trading field for retail investors. The backend should be handled invisibly behind the scenes. There should further be an exchange version of a Netflix chaos monkey running constantly to ensure such a critical infrastructure is robust. The fact that these systems do not exist is an exchange problem, not a "staffer".

bagacrap 4y ago
why do retail traders "need" to exist? Retail traders realistically have only luck to rely on to beat hedge funds and banks. What they do is akin to gambling, which is on net quite negative for those who participate in it and heavily regulated. Retail traders don't serve any purpose in our society. They don't help with efficient allocation of capital and anyone who might be an actual savant in trading can join or start a firm rather than staying independent and unlicensed.

tgtweak 4y ago

https://www.nyse.com/publicdocs/support/DisasterRecoveryFAQs.pdf https://www.nyse.com/publicdocs/support/DisasterRecoveryFAQs... > Question: Can I connect to both the production and the DR site at the same time? Answer: No, only one site is available at a time. When the primary site is up, the DR site is down; and when the DR site is activated, the primary site is down. I think they need to update these docs to say /should/ be down

jmount 4y ago

Galileo's Principle bites hard, great explanation here: https://99percentinvisible.org/episode/cautionary-tales/transcript/ https://99percentinvisible.org/episode/cautionary-tales/tran...

PaulHoule 4y ago

See https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-...

sabujp 4y ago

yeah absolutely nothing to do the SPX hitting a target 4k level and messing up everything just as it hit that level