45 ms·

Seems so weird to not have automated checks for something that seems to be described as "someone left the light on" and also not have the exchange automatically

by bmitc 4y ago

Seems so weird to not have automated checks for something that seems to be described as "someone left the light on" and also not have the exchange automatically initiate itself. However, it still isn't that clear what the problem was. Were prices not "real" or correct? Stuff like this will happen more and more. We treat software driven systems rather recklessly.

mrkeen 4y ago
More automation -> more code -> more things to go wrong
- bongobingo1 4y ago
  The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.
  - junon 4y ago
    I've not seen it put so eloquently.
    - dsr3 4y ago
      To err is human, to really foul things up requires a computer - William E. Vaughan https://quoteinvestigator.com/2010/12/07/foul-computer/ https://quoteinvestigator.com/2010/12/07/foul-computer/
      - erik_seaberg 4y ago
        “A computer lets you make more mistakes faster than any other invention, with the possible exceptions of handguns and tequila.”—Mitch Ratcliffe
  - WJW 4y ago
    Some submarine operations are deliberately not automated, because if eg a sensor is broken or miscalibrated it could sink the entire boat if a computer very rapidly acts on the wrong information. Rather they do those operations with one person operating the valve/machine/device/etc, another watching and confirming readings, and the whole thing is on a constant audio link with a third person in an engineering room who watches the readings through a centralized system. It's clearly not optimized for efficient use of personnel, but the personnel complement will have been designed to provide sufficient people at all times and the cost of getting it wrong can be very large indeed.
    - TeMPOraL 4y ago
      This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general).
      - dragonwriter 4y ago
        > This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general The canon explanation is that automation-in-charge was experimented with and went really badly, though periodically they try something approaching it again. https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(episode) https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(... (AI, human genetic engineering, and a number of other areas of technology are affected by variants of this issue in the Trek canon.)
      - mschuster91 4y ago
        At least in ST:VOY, it has been shown that a single hologram is capable of running the ship - although one might argue "exceptional circumstances" ;)
        wongarsu 4y ago
        In Star Trek III they jury-rig the Enterprise to fly with a crew of 5, instead of the regular crew of 400. Though in that state it can't do much more than fly: combat capabilities are strongly diminished, maintenance doesn't happen, post-combat repairs are out of the question, science missions would be much harder. On occasion the Enterprise has transported 150 passengers, so I imagine there's a lot of kitchen staff, security, etc. You only need 5 people to fly the ship, maybe 40 to fly sustainably with maintenance, but to actually accomplish their reglar mission you need the other 300 people.
  - ynniv 4y ago
    Knight Capital will forever haunt fintech engineers... https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-...
- themitigating 4y ago
  This employee left the backup system running. There's obviously some automation but what is the solution? Process changes that people have to remember or more systems to prevent the issue. So I don't get your statement related to this article
  - nix23 4y ago
    >There's obviously some automation but what is the solution? Shutdown and wake-up time in bios of server and switch ;)
  - deleted 4y ago
    [deleted]
  - mrkeen 4y ago
    I don't imply that there is a solution. We will simply create a second system (B) to monitor the first system (A). Now we have two systems to maintain. System B will not be capable of steering A by itself. So we still need to know how to diagnose and repair A, and we also need to know about B too. Maybe system B can talk to a Prometheus/Grafana stack (if it's up). And that can put alerts into Slack (which we ignore because there's always alerts in Slack). And after standup we can take turns looking at graphs with consternation. > Stuff like this will happen more and more. We treat software driven systems rather recklessly. That sentence is where I go when I hear the word 'automation'.
- nix23 4y ago
  >More automation -> more code -> more things to go wrong More people -> more entropy -> much more things to go wrong
  - wongarsu 4y ago
    One person might be inattentive or drunk, but it's less likely that two people are. So you institue a two-person rule. And if that's not good enough, add a third person to double-check. Maybe a supervisor to observe the people doing all of the above, to catch any mistakes or negligent behavior. Also have them write down the steps they have taken, and have somebody else read through that to verify. Just keep adding people until you are satisfied with your odds (and hope you are not making it worse through second-order effects)
    - nix23 4y ago
      So two people are more reliable as a automated system you wanna say? That's totally wrong....
      - reaperducer 4y ago
        Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. I did last year, and my company is in the process of de-automating certain processes that can endanger that company if they go wrong. There are many things in tech that are too important to automate. I'd even posit that the more experience you have in tech, the more you've seen how things go wrong, and the more you realize that automation is a tool for humans to use, not a replacement for humans doing a task.
        nix23 4y ago
        >Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. Much much much more due to human error......but hey maybe you are the worst programmer ever..but even then i would say your programs are more reliable then a human.
baxtr 4y ago
[flagged]
qrybam 4y ago
The auction is meant to find a stable price and can have some wild prices coming in, because no matching happens at the point of entry, the market will naturally find a level before trading commences. In this case, those wild prices were matching, resulting in crazy trades no-one would have expected. You'd be surprised how many manual processes there are in places like this. It's a combination of legacy systems / processes, and a general paranoia around automation going wrong. I wouldn't be surprised if they always have someone there to shepherd the system along.
- benjaminwootton 4y ago
  When I worked on a trading platform, I spent many a happy Sunday night waiting for the Australian market to open and watch the first orders go through successfully. We had hundreds of jobs and upgrades happening over each weekend. It definetly needed an eye casting over it regardless of the automation.
  - xwolfi 4y ago
    I am working at one now and nothing has changed. Automations are so many we recruit an army of people just checking if they ran, while knowing how to replace them if they didnt (or, more accurately, who to call at night to fix it asap). And the AU orders going through is a good sign, but it's far from guaranteeing a free monday, as Japan, Korea or Shanghai can fuck it up, each in their own little ways. Hong Kong is the best, low regulatory crap, invested regulator, high volume low latency traffic everyday (relative to the region), I cant recall a time it broke. Once, someone fat fingered an excel import at close, and we lost our trading license for that entire country for 18 months. And we're not small. But the amount mismatched at settlement was super tiny. High attack surface, low holistic understanding (it works despite us, we honestly have no clue sometimes), heavy consequences on screwup.
helsinkiandrew 4y ago
Matt Levine at Bloomberg gives a good explanation: https://www.bloomberg.com/opinion/articles/2023-01-25/nyse-forgot-to-open-yesterday https://www.bloomberg.com/opinion/articles/2023-01-25/nyse-f... Basically at the market open all the requests to buy and sell get matched at the same "open" auction price, then (a second later) the orders get sent to the order book, where the price can go up and down based on size. Because the system didn't think there was an opening, there wasn't the opening auction, the prices went straight to the book and there were large swings in price.
- usefulcat 4y ago
  https://archive.is/LovhH https://archive.is/LovhH
tgtweak 4y ago
I don't think it's that they "left it running" like you would leave a backup app running... they literally left the entire disaster-recovery site up and running and live. Cermak (referred to in the article as the "backup") is an entire datacenter, hosting a running copy of the exchange to be used in a failover scenario. You'd have to have more than 1 person involved to forget that DR is still active when completing these failover exercises and tests off-hours.
- Kon-Peki 4y ago
  > Cermak (referred to in the article as the "backup") is an entire datacenter Well, they are a tenent at the Cermak data center. It’s a truly massive building with huge amounts of connectivity and colo opportunities. Probably also the only 100+ year old data center building on the US register of historic places, lol (it’s a former catalog printing facility, built to hold insanely heavy printing presses on 8 or 9 really tall floors, so it has no problems with densely-packed server racks)
  - tgtweak 4y ago
    Yeah my point is only that it's a full DR site, not a "backup" that was left running as the article pointed out (and as a lot of commenters are insinuating).
    - Kon-Peki 4y ago
      Ah, I see. Some quick internet searching shows that from time to time (rarely), NYSE operates the Cermak site as the primary site for at least part of their operations - for up to a week at a time. It seems to have multiple purposes - DR, customer software testing, etc.
      - wallstprog 4y ago
        Well, you do that to make sure it works. At one of my previous jobs, we'd switch the primary and secondary datacenters every six months just to make sure we could.

by bmitc 4y ago

mrkeen 4y ago

More automation -> more code -> more things to go wrong

bongobingo1 4y ago
The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.
- junon 4y ago
  I've not seen it put so eloquently.
  - dsr3 4y ago
    To err is human, to really foul things up requires a computer - William E. Vaughan https://quoteinvestigator.com/2010/12/07/foul-computer/ https://quoteinvestigator.com/2010/12/07/foul-computer/
    - erik_seaberg 4y ago
      “A computer lets you make more mistakes faster than any other invention, with the possible exceptions of handguns and tequila.”—Mitch Ratcliffe
- WJW 4y ago
  Some submarine operations are deliberately not automated, because if eg a sensor is broken or miscalibrated it could sink the entire boat if a computer very rapidly acts on the wrong information. Rather they do those operations with one person operating the valve/machine/device/etc, another watching and confirming readings, and the whole thing is on a constant audio link with a third person in an engineering room who watches the readings through a centralized system. It's clearly not optimized for efficient use of personnel, but the personnel complement will have been designed to provide sufficient people at all times and the cost of getting it wrong can be very large indeed.
  - TeMPOraL 4y ago
    This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general).
    - dragonwriter 4y ago
      > This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general The canon explanation is that automation-in-charge was experimented with and went really badly, though periodically they try something approaching it again. https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(episode) https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(... (AI, human genetic engineering, and a number of other areas of technology are affected by variants of this issue in the Trek canon.)
    - mschuster91 4y ago
      At least in ST:VOY, it has been shown that a single hologram is capable of running the ship - although one might argue "exceptional circumstances" ;)
      - wongarsu 4y ago
        In Star Trek III they jury-rig the Enterprise to fly with a crew of 5, instead of the regular crew of 400. Though in that state it can't do much more than fly: combat capabilities are strongly diminished, maintenance doesn't happen, post-combat repairs are out of the question, science missions would be much harder. On occasion the Enterprise has transported 150 passengers, so I imagine there's a lot of kitchen staff, security, etc. You only need 5 people to fly the ship, maybe 40 to fly sustainably with maintenance, but to actually accomplish their reglar mission you need the other 300 people.
- ynniv 4y ago
  Knight Capital will forever haunt fintech engineers... https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-...
themitigating 4y ago
This employee left the backup system running. There's obviously some automation but what is the solution? Process changes that people have to remember or more systems to prevent the issue. So I don't get your statement related to this article
- nix23 4y ago
  >There's obviously some automation but what is the solution? Shutdown and wake-up time in bios of server and switch ;)
- deleted 4y ago
  [deleted]
- mrkeen 4y ago
  I don't imply that there is a solution. We will simply create a second system (B) to monitor the first system (A). Now we have two systems to maintain. System B will not be capable of steering A by itself. So we still need to know how to diagnose and repair A, and we also need to know about B too. Maybe system B can talk to a Prometheus/Grafana stack (if it's up). And that can put alerts into Slack (which we ignore because there's always alerts in Slack). And after standup we can take turns looking at graphs with consternation. > Stuff like this will happen more and more. We treat software driven systems rather recklessly. That sentence is where I go when I hear the word 'automation'.
nix23 4y ago
>More automation -> more code -> more things to go wrong More people -> more entropy -> much more things to go wrong
- wongarsu 4y ago
  One person might be inattentive or drunk, but it's less likely that two people are. So you institue a two-person rule. And if that's not good enough, add a third person to double-check. Maybe a supervisor to observe the people doing all of the above, to catch any mistakes or negligent behavior. Also have them write down the steps they have taken, and have somebody else read through that to verify. Just keep adding people until you are satisfied with your odds (and hope you are not making it worse through second-order effects)
  - nix23 4y ago
    So two people are more reliable as a automated system you wanna say? That's totally wrong....
    - reaperducer 4y ago
      Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. I did last year, and my company is in the process of de-automating certain processes that can endanger that company if they go wrong. There are many things in tech that are too important to automate. I'd even posit that the more experience you have in tech, the more you've seen how things go wrong, and the more you realize that automation is a tool for humans to use, not a replacement for humans doing a task.
      - nix23 4y ago
        >Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. Much much much more due to human error......but hey maybe you are the worst programmer ever..but even then i would say your programs are more reliable then a human.

baxtr 4y ago

[flagged]

qrybam 4y ago

The auction is meant to find a stable price and can have some wild prices coming in, because no matching happens at the point of entry, the market will naturally find a level before trading commences. In this case, those wild prices were matching, resulting in crazy trades no-one would have expected. You'd be surprised how many manual processes there are in places like this. It's a combination of legacy systems / processes, and a general paranoia around automation going wrong. I wouldn't be surprised if they always have someone there to shepherd the system along.

benjaminwootton 4y ago
When I worked on a trading platform, I spent many a happy Sunday night waiting for the Australian market to open and watch the first orders go through successfully. We had hundreds of jobs and upgrades happening over each weekend. It definetly needed an eye casting over it regardless of the automation.
- xwolfi 4y ago
  I am working at one now and nothing has changed. Automations are so many we recruit an army of people just checking if they ran, while knowing how to replace them if they didnt (or, more accurately, who to call at night to fix it asap). And the AU orders going through is a good sign, but it's far from guaranteeing a free monday, as Japan, Korea or Shanghai can fuck it up, each in their own little ways. Hong Kong is the best, low regulatory crap, invested regulator, high volume low latency traffic everyday (relative to the region), I cant recall a time it broke. Once, someone fat fingered an excel import at close, and we lost our trading license for that entire country for 18 months. And we're not small. But the amount mismatched at settlement was super tiny. High attack surface, low holistic understanding (it works despite us, we honestly have no clue sometimes), heavy consequences on screwup.

helsinkiandrew 4y ago

Matt Levine at Bloomberg gives a good explanation: https://www.bloomberg.com/opinion/articles/2023-01-25/nyse-forgot-to-open-yesterday https://www.bloomberg.com/opinion/articles/2023-01-25/nyse-f... Basically at the market open all the requests to buy and sell get matched at the same "open" auction price, then (a second later) the orders get sent to the order book, where the price can go up and down based on size. Because the system didn't think there was an opening, there wasn't the opening auction, the prices went straight to the book and there were large swings in price.

usefulcat 4y ago
https://archive.is/LovhH https://archive.is/LovhH

tgtweak 4y ago

I don't think it's that they "left it running" like you would leave a backup app running... they literally left the entire disaster-recovery site up and running and live. Cermak (referred to in the article as the "backup") is an entire datacenter, hosting a running copy of the exchange to be used in a failover scenario. You'd have to have more than 1 person involved to forget that DR is still active when completing these failover exercises and tests off-hours.

Kon-Peki 4y ago
> Cermak (referred to in the article as the "backup") is an entire datacenter Well, they are a tenent at the Cermak data center. It’s a truly massive building with huge amounts of connectivity and colo opportunities. Probably also the only 100+ year old data center building on the US register of historic places, lol (it’s a former catalog printing facility, built to hold insanely heavy printing presses on 8 or 9 really tall floors, so it has no problems with densely-packed server racks)
- tgtweak 4y ago
  Yeah my point is only that it's a full DR site, not a "backup" that was left running as the article pointed out (and as a lot of commenters are insinuating).
  - Kon-Peki 4y ago
    Ah, I see. Some quick internet searching shows that from time to time (rarely), NYSE operates the Cermak site as the primary site for at least part of their operations - for up to a week at a time. It seems to have multiple purposes - DR, customer software testing, etc.
    - wallstprog 4y ago
      Well, you do that to make sure it works. At one of my previous jobs, we'd switch the primary and secondary datacenters every six months just to make sure we could.