4 ms·

More automation -> more code -> more things to go wrong

by mrkeen 4y ago

bongobingo1 4y ago
The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.
- junon 4y ago
  I've not seen it put so eloquently.
  - dsr3 4y ago
    To err is human, to really foul things up requires a computer - William E. Vaughan https://quoteinvestigator.com/2010/12/07/foul-computer/ https://quoteinvestigator.com/2010/12/07/foul-computer/
    - erik_seaberg 4y ago
      “A computer lets you make more mistakes faster than any other invention, with the possible exceptions of handguns and tequila.”—Mitch Ratcliffe
- WJW 4y ago
  Some submarine operations are deliberately not automated, because if eg a sensor is broken or miscalibrated it could sink the entire boat if a computer very rapidly acts on the wrong information. Rather they do those operations with one person operating the valve/machine/device/etc, another watching and confirming readings, and the whole thing is on a constant audio link with a third person in an engineering room who watches the readings through a centralized system. It's clearly not optimized for efficient use of personnel, but the personnel complement will have been designed to provide sufficient people at all times and the cost of getting it wrong can be very large indeed.
  - TeMPOraL 4y ago
    This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general).
    - dragonwriter 4y ago
      > This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general The canon explanation is that automation-in-charge was experimented with and went really badly, though periodically they try something approaching it again. https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(episode) https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(... (AI, human genetic engineering, and a number of other areas of technology are affected by variants of this issue in the Trek canon.)
    - mschuster91 4y ago
      At least in ST:VOY, it has been shown that a single hologram is capable of running the ship - although one might argue "exceptional circumstances" ;)
      - wongarsu 4y ago
        In Star Trek III they jury-rig the Enterprise to fly with a crew of 5, instead of the regular crew of 400. Though in that state it can't do much more than fly: combat capabilities are strongly diminished, maintenance doesn't happen, post-combat repairs are out of the question, science missions would be much harder. On occasion the Enterprise has transported 150 passengers, so I imagine there's a lot of kitchen staff, security, etc. You only need 5 people to fly the ship, maybe 40 to fly sustainably with maintenance, but to actually accomplish their reglar mission you need the other 300 people.
    - krapp 4y ago
      On the one hand, given how often sensors fail and AIs flip the evil bit in Star Trek, limiting automation is probably a good idea. On the other hand, the Enterprise won't even warn anyone when command staff are injured, cloned, mind-controlled or vanish from the ship altogether unless a human asks the computer where a specific person is first. Of course there are Doylist reasons for all of this but I do like the premise of a general fear of AI and possible weird space BS being a factor.
- ynniv 4y ago
  Knight Capital will forever haunt fintech engineers... https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-...
themitigating 4y ago
This employee left the backup system running. There's obviously some automation but what is the solution? Process changes that people have to remember or more systems to prevent the issue. So I don't get your statement related to this article
- nix23 4y ago
  >There's obviously some automation but what is the solution? Shutdown and wake-up time in bios of server and switch ;)
- deleted 4y ago
  [deleted]
- mrkeen 4y ago
  I don't imply that there is a solution. We will simply create a second system (B) to monitor the first system (A). Now we have two systems to maintain. System B will not be capable of steering A by itself. So we still need to know how to diagnose and repair A, and we also need to know about B too. Maybe system B can talk to a Prometheus/Grafana stack (if it's up). And that can put alerts into Slack (which we ignore because there's always alerts in Slack). And after standup we can take turns looking at graphs with consternation. > Stuff like this will happen more and more. We treat software driven systems rather recklessly. That sentence is where I go when I hear the word 'automation'.
nix23 4y ago
>More automation -> more code -> more things to go wrong More people -> more entropy -> much more things to go wrong
- wongarsu 4y ago
  One person might be inattentive or drunk, but it's less likely that two people are. So you institue a two-person rule. And if that's not good enough, add a third person to double-check. Maybe a supervisor to observe the people doing all of the above, to catch any mistakes or negligent behavior. Also have them write down the steps they have taken, and have somebody else read through that to verify. Just keep adding people until you are satisfied with your odds (and hope you are not making it worse through second-order effects)
  - nix23 4y ago
    So two people are more reliable as a automated system you wanna say? That's totally wrong....
    - reaperducer 4y ago
      Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. I did last year, and my company is in the process of de-automating certain processes that can endanger that company if they go wrong. There are many things in tech that are too important to automate. I'd even posit that the more experience you have in tech, the more you've seen how things go wrong, and the more you realize that automation is a tool for humans to use, not a replacement for humans doing a task.
      - nix23 4y ago
        >Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. Much much much more due to human error......but hey maybe you are the worst programmer ever..but even then i would say your programs are more reliable then a human.

More automation -> more code -> more things to go wrong

by mrkeen 4y ago

More automation -> more code -> more things to go wrong

bongobingo1 4y ago

The great thing about automation is the breadth, depth and speed at which I can propagate mistakes.

junon 4y ago
I've not seen it put so eloquently.
- dsr3 4y ago
  To err is human, to really foul things up requires a computer - William E. Vaughan https://quoteinvestigator.com/2010/12/07/foul-computer/ https://quoteinvestigator.com/2010/12/07/foul-computer/
  - erik_seaberg 4y ago
    “A computer lets you make more mistakes faster than any other invention, with the possible exceptions of handguns and tequila.”—Mitch Ratcliffe
WJW 4y ago
Some submarine operations are deliberately not automated, because if eg a sensor is broken or miscalibrated it could sink the entire boat if a computer very rapidly acts on the wrong information. Rather they do those operations with one person operating the valve/machine/device/etc, another watching and confirming readings, and the whole thing is on a constant audio link with a third person in an engineering room who watches the readings through a centralized system. It's clearly not optimized for efficient use of personnel, but the personnel complement will have been designed to provide sufficient people at all times and the cost of getting it wrong can be very large indeed.
- TeMPOraL 4y ago
  This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general).
  - dragonwriter 4y ago
    > This is now my headcanon explanation for why starships in Star Trek still require large crews (or crews in general The canon explanation is that automation-in-charge was experimented with and went really badly, though periodically they try something approaching it again. https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(episode) https://memory-alpha.fandom.com/wiki/The_Ultimate_Computer_(... (AI, human genetic engineering, and a number of other areas of technology are affected by variants of this issue in the Trek canon.)
  - mschuster91 4y ago
    At least in ST:VOY, it has been shown that a single hologram is capable of running the ship - although one might argue "exceptional circumstances" ;)
    - wongarsu 4y ago
      In Star Trek III they jury-rig the Enterprise to fly with a crew of 5, instead of the regular crew of 400. Though in that state it can't do much more than fly: combat capabilities are strongly diminished, maintenance doesn't happen, post-combat repairs are out of the question, science missions would be much harder. On occasion the Enterprise has transported 150 passengers, so I imagine there's a lot of kitchen staff, security, etc. You only need 5 people to fly the ship, maybe 40 to fly sustainably with maintenance, but to actually accomplish their reglar mission you need the other 300 people.
  - krapp 4y ago
    On the one hand, given how often sensors fail and AIs flip the evil bit in Star Trek, limiting automation is probably a good idea. On the other hand, the Enterprise won't even warn anyone when command staff are injured, cloned, mind-controlled or vanish from the ship altogether unless a human asks the computer where a specific person is first. Of course there are Doylist reasons for all of this but I do like the premise of a general fear of AI and possible weird space BS being a factor.
ynniv 4y ago
Knight Capital will forever haunt fintech engineers... https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html https://www.henricodolfing.com/2019/06/project-failure-case-...

themitigating 4y ago

This employee left the backup system running. There's obviously some automation but what is the solution? Process changes that people have to remember or more systems to prevent the issue. So I don't get your statement related to this article

nix23 4y ago
>There's obviously some automation but what is the solution? Shutdown and wake-up time in bios of server and switch ;)
deleted 4y ago
[deleted]
mrkeen 4y ago
I don't imply that there is a solution. We will simply create a second system (B) to monitor the first system (A). Now we have two systems to maintain. System B will not be capable of steering A by itself. So we still need to know how to diagnose and repair A, and we also need to know about B too. Maybe system B can talk to a Prometheus/Grafana stack (if it's up). And that can put alerts into Slack (which we ignore because there's always alerts in Slack). And after standup we can take turns looking at graphs with consternation. > Stuff like this will happen more and more. We treat software driven systems rather recklessly. That sentence is where I go when I hear the word 'automation'.

nix23 4y ago

>More automation -> more code -> more things to go wrong More people -> more entropy -> much more things to go wrong

wongarsu 4y ago
One person might be inattentive or drunk, but it's less likely that two people are. So you institue a two-person rule. And if that's not good enough, add a third person to double-check. Maybe a supervisor to observe the people doing all of the above, to catch any mistakes or negligent behavior. Also have them write down the steps they have taken, and have somebody else read through that to verify. Just keep adding people until you are satisfied with your odds (and hope you are not making it worse through second-order effects)
- nix23 4y ago
  So two people are more reliable as a automated system you wanna say? That's totally wrong....
  - reaperducer 4y ago
    Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. I did last year, and my company is in the process of de-automating certain processes that can endanger that company if they go wrong. There are many things in tech that are too important to automate. I'd even posit that the more experience you have in tech, the more you've seen how things go wrong, and the more you realize that automation is a tool for humans to use, not a replacement for humans doing a task.
    - nix23 4y ago
      >Sounds like you've never experienced a catastrophic failure due to an automation that didn't work right. Much much much more due to human error......but hey maybe you are the worst programmer ever..but even then i would say your programs are more reliable then a human.