4 ms·

I confess this line always upset me. It is cute, but it directly points to the idea that the main recourse for a mistake is to take umbridge with an individual

by taeric 2y ago

I confess this line always upset me. It is cute, but it directly points to the idea that the main recourse for a mistake is to take umbridge with an individual that must have obviously been wrong. No. If a mistake is made and it impacts people, take action to make the impacted people whole and change the system so that a similar mistake won't be made again. If you want, you can argue that the system can be held accountable and changed. Further, if there is evidence of a bad actor that is purposely making choices that hurt people. Take action on that. But accountability of actors in the system is almost certainly immune by policy. And for good reason.

miltonlost 2y ago
A system!!! Held accountable!!!! A system, just like a computer, cannot be held accountable for the reason that a system, as like a computer, is not alive and cannot actual be held accountable in a way that the system or computer cares. But what is a system made of? People who are doing bad decisions and should be held accountable for that. Without accountability of bad actors in systems, you get companies committing crimes because no one at the top rarely sees fines or jail time. The same immunity from responsibility you think is a good thing in a system is what I would say is corporate america’s major sin. You’re upset at the line because you make a fundamental misunderstanding of what it means for someone to be held accountable for something.
- coliveira 2y ago
  This hits the nail on the head. The issue is that humans are accountable, but systems are not. And smart humans learned how to hide behind systems to avoid accountability. That's the whole strategy of using corporations, a social structure that removes individuals from responsibility. A corporation can do pretty much anything, including criminal acts, and the humans benefiting from it are shield from the negative results except for financial losses. What we're seeing is just the whole strategy moving into the level of computer systems (and Google has already used this accountability skirting strategy for more than two decades).
  - taeric 2y ago
    The problem you will quickly run into is individuals cannot take the load that this line of thinking leads to. Accidentally typo an extra 0 on a payment process? Hope you have the funds to take accountability on that mistake. After all, we have taken the choice that the system cannot be operated to rollback any choice, as that would lower human accountability...
    - the8472 2y ago
      Then that person should have the right to refuse, ask for appropriate compensation or insurance. It's often argued here that the C-suite earns their high compensation by taking on those risks. So let the CFO fatfinger those unchecked transfers if he wishes. But if rollbacks are possible and cheap then we need all that much accountability. People want accountability when there's no rollback possible, e.g. when decisions leads to deaths or life-years wasted.
      - taeric 2y ago
        Your last sentence is basically my disagreement and push, here? Systems can be made so that they can be rolled back. More, systems that oversee life and death situations often have to act faster than a human can respond to them. I'm all for trying to get personal accountability to lead to a personal liability as much as we want. That, just isn't going to happen. So, if you want to change the manual page in question to be that systems should be reversible or "fixable", for whatever that means, sure. But "management decisions" are usually among the most reversible choices any company will ever make.
  - EtCepeyd 2y ago
    > smart humans learned how to hide behind systems to avoid accountability And "systems" is more general in this sense than just technology. Consider stoning, and firing squads. Both were invented to remove individual responsibility.
- taeric 2y ago
  But your reading presumes that you would hold individuals responsible in a company for something that goes wrong. Which, without showing intent on individual actors, feels very very unlikely. Both in actuality, and in desire. Consider, if it is found that salmonella contaminated the spinach of a local farm, I want it recalled and for better systems in place to catch contamination. I don't want to find the farmer that was responsible for the acre of land that introduced the contaminant. I think this idea flows from some thought that people will put more effort into things that they think they could be held accountable for. In reality, that just isn't the case. People will instead stall and stall on things they think they could be held accountable on if things go south. From my vantage, that style of "accountability" just leads to more and more red tape as you do what you can to line item identify who could have prevented something. That is not productive.
  - miltonlost 2y ago
    > Consider, if it is found that salmonella contaminated the spinach of a local farm, I want it recalled and for better systems in place to catch contamination. I don't want to find the farmer that was responsible for the acre of land that introduced the contaminant. Wow a local farm!!!! We must think of the plight of the local farmer! And not the multinational corporations! Consider, if it is found that listeria contaminated the meat of a national Chain, Boar's Head, I want it recalled and for better systems in place to catch contamination. I also want the plant manager and executives who allowed for the massively unsanitary state to continue for years. https://arstechnica.com/science/2024/09/10th-person-dead-in-listeria-outbreak-linked-to-boars-head-meats/ https://arstechnica.com/science/2024/09/10th-person-dead-in-... The way you're talking , we should be going back to pre-Upton Sinclair The Jungle and letting our food run full of contaminants because why make any one person accountable for their willful addition of sawdust to their flour?
    - taeric 2y ago
      If there is provable negligence on the plant manager for allowing it, I'm game for them getting in trouble. My point was more that there are a TON of individuals that can each be shown to have done something specific to spread the problem. We don't hold them accountable. By design. Now, I'm all for more directly holding companies responsible. Such that I think they probably deserve less protections than they almost certainly have here. But that is a different thing and, again, is unrelated to "a person taking accountability."
Handprint4469 2y ago
I disagree. The main point of this line is not about what to do _after_ a mistake (assign blame, punish, etc), but rather about setting up the correct incentives _before_ anything happens so that a mistake is less likely. When you're accountable you suddenly have skin in the game, so you'll be more careful about whatever you're doing.
- taeric 2y ago
  Right, I guessed this is what people had in mind. I'll note that this line of thinking typically doesn't get better results. It largely just gets more "red tape" so that you have to get people to sign off on things. And the person that shows up to do something will have all of their red tape in order so that they are not responsible for any damages that result from carrying out their job. Agreed that personal responsibility is important and people should strive to it more. Disagree that accountability is the same thing, or that you can implement it by policy. Still more strongly disagreed that you should look for a technical solution to what is largely not a technical problem.
medhir 2y ago
so if someone makes a change to the system… there’s a person somewhere holding themselves accountable for the faults of the system, no?
- coliveira 2y ago
  No, if there are multiple people who in principle are not directly coordinated to make that happen. They can always point the finger at others and say they're not responsible for that bad outcome.
  - taeric 2y ago
    Exactly. And this is a direct consequence of trying to pin things on individuals.
landryraccoon 2y ago
Fwiw I agree with you. I feel that in general people obsess over assigning blame to the detriment of actually correcting the situation. Take the example of punishing crimes. If we don’t punish theft, we’ll get more theft right? But what do you do when you have harsh penalties for crime, but crime keeps happening? Do you accept crime as immutable, or actually begin to address root causes to try to reduce crime systemically? Punishment is only one tool in a toolbox for correcting bad behavior. I am dismayed that people are fearful enough of the loss of this single tool as to want to architect our entire society around making sure it is available. With AI we have a chance to chart a different course. If a machine makes a mistake, the priority can and should be fixing the error in that machine so the same mistake can never happen again. In this way, fixing an AI can be more reliable than trying to punish human beings ever could.
- taeric 2y ago
  I'm always hesitant to enter "punishing crimes" discussions on this one. Those, by definition, establish intent to commit the crime in a majority of cases. As such, they would almost certainly hit some "accountability" even if they were in a company. Heck, even qualified immunity for government actors typically falls on that. That said, I do think we are in alignment here. Punitive actions are but a tool. I don't think it should be tossed out. But I also suspect it is one of the lesser effective tools we have.
michael1999 2y ago
You have to remember what "calling to account" is. It is a demand that you explain yourself. In the case of a business venture, it means to present the books and detail the entries. A court, congress, or your boss can demand your presence at a meeting to "explain yourself". Accountability doesn't mean punishment, it means you are subject to demands to make an "account" of your self. Punishment is a separate thing from the account. If your account implicates yourself in malfeasance, you might be punished. But that's good!. But there are other kinds of accountability. The FAA is very clear that you must make an accounting for yourself. But the FAA is also very clear that they won't punish you for your account. That doesn't mean you aren't accountable! Most computer systems can not do this! They can not describe the inputs to a decision, the criteria used, the history of policy that led to the current configuration, or the reliability of the data. That's why lawsuits always have to have technical witnesses to do these things. And why the UK Postal scandal was so cruel. Systems that grant actors immunity from accountability as a matter of policy are terrible systems that produce terrible results. Witness US prosecutorial immunity.
- taeric 2y ago
  But, in that regard, systems can be made such that they can, in fact, be held accountable? You can design them such that they can list their inputs and why the outputs were set to what they are. With the speed at which systems operate today, it is actually expected for many systems that they can operate before a person does anything, and that they do so. The world is rife with examples where humans overrode a safety system to their peril. (I can build a small list, if needed.) This doesn't mean we haven't had safety systems mess up. But nor does it mean that we should not make more safety systems.
  - michael1999 2y ago
    Yes. Some systems can be made more accountable. They can provide traceability from their inputs to output, provide reflexive access to source code they run, and provide evidentiary traces for reliability. Safety critical systems that operate faster than human reactions are not accountable. So that's why we never make them responsible. So who is? Same as for bridges that fall down -- the engineers. People forget that civil engineers sign up for accountability that could lead to serious civil or even criminal liability. Which is exactly the point of this aphorism. Boeing was facing criminal charges, and is currently under a consent decree for exactly this kind of sloppy systems work.
_Algernon_ 2y ago
This only works if the party that is wronged and the party doing it are roughly equal in power. For an individual being wronged by eg. Google's algorithms, the path to getting recompense and changing the system is non-existent.
gsuuon 2y ago
This makes sense, effective post-mortems don't focus on assigning blame. They try to identify and fix the problem. The issue with AI is that it's such a black-box right now, this process is likely not feasible. If an AI makes a decision that turns out to be wrong, there's no reliable way to identify and fix the problem. You can prompt engineer or finetune or re-train the model, but in the end you can only hope that issue has been fixed. I think the black-box nature of AI makes it different from other systems we use.

by taeric 2y ago

miltonlost 2y ago

A system!!! Held accountable!!!! A system, just like a computer, cannot be held accountable for the reason that a system, as like a computer, is not alive and cannot actual be held accountable in a way that the system or computer cares. But what is a system made of? People who are doing bad decisions and should be held accountable for that. Without accountability of bad actors in systems, you get companies committing crimes because no one at the top rarely sees fines or jail time. The same immunity from responsibility you think is a good thing in a system is what I would say is corporate america’s major sin. You’re upset at the line because you make a fundamental misunderstanding of what it means for someone to be held accountable for something.

coliveira 2y ago
This hits the nail on the head. The issue is that humans are accountable, but systems are not. And smart humans learned how to hide behind systems to avoid accountability. That's the whole strategy of using corporations, a social structure that removes individuals from responsibility. A corporation can do pretty much anything, including criminal acts, and the humans benefiting from it are shield from the negative results except for financial losses. What we're seeing is just the whole strategy moving into the level of computer systems (and Google has already used this accountability skirting strategy for more than two decades).
- taeric 2y ago
  The problem you will quickly run into is individuals cannot take the load that this line of thinking leads to. Accidentally typo an extra 0 on a payment process? Hope you have the funds to take accountability on that mistake. After all, we have taken the choice that the system cannot be operated to rollback any choice, as that would lower human accountability...
  - the8472 2y ago
    Then that person should have the right to refuse, ask for appropriate compensation or insurance. It's often argued here that the C-suite earns their high compensation by taking on those risks. So let the CFO fatfinger those unchecked transfers if he wishes. But if rollbacks are possible and cheap then we need all that much accountability. People want accountability when there's no rollback possible, e.g. when decisions leads to deaths or life-years wasted.
    - taeric 2y ago
      Your last sentence is basically my disagreement and push, here? Systems can be made so that they can be rolled back. More, systems that oversee life and death situations often have to act faster than a human can respond to them. I'm all for trying to get personal accountability to lead to a personal liability as much as we want. That, just isn't going to happen. So, if you want to change the manual page in question to be that systems should be reversible or "fixable", for whatever that means, sure. But "management decisions" are usually among the most reversible choices any company will ever make.
- EtCepeyd 2y ago
  > smart humans learned how to hide behind systems to avoid accountability And "systems" is more general in this sense than just technology. Consider stoning, and firing squads. Both were invented to remove individual responsibility.
taeric 2y ago
But your reading presumes that you would hold individuals responsible in a company for something that goes wrong. Which, without showing intent on individual actors, feels very very unlikely. Both in actuality, and in desire. Consider, if it is found that salmonella contaminated the spinach of a local farm, I want it recalled and for better systems in place to catch contamination. I don't want to find the farmer that was responsible for the acre of land that introduced the contaminant. I think this idea flows from some thought that people will put more effort into things that they think they could be held accountable for. In reality, that just isn't the case. People will instead stall and stall on things they think they could be held accountable on if things go south. From my vantage, that style of "accountability" just leads to more and more red tape as you do what you can to line item identify who could have prevented something. That is not productive.
- miltonlost 2y ago
  > Consider, if it is found that salmonella contaminated the spinach of a local farm, I want it recalled and for better systems in place to catch contamination. I don't want to find the farmer that was responsible for the acre of land that introduced the contaminant. Wow a local farm!!!! We must think of the plight of the local farmer! And not the multinational corporations! Consider, if it is found that listeria contaminated the meat of a national Chain, Boar's Head, I want it recalled and for better systems in place to catch contamination. I also want the plant manager and executives who allowed for the massively unsanitary state to continue for years. https://arstechnica.com/science/2024/09/10th-person-dead-in-listeria-outbreak-linked-to-boars-head-meats/ https://arstechnica.com/science/2024/09/10th-person-dead-in-... The way you're talking , we should be going back to pre-Upton Sinclair The Jungle and letting our food run full of contaminants because why make any one person accountable for their willful addition of sawdust to their flour?
  - taeric 2y ago
    If there is provable negligence on the plant manager for allowing it, I'm game for them getting in trouble. My point was more that there are a TON of individuals that can each be shown to have done something specific to spread the problem. We don't hold them accountable. By design. Now, I'm all for more directly holding companies responsible. Such that I think they probably deserve less protections than they almost certainly have here. But that is a different thing and, again, is unrelated to "a person taking accountability."

Handprint4469 2y ago

I disagree. The main point of this line is not about what to do _after_ a mistake (assign blame, punish, etc), but rather about setting up the correct incentives _before_ anything happens so that a mistake is less likely. When you're accountable you suddenly have skin in the game, so you'll be more careful about whatever you're doing.

taeric 2y ago
Right, I guessed this is what people had in mind. I'll note that this line of thinking typically doesn't get better results. It largely just gets more "red tape" so that you have to get people to sign off on things. And the person that shows up to do something will have all of their red tape in order so that they are not responsible for any damages that result from carrying out their job. Agreed that personal responsibility is important and people should strive to it more. Disagree that accountability is the same thing, or that you can implement it by policy. Still more strongly disagreed that you should look for a technical solution to what is largely not a technical problem.

medhir 2y ago

so if someone makes a change to the system… there’s a person somewhere holding themselves accountable for the faults of the system, no?

coliveira 2y ago
No, if there are multiple people who in principle are not directly coordinated to make that happen. They can always point the finger at others and say they're not responsible for that bad outcome.
- taeric 2y ago
  Exactly. And this is a direct consequence of trying to pin things on individuals.

landryraccoon 2y ago

Fwiw I agree with you. I feel that in general people obsess over assigning blame to the detriment of actually correcting the situation. Take the example of punishing crimes. If we don’t punish theft, we’ll get more theft right? But what do you do when you have harsh penalties for crime, but crime keeps happening? Do you accept crime as immutable, or actually begin to address root causes to try to reduce crime systemically? Punishment is only one tool in a toolbox for correcting bad behavior. I am dismayed that people are fearful enough of the loss of this single tool as to want to architect our entire society around making sure it is available. With AI we have a chance to chart a different course. If a machine makes a mistake, the priority can and should be fixing the error in that machine so the same mistake can never happen again. In this way, fixing an AI can be more reliable than trying to punish human beings ever could.

taeric 2y ago
I'm always hesitant to enter "punishing crimes" discussions on this one. Those, by definition, establish intent to commit the crime in a majority of cases. As such, they would almost certainly hit some "accountability" even if they were in a company. Heck, even qualified immunity for government actors typically falls on that. That said, I do think we are in alignment here. Punitive actions are but a tool. I don't think it should be tossed out. But I also suspect it is one of the lesser effective tools we have.

michael1999 2y ago

You have to remember what "calling to account" is. It is a demand that you explain yourself. In the case of a business venture, it means to present the books and detail the entries. A court, congress, or your boss can demand your presence at a meeting to "explain yourself". Accountability doesn't mean punishment, it means you are subject to demands to make an "account" of your self. Punishment is a separate thing from the account. If your account implicates yourself in malfeasance, you might be punished. But that's good!. But there are other kinds of accountability. The FAA is very clear that you must make an accounting for yourself. But the FAA is also very clear that they won't punish you for your account. That doesn't mean you aren't accountable! Most computer systems can not do this! They can not describe the inputs to a decision, the criteria used, the history of policy that led to the current configuration, or the reliability of the data. That's why lawsuits always have to have technical witnesses to do these things. And why the UK Postal scandal was so cruel. Systems that grant actors immunity from accountability as a matter of policy are terrible systems that produce terrible results. Witness US prosecutorial immunity.

taeric 2y ago
But, in that regard, systems can be made such that they can, in fact, be held accountable? You can design them such that they can list their inputs and why the outputs were set to what they are. With the speed at which systems operate today, it is actually expected for many systems that they can operate before a person does anything, and that they do so. The world is rife with examples where humans overrode a safety system to their peril. (I can build a small list, if needed.) This doesn't mean we haven't had safety systems mess up. But nor does it mean that we should not make more safety systems.
- michael1999 2y ago
  Yes. Some systems can be made more accountable. They can provide traceability from their inputs to output, provide reflexive access to source code they run, and provide evidentiary traces for reliability. Safety critical systems that operate faster than human reactions are not accountable. So that's why we never make them responsible. So who is? Same as for bridges that fall down -- the engineers. People forget that civil engineers sign up for accountability that could lead to serious civil or even criminal liability. Which is exactly the point of this aphorism. Boeing was facing criminal charges, and is currently under a consent decree for exactly this kind of sloppy systems work.

_Algernon_ 2y ago

This only works if the party that is wronged and the party doing it are roughly equal in power. For an individual being wronged by eg. Google's algorithms, the path to getting recompense and changing the system is non-existent.

gsuuon 2y ago

This makes sense, effective post-mortems don't focus on assigning blame. They try to identify and fix the problem. The issue with AI is that it's such a black-box right now, this process is likely not feasible. If an AI makes a decision that turns out to be wrong, there's no reliable way to identify and fix the problem. You can prompt engineer or finetune or re-train the model, but in the end you can only hope that issue has been fixed. I think the black-box nature of AI makes it different from other systems we use.