4 ms·

Why do you and others on the thread keep asking that. If a human made a mistake he or she is held accountable individually and punished. The software cannot be

by _8j50 3y ago

Why do you and others on the thread keep asking that. If a human made a mistake he or she is held accountable individually and punished. The software cannot be punished, it does not have the ability to fear punishment and be perfectly careful. The point you are missing is that the software isn't augmenting human responsibility, it is largley replacing it. Even if not on paper, humans now have an excuse to lay responsibility on the software. This is the same problem I have with ML and my work, if I have to triple check its output anyways then what good is it? If I don't, who will take responsibility for the mistake? Even if humans made more errors, you can tune humans by adjusting their incentives. For example, if they faced a couple of years of jail time would humans keep the same amount of error rate? The proper use of such software is to narrow down the list of suspects, humans then have to do much more vetting and verification, assuming the software's results are false before concluding the identification. The time saved scouring through a large list of suspects should be spent painstakingly vetting and independently corroborating software identified hits. There is no replacement for humans doing the actual/final identification and assuming the responsibility if their mistakes.

golergka 3y ago
> If a human made a mistake he or she is held accountable individually and punished. Is she?
- _8j50 3y ago
  Yes, in a properly run police department, rare as they may be.
whatshisface 3y ago
>If a human made a mistake he or she is held accountable individually and punished. Really? I am not so sure about that. Who'd get punished for misidentifying a face in book, say, of faces?
- BaseballPhysics 3y ago
  Fine, can be held accountable. Seriously, are you intentionally trying to miss the point in order to nitpick on this minor detail?
  - whatshisface 3y ago
    Engineers can be held accountable for a failure in a system they said would work, and it happens when something goes wrong with a bridge or building. I do not agree with the point. It is like saying, "A condominium in Florida cannot be held accountable." :-)
    - BaseballPhysics 3y ago
      You are entirely missing the point. We're talking about systems, algorithms, black box models that we don't understand or have the ability to reason about, that are being used to make decisions for us. Those systems are already proven to have embedded biases. But now instead of a human we can question, we only have inscrutable machines that no one, not even the people who built them, understand. If condominiums in Florida were making the decision about who could get a mortgage and live in them, then your analogy would make sense. But they don't. There are, on the other hand, computer systems that absolutely do, and it's very possible (I would predict probable) that they will end up effectively re-creating redlining. Furthermore, going back to your broken analogy, unlike the case with building codes and engineering professional organizations, there are no laws or regulations that could be used to hold accountable those who build or use those systems.
    - _8j50 3y ago
      You can inspect the design of a condo but ML models can't be examined, their training data is the only thing you can examine. It's like being able to inspect the raw materials but nothing else for the condo.
      - deleted 3y ago
        [deleted]
tbrownaw 3y ago
Maybe try reading up on the reasoning behind the whole "blameless retrospective" thing that's popular to advocate for when software development discussions pop up around here?
- eropple 3y ago
  Blameless retrospectives only make sense when everyone is actually incentivized towards quality of output. That is historically true of neither (American) police organizations nor the vendors who sell to them.
BaseballPhysics 3y ago
> Even if not on paper, humans now have an excuse to lay responsibility on the software. This, right here, is the real key and I hope people really fully appreciate this. We can now have machines that can be biased for us, and when that bias becomes apparent, people can shrug their shoulders and say "what can you do, it's the machine's fault". That is a very dangerous road, and we're already well on our way down that path.
- LocalH 3y ago
  It honestly seems to me that the core issue here is actually the same one found in digital voting technology - where is the vetting? There must be vetting at multiple levels of the process, or these use cases are negligent at best and malicious at worst. "Our highly advanced computer technology has analyzed the voting results, and thrown out all of the results it has determined to be fraudulent. If your vote was thrown out due to the algorithm's fraud determination, you will be arrested for attempting to subvert democracy. What's that? You want to double check the computer? That's not necessary, because it's a computer after all, it doesn't make mistakes." "Our highly advanced computer technology said with 100% certainty that you are the person responsible for this murder. You say you were on the other side of town at that time, and you claim to have witnesses. This is impossible, because our computer algorithm doesn't make mistakes, it's a computer after all. You are hereby sentenced to the death penalty, and the computer has recommended that you receive no appeal, since it has determined you to be guilty beyond any doubt."
  - BaseballPhysics 3y ago
    > There must be vetting at multiple levels of the process, or these use cases are negligent at best and malicious at worst. You've got it exactly. The problem is modern AI systems cannot be vetted because neither their creators nor their users actually understand how they work (yes, the mechanics are understood, but the emergent properties are not). I believe we need regulations that prevent the use of unauditable technologies in a bunch of key industries, including banking, law enforcement, healthcare, etc. Basically anywhere where a person's rights may be violated by these systems. We actually fought this same battle not too long ago with photo radar systems due to issues with their accuracy, and the result was mandated transparency (at least in some jurisdictions). These new technologies just supercharge the problem.

by _8j50 3y ago

golergka 3y ago

> If a human made a mistake he or she is held accountable individually and punished. Is she?

_8j50 3y ago
Yes, in a properly run police department, rare as they may be.

whatshisface 3y ago

>If a human made a mistake he or she is held accountable individually and punished. Really? I am not so sure about that. Who'd get punished for misidentifying a face in book, say, of faces?

BaseballPhysics 3y ago
Fine, can be held accountable. Seriously, are you intentionally trying to miss the point in order to nitpick on this minor detail?
- whatshisface 3y ago
  Engineers can be held accountable for a failure in a system they said would work, and it happens when something goes wrong with a bridge or building. I do not agree with the point. It is like saying, "A condominium in Florida cannot be held accountable." :-)
  - BaseballPhysics 3y ago
    You are entirely missing the point. We're talking about systems, algorithms, black box models that we don't understand or have the ability to reason about, that are being used to make decisions for us. Those systems are already proven to have embedded biases. But now instead of a human we can question, we only have inscrutable machines that no one, not even the people who built them, understand. If condominiums in Florida were making the decision about who could get a mortgage and live in them, then your analogy would make sense. But they don't. There are, on the other hand, computer systems that absolutely do, and it's very possible (I would predict probable) that they will end up effectively re-creating redlining. Furthermore, going back to your broken analogy, unlike the case with building codes and engineering professional organizations, there are no laws or regulations that could be used to hold accountable those who build or use those systems.
  - _8j50 3y ago
    You can inspect the design of a condo but ML models can't be examined, their training data is the only thing you can examine. It's like being able to inspect the raw materials but nothing else for the condo.
    - deleted 3y ago
      [deleted]

tbrownaw 3y ago

Maybe try reading up on the reasoning behind the whole "blameless retrospective" thing that's popular to advocate for when software development discussions pop up around here?

eropple 3y ago
Blameless retrospectives only make sense when everyone is actually incentivized towards quality of output. That is historically true of neither (American) police organizations nor the vendors who sell to them.

BaseballPhysics 3y ago

> Even if not on paper, humans now have an excuse to lay responsibility on the software. This, right here, is the real key and I hope people really fully appreciate this. We can now have machines that can be biased for us, and when that bias becomes apparent, people can shrug their shoulders and say "what can you do, it's the machine's fault". That is a very dangerous road, and we're already well on our way down that path.

LocalH 3y ago
It honestly seems to me that the core issue here is actually the same one found in digital voting technology - where is the vetting? There must be vetting at multiple levels of the process, or these use cases are negligent at best and malicious at worst. "Our highly advanced computer technology has analyzed the voting results, and thrown out all of the results it has determined to be fraudulent. If your vote was thrown out due to the algorithm's fraud determination, you will be arrested for attempting to subvert democracy. What's that? You want to double check the computer? That's not necessary, because it's a computer after all, it doesn't make mistakes." "Our highly advanced computer technology said with 100% certainty that you are the person responsible for this murder. You say you were on the other side of town at that time, and you claim to have witnesses. This is impossible, because our computer algorithm doesn't make mistakes, it's a computer after all. You are hereby sentenced to the death penalty, and the computer has recommended that you receive no appeal, since it has determined you to be guilty beyond any doubt."
- BaseballPhysics 3y ago
  > There must be vetting at multiple levels of the process, or these use cases are negligent at best and malicious at worst. You've got it exactly. The problem is modern AI systems cannot be vetted because neither their creators nor their users actually understand how they work (yes, the mechanics are understood, but the emergent properties are not). I believe we need regulations that prevent the use of unauditable technologies in a bunch of key industries, including banking, law enforcement, healthcare, etc. Basically anywhere where a person's rights may be violated by these systems. We actually fought this same battle not too long ago with photo radar systems due to issues with their accuracy, and the result was mandated transparency (at least in some jurisdictions). These new technologies just supercharge the problem.