6 ms·

When i read these implementations, one thing I think about is the British post office scandal, also known as Horizon IT scandal[1]. These technologies are prob

When i read these implementations, one thing I think about is the British post office scandal, also known as Horizon IT scandal[1]. These technologies are probabilistic with some nonzero chances of error which can't always be easily detected. These are implemented by people with some knowledge of the tech used (and hopefully the understanding of error), but are to be used by people who expect the output to be 100% true. I do not know how they would deal with tech errors which do not look like errors, but feel like there should be more cognizance about this and expectations should be set accordingly. Same goes for any facial recognition tech deployed by authorities. [1]: https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal

pompino 2y ago
how does it compare versus human error?
- lolinder 2y ago
  What the exact error rate is isn't relevant to OP's concern, which is that operators of the tooling will treat it as having a 0% error rate in a way that they don't with human judgment. We've already seen this play out with algorithms designed to predict recidivism or what have you—law enforcement and judicial officials place undue weight on the outputs of algorithms and trust them implicitly where they might ask questions of a human expert.
  - acdha 2y ago
    > which is that operators of the tooling will treat it as having a 0% error rate in a way that they don't with human judgment This is possible but we have decades of camera enforcement experience suggesting that courts are aware that these systems are not as reliable as the manufacturers claim. People have been challenging photo tickets since the previous century and there’s a well-established process for challenging them.
- antegamisou 2y ago
  Techbros always love citing the supposedly devastating human fallibility yet experience shows poorly configured/developed automated systems are much more of a concern.
  - pompino 2y ago
    Nice straw-man, but getting back to reality, nobody in this thread has put forth the belief that automated systems are infallible, or is arguing that point. Also the fact that humans are fallible, hasn't stopped us from setting up judicial systems based on human judgement - which work pretty well in several scenarios.
    - antegamisou 2y ago
      sTrAwmAn
- ankit219 2y ago
  I am not comparing it to a human error. There is a conventional wisdom that humans make mistakes, and we have redundancy built into various systems to counter the same. With machines and tech, the expectation is that they are accurate (which is not true, but mostly true in the important ways). The reason I brought it up is to hope we have that redundancy in this case and future implementations of AI as well. (Horizon had a bug in calculation system - which seems easy and straightforward and hence unexpected and undetected.) A typical example would be breath analysers. If humans were to judge who is drunk, there is a high error rate. With tech, the error rate is low, but if there is a bug in one of the equipments, many wrong and unnecessary arrests might be made.
  - acdha 2y ago
    > With machines and tech, the expectation is that they are accurate (which is not true, but mostly true in the important ways). I find this an interesting comment with your choice of examples as there is an extensive legal history about false positives (medical conditions, mouthwash, etc.) and the need to show maintenance and calibration records. I was on a jury back in the 2000s where the San Diego police had to have someone from the crime lab testify about their process for managing their breathalyzers, and we ended up unanimously finding the guy not guilty on that charge because they’d messed up the process and had been a year late on calibration testing despite everyone agreeing he was drunk (based on his own testimony, receipts, and the bar staff, he’d had 3 double vodkas in the hour before getting behind the wheel, so we found him guilty on the other prong of California’s DUI statute - which is exactly what his attorney told us afterwards had been his prediction).
    - ankit219 2y ago
      Your experience is very helpful. I took a theoretical example since where I live breath analysers are how you get fined for drunk driving. I didn't think it was possible to contest that. I would rather that these things happen preemptively so that there are no such challenges to fight or tickets to contest just because an AI flagged something and humans went ahead with it. I mean that other tech is deterministic, while AI is probabilistic, but people still havent updated their models for such tech. And we need to keep those examples in mind.
rvnx 2y ago
Not a problem, if person contests, double-check the picture. Like with a speed camera. For most of the cases, won’t need a human in the loop.
- squigz 2y ago
  One might argue that a citizen shouldn't need to prove they didn't break a law, but rather the government has to (reliably) prove that they did. I wonder if there's a phrase for that.
  - simmerup 2y ago
    That’s what the picture is for
  - hackernewds 2y ago
    False positives can be human validated
    - eli 2y ago
      false positives can also be human generated
  - yardie 2y ago
    I've contested the red light camera fines in my municipality a few times now. In every case I request the footage and each time it's been overturned because the footage clearly shows me stopping before turning right on read. Ive never had to provide proof that is the states job.
    - lolinder 2y ago
      Meanwhile, I've never had to contest a false positive because my state doesn't allow red light cameras. That's a lot of time that I haven't had to spend defending myself against spurious charges. It doesn't especially matter where the burden of proof lies if the process for contesting requires any substantial amount of effort on the part of the victim of the false positive.
ensignavenger 2y ago
Which is why a human being has to review the camera footage and decide to issue the citation. If the system makes an error, and the human checking it makes an error, then the human receiving the citation can contest it. Seems a reasonable number of safeguards to me. Certainly a lot better than the big tech "we banned you but won't tell you why or allow you to talk to a human about it". My only problem with more camera-based law enforcement for traffic violations is the fact that they can't tell who was driving. SO the fine goes to the registered owner of the vehicle, so folks won't want to lend vehicles. Of course, one argument is that you shouldn't loan your vehicle to some one who is going to be breaking the law... but is that really the world we want to live in, where we can't loan anything to anyone for fear of them misusing it and the owner being held responsible? I don't have a good answer there.
- Chris2048 2y ago
  > one argument is that you shouldn't loan your vehicle to some one who is going to be breaking the law... but is that really the world we want to live in If you are talking about a car, then yes, that's the world I want to live in. Would they even be insured anyway?
  - ensignavenger 2y ago
    You can get insurance that will cover you when driving some one elses car, and you can get insurance that covers some one else driving your car. Wether either of you have such insurance, is a good question to ask before lending your car to some one.
    - Kirby64 2y ago
      By default most insurance policies (in the US, at least) cover incidental occasional use of drivers you permit to use your car that don’t actually live with you. They have to be licensed to drive, but that’s essentially the only restriction.
      - Chris2048 2y ago
hi-v-rocknroll 2y ago
Neither Kafka, Joseph Heller, nor Huxley could conceive of technology-enabled ubiquitous, concealed brutality and extortion. The Western world is rapidly flirting with the specters of pre-crime and automated enforcement that puts the burden on the accused to prove their innocence with Hobson choices while extorting fees from average citizens as a profit-center. And in Western countries with weak privacy laws, this class of technology also raises the specter of private "social credit" behavioral tracking to discriminates against and quantize people by grading their online and offline behaviors in totally opaque manners.

pompino 2y ago

how does it compare versus human error?

lolinder 2y ago
What the exact error rate is isn't relevant to OP's concern, which is that operators of the tooling will treat it as having a 0% error rate in a way that they don't with human judgment. We've already seen this play out with algorithms designed to predict recidivism or what have you—law enforcement and judicial officials place undue weight on the outputs of algorithms and trust them implicitly where they might ask questions of a human expert.
- acdha 2y ago
  > which is that operators of the tooling will treat it as having a 0% error rate in a way that they don't with human judgment This is possible but we have decades of camera enforcement experience suggesting that courts are aware that these systems are not as reliable as the manufacturers claim. People have been challenging photo tickets since the previous century and there’s a well-established process for challenging them.
antegamisou 2y ago
Techbros always love citing the supposedly devastating human fallibility yet experience shows poorly configured/developed automated systems are much more of a concern.
- pompino 2y ago
  Nice straw-man, but getting back to reality, nobody in this thread has put forth the belief that automated systems are infallible, or is arguing that point. Also the fact that humans are fallible, hasn't stopped us from setting up judicial systems based on human judgement - which work pretty well in several scenarios.
  - antegamisou 2y ago
    sTrAwmAn
ankit219 2y ago
I am not comparing it to a human error. There is a conventional wisdom that humans make mistakes, and we have redundancy built into various systems to counter the same. With machines and tech, the expectation is that they are accurate (which is not true, but mostly true in the important ways). The reason I brought it up is to hope we have that redundancy in this case and future implementations of AI as well. (Horizon had a bug in calculation system - which seems easy and straightforward and hence unexpected and undetected.) A typical example would be breath analysers. If humans were to judge who is drunk, there is a high error rate. With tech, the error rate is low, but if there is a bug in one of the equipments, many wrong and unnecessary arrests might be made.
- acdha 2y ago
  > With machines and tech, the expectation is that they are accurate (which is not true, but mostly true in the important ways). I find this an interesting comment with your choice of examples as there is an extensive legal history about false positives (medical conditions, mouthwash, etc.) and the need to show maintenance and calibration records. I was on a jury back in the 2000s where the San Diego police had to have someone from the crime lab testify about their process for managing their breathalyzers, and we ended up unanimously finding the guy not guilty on that charge because they’d messed up the process and had been a year late on calibration testing despite everyone agreeing he was drunk (based on his own testimony, receipts, and the bar staff, he’d had 3 double vodkas in the hour before getting behind the wheel, so we found him guilty on the other prong of California’s DUI statute - which is exactly what his attorney told us afterwards had been his prediction).
  - ankit219 2y ago
    Your experience is very helpful. I took a theoretical example since where I live breath analysers are how you get fined for drunk driving. I didn't think it was possible to contest that. I would rather that these things happen preemptively so that there are no such challenges to fight or tickets to contest just because an AI flagged something and humans went ahead with it. I mean that other tech is deterministic, while AI is probabilistic, but people still havent updated their models for such tech. And we need to keep those examples in mind.

rvnx 2y ago

Not a problem, if person contests, double-check the picture. Like with a speed camera. For most of the cases, won’t need a human in the loop.

squigz 2y ago
One might argue that a citizen shouldn't need to prove they didn't break a law, but rather the government has to (reliably) prove that they did. I wonder if there's a phrase for that.
- simmerup 2y ago
  That’s what the picture is for
- hackernewds 2y ago
  False positives can be human validated
  - eli 2y ago
    false positives can also be human generated
- yardie 2y ago
  I've contested the red light camera fines in my municipality a few times now. In every case I request the footage and each time it's been overturned because the footage clearly shows me stopping before turning right on read. Ive never had to provide proof that is the states job.
  - lolinder 2y ago
    Meanwhile, I've never had to contest a false positive because my state doesn't allow red light cameras. That's a lot of time that I haven't had to spend defending myself against spurious charges. It doesn't especially matter where the burden of proof lies if the process for contesting requires any substantial amount of effort on the part of the victim of the false positive.

ensignavenger 2y ago

Which is why a human being has to review the camera footage and decide to issue the citation. If the system makes an error, and the human checking it makes an error, then the human receiving the citation can contest it. Seems a reasonable number of safeguards to me. Certainly a lot better than the big tech "we banned you but won't tell you why or allow you to talk to a human about it". My only problem with more camera-based law enforcement for traffic violations is the fact that they can't tell who was driving. SO the fine goes to the registered owner of the vehicle, so folks won't want to lend vehicles. Of course, one argument is that you shouldn't loan your vehicle to some one who is going to be breaking the law... but is that really the world we want to live in, where we can't loan anything to anyone for fear of them misusing it and the owner being held responsible? I don't have a good answer there.

Chris2048 2y ago
> one argument is that you shouldn't loan your vehicle to some one who is going to be breaking the law... but is that really the world we want to live in If you are talking about a car, then yes, that's the world I want to live in. Would they even be insured anyway?
- ensignavenger 2y ago
  You can get insurance that will cover you when driving some one elses car, and you can get insurance that covers some one else driving your car. Wether either of you have such insurance, is a good question to ask before lending your car to some one.
  - Kirby64 2y ago
    By default most insurance policies (in the US, at least) cover incidental occasional use of drivers you permit to use your car that don’t actually live with you. They have to be licensed to drive, but that’s essentially the only restriction.
    - Chris2048 2y ago

hi-v-rocknroll 2y ago

Neither Kafka, Joseph Heller, nor Huxley could conceive of technology-enabled ubiquitous, concealed brutality and extortion. The Western world is rapidly flirting with the specters of pre-crime and automated enforcement that puts the burden on the accused to prove their innocence with Hobson choices while extorting fees from average citizens as a profit-center. And in Western countries with weak privacy laws, this class of technology also raises the specter of private "social credit" behavioral tracking to discriminates against and quantize people by grading their online and offline behaviors in totally opaque manners.