5 ms·

Seems like an opportunity for the lawyers to me. Being evicted from a store is necessarily a public act. Telling someone wrongly that they are a thief, in publ

• Latty 2y ago
  When I worked on analysis software for police & intelligence agencies, it quickly became clear that no matter how much you present something with clear "this is a heuristic, it can give you leads", a scary number of people will take it as "this will tell you whodunit" and just assume it's infallible. If people are going to treat these things like they are forensic evidence and not lead generation, then they shouldn't be legal. Some precedent that people need to check these things before acting on them would be very good.
  • ajb 2y ago
    Very interesting - thanks for giving us the benefit of your experience. In the case of store workers, it's not obvious what other checking they could do. The only reasonable use I can see is for them to keep an eye on the person that has been flagged. But I expect that the entire purpose of this tech is to reduce the number of store guards they need to pay, so they are likely not to have the manpower. If the policy is 'don't let them in the store', I don't see how a store worker can avoid making false accusations.
    • namaria 2y ago
      There's something fundamentally wrong with these probabilistic analysis tools being sold currently as "artificial intelligence". The nuance in "probabilistic" is quickly lost and the output is taken as deterministic. Any piece of software that puts out data that is acted upon should have clear disclaimers attached. Calling it "intelligent" is irresponsible.
      • tomNth 2y ago
        "Calling it "intelligent" is irresponsible." How so ? If a human of animal behaving the same were to be describe as intelligent, than so are those systems.
        • namaria 2y ago
          How so? People take it at face value and harm other people based on its outputs, as per the article linked. > If a human of animal behaving the same were to be describe as intelligent, than so are those systems. I cannot parse this sentence. Behaving the same as what? What bearing do claims of human or animal intelligence have on me stating that generating liability by trusting probabilistic analysis, and actively harming people, by calling software "intelligent", is not responsible behavior?
          • exe34 2y ago
            we can call it frediness, but the problem isn't what it's called, the problem is that the profit motive only works when there's competition. if there were several shops nearby and one of them started rejecting 80% of customers based on frediness, they'd go out of business and the others who prefer having a human guard would prosper. if it's the only shop within a 10 mile radius, and it only flags out 1% of innocent people as shoplifters, then the profit motive doesn't provide the balancing force. that's what regulation is for.
            • deleted 2y ago
              [deleted]
          • tomNth 2y ago
            The bearing is on the claim that calling software "intelligent" have anything to do with trusting or liability. Dumb or not, no one is supposed to blindly trust it.
  • kreyenborgi 2y ago
    From The flaws of policies requiring human oversight of government algorithms https://www.sciencedirect.com/science/article/pii/S0267364922000292 https://www.sciencedirect.com/science/article/pii/S026736492... : human oversight policies provide a false sense of security in adopting algorithms and enable vendors and agencies to shirk accountability for algorithmic harms. And the better The Algorithm is, the less people check, and the more of its faults pass through. Having worked on systems that required human oversight, I've seen first-hand, and analyzed data on, how people click the "LGTM" button without reflecting. We even noticed how when we improved our predictor with expert input, so that it performed better according to experts who worked with it, it would perform worse when we benchmarked on the previously "manually checked" data set. Analysis showed that the manual checks had let through so many of the algorithm's faults, that when the algorithm improved, it got a worse score. See also https://predictive-optimization.cs.princeton.edu/ https://predictive-optimization.cs.princeton.edu/ : We present seven flaws of predictive optimization. Our aim is to outline a set of objections inherent to predictive optimization that cannot be easily fixed using a design or engineering change. Taken together, these critical flaws undermine the legitimacy of applications of predictive optimization. 1. Good predictions may not lead to good decisions 2. It's hard to measure what we truly care about 3. The training data rarely matches the deployment setting 4. Social outcomes aren’t accurately predictable, with or without machine learning 5. Disparate performance between groups can’t be fixed by algorithmic interventions 6. Providing adequate contestability undercuts putative efficiency benefits 7. Predictive optimization doesn't account for strategic behavior
    • Latty 2y ago
      True, "check" was poor phrasing, because these things don't produce evidence, they produce alerts to guide investigation. It should be a given that these systems don't produce anything actionable, they are a guide for a user to look more closely, of course, that requires someone in the chain stops and says "this isn't usable evidence".
  • thfuran 2y ago
    >If people are going to treat these things like they are forensic evidence Are most kinds of forensic evidence actually any better?
    • Latty 2y ago
      Well, plenty of bunk stuff (bite mark analysis, hair analysis, etc...) has been used in the past, but in theory anything used today should be backed up by significant evidence and rigorous methodology, and obviously it'll vary based on jurisdiction, but in theory courts should require that background. Certainly, more transparency than "we bought some facial recognition software, no, you can't see the code, that's a trade secret".
  • dec0dedab0de 2y ago
    Similar to story points and task estimates suddenly becoming deadlines
  • BLKNSLVR 2y ago
    > no matter how much you present something with clear "this is a heuristic, it can give you leads", a scary number of people will take it as "this will tell you whodunit" and just assume it's infallible. That's precisely my anecdotal experience. Subtlety and nuance are easily ignored where their absence appears to provide easy answers and the avoidance of responsibility for their outcomes. Similarly to "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It".
    • cogman10 2y ago
      I don't trust a sales person or marketer to not hype such a system as being flawless whodonit machine. I certainly don't trust that the people purchasing these machines won't communicate to their staff that it's a flawless whodonit machine. That's because the step from "potential leads" to "flawless machine" is a relatively small leap easily hidden by throwing out bad stats and case studies about people caught red handed. Even if you say the machine is 80% accurate, a lot of people will happily accept harming the 20% on the chance they might be in the 80%. (Though, I'm willing to bet the false identification percentages are both not public and not kept up to date, if they are even known).
  • mcherm 2y ago
    I can think of one possible regulation that I think would produce the proper behavior incentives. Require that the system intentionally produce false positives (selected at random) at approximately the same rate as the true positives. This means that the users of the system will be aware that any indication has only a 50% chance of being "real", and therefore can be used as an indicator to increase caution but NOT on it's own as a signal of guilt.
    • exe34 2y ago
      Why not just flip a coin then? It would be much cheaper?
      • satellite2 2y ago
        I imagine the solution space is much bigger. I think it could be an actually pretty good solution for this. A bit like Chaos Monkey but for human systems. Introducing error often enough that operators don't assume it never happens.
      • mcherm 2y ago
        If the expected rate of known shoplifters is 1 in 10,000 then the human security team can keep a close eye on 10,000 people (one of whom is probably a shoplifter) or on just 2 people (one of whom is probably a shoplifter). The latter is much more efficient. Without the intentional false positives they could simply throw out 1 individual with no need to keep an eye on anyone at all. That's enen MORE efficient, but it utterly ruins the lives of anyone falsely identified.
        • exe34 2y ago
          they'll just throw both out.
• btasker 2y ago
  > Telling someone wrongly that they are a thief, in public, seems like libel Slander. Libel is published, Slander is spoken. If they put a picture of her on a sheet that said "shoplifter" and posted in on the wall (or the net), it'd be libel. But, I don't think you'd win a case based on Slander either: * you'd still need to show it was published to a third party (though as you say, maybe you had friends present) * They can show that they reasonably believed it to be true (the system told them so). Their wording then becomes incredibly important - did they say "you're a shoplifter" or "you've been identified by our shoplifter recognition system" They might also try to argue that stopping shoplifting is in the public interest and that this was a statement of opinion related tot hat. I agree though, if this happened to me, I'd definitely be sending a letter or two to dissuade them from making a similar mistake in future (I think I'd also be trying my luck with Facewatch given that their system disseminated that "information")
  • ajb 2y ago
    Yes, I'd forgotten that libel is published information. Although, given that this is third party tech, maybe Facewatch counts as the publisher.
    • davidhbolton 2y ago
      Wouldn't there be a better outcome for those misidentified and publicly slandered and inconvenienced if they sued for stress? Some unlucky sod could have a heart attack from this.
  • prepend 2y ago
    Libel and slander must have malicious intent. I think the better legal tactic would be defamation as it doesn’t require intent.
• SapporoChris 2y ago
  It seems that the proponents of this technology believe the benefits far outweigh the harms. Why not have a standard generous payout to anyone misidentified by the system. The payout would be funded by all parties involved, store, police, tech company. If the benefits really do outweigh the harms and the compensation is enough, I think most people would begin welcoming the program. If the system got gamed (people with a history of false positives seeking to be misidentified), fine, it would encourage the technology to improve and eliminate false positives.
  • thfuran 2y ago
    The payout would have to be at least enough to both fund and cover the inconvenience of moving to a new neighborhood closer to stores you aren't yet banned from. Of course, that becomes impossible if the tech is widespread enough.
• calewis 2y ago
  It doesn’t really work like that in the UK. I doubt you’d get anyone to take it on, let alone for feee. Proving the material impact would be hard. She’s not been named publicly, no lost earnings etc. I would like to think this sort of thing means we get better regulation, but who am I kidding.