6 ms·

When I worked on analysis software for police & intelligence agencies, it quickly became clear that no matter how much you present something with clear "this is

• ajb 2y ago
  Very interesting - thanks for giving us the benefit of your experience. In the case of store workers, it's not obvious what other checking they could do. The only reasonable use I can see is for them to keep an eye on the person that has been flagged. But I expect that the entire purpose of this tech is to reduce the number of store guards they need to pay, so they are likely not to have the manpower. If the policy is 'don't let them in the store', I don't see how a store worker can avoid making false accusations.
  • namaria 2y ago
    There's something fundamentally wrong with these probabilistic analysis tools being sold currently as "artificial intelligence". The nuance in "probabilistic" is quickly lost and the output is taken as deterministic. Any piece of software that puts out data that is acted upon should have clear disclaimers attached. Calling it "intelligent" is irresponsible.
    • tomNth 2y ago
      "Calling it "intelligent" is irresponsible." How so ? If a human of animal behaving the same were to be describe as intelligent, than so are those systems.
      • namaria 2y ago
        How so? People take it at face value and harm other people based on its outputs, as per the article linked. > If a human of animal behaving the same were to be describe as intelligent, than so are those systems. I cannot parse this sentence. Behaving the same as what? What bearing do claims of human or animal intelligence have on me stating that generating liability by trusting probabilistic analysis, and actively harming people, by calling software "intelligent", is not responsible behavior?
        • exe34 2y ago
          we can call it frediness, but the problem isn't what it's called, the problem is that the profit motive only works when there's competition. if there were several shops nearby and one of them started rejecting 80% of customers based on frediness, they'd go out of business and the others who prefer having a human guard would prosper. if it's the only shop within a 10 mile radius, and it only flags out 1% of innocent people as shoplifters, then the profit motive doesn't provide the balancing force. that's what regulation is for.
          • deleted 2y ago
            [deleted]
        • tomNth 2y ago
          The bearing is on the claim that calling software "intelligent" have anything to do with trusting or liability. Dumb or not, no one is supposed to blindly trust it.
• kreyenborgi 2y ago
  From The flaws of policies requiring human oversight of government algorithms https://www.sciencedirect.com/science/article/pii/S0267364922000292 https://www.sciencedirect.com/science/article/pii/S026736492... : human oversight policies provide a false sense of security in adopting algorithms and enable vendors and agencies to shirk accountability for algorithmic harms. And the better The Algorithm is, the less people check, and the more of its faults pass through. Having worked on systems that required human oversight, I've seen first-hand, and analyzed data on, how people click the "LGTM" button without reflecting. We even noticed how when we improved our predictor with expert input, so that it performed better according to experts who worked with it, it would perform worse when we benchmarked on the previously "manually checked" data set. Analysis showed that the manual checks had let through so many of the algorithm's faults, that when the algorithm improved, it got a worse score. See also https://predictive-optimization.cs.princeton.edu/ https://predictive-optimization.cs.princeton.edu/ : We present seven flaws of predictive optimization. Our aim is to outline a set of objections inherent to predictive optimization that cannot be easily fixed using a design or engineering change. Taken together, these critical flaws undermine the legitimacy of applications of predictive optimization. 1. Good predictions may not lead to good decisions 2. It's hard to measure what we truly care about 3. The training data rarely matches the deployment setting 4. Social outcomes aren’t accurately predictable, with or without machine learning 5. Disparate performance between groups can’t be fixed by algorithmic interventions 6. Providing adequate contestability undercuts putative efficiency benefits 7. Predictive optimization doesn't account for strategic behavior
  • Latty 2y ago
    True, "check" was poor phrasing, because these things don't produce evidence, they produce alerts to guide investigation. It should be a given that these systems don't produce anything actionable, they are a guide for a user to look more closely, of course, that requires someone in the chain stops and says "this isn't usable evidence".
• thfuran 2y ago
  >If people are going to treat these things like they are forensic evidence Are most kinds of forensic evidence actually any better?
  • Latty 2y ago
    Well, plenty of bunk stuff (bite mark analysis, hair analysis, etc...) has been used in the past, but in theory anything used today should be backed up by significant evidence and rigorous methodology, and obviously it'll vary based on jurisdiction, but in theory courts should require that background. Certainly, more transparency than "we bought some facial recognition software, no, you can't see the code, that's a trade secret".
• dec0dedab0de 2y ago
  Similar to story points and task estimates suddenly becoming deadlines
• BLKNSLVR 2y ago
  > no matter how much you present something with clear "this is a heuristic, it can give you leads", a scary number of people will take it as "this will tell you whodunit" and just assume it's infallible. That's precisely my anecdotal experience. Subtlety and nuance are easily ignored where their absence appears to provide easy answers and the avoidance of responsibility for their outcomes. Similarly to "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It".
  • cogman10 2y ago
    I don't trust a sales person or marketer to not hype such a system as being flawless whodonit machine. I certainly don't trust that the people purchasing these machines won't communicate to their staff that it's a flawless whodonit machine. That's because the step from "potential leads" to "flawless machine" is a relatively small leap easily hidden by throwing out bad stats and case studies about people caught red handed. Even if you say the machine is 80% accurate, a lot of people will happily accept harming the 20% on the chance they might be in the 80%. (Though, I'm willing to bet the false identification percentages are both not public and not kept up to date, if they are even known).
• mcherm 2y ago
  I can think of one possible regulation that I think would produce the proper behavior incentives. Require that the system intentionally produce false positives (selected at random) at approximately the same rate as the true positives. This means that the users of the system will be aware that any indication has only a 50% chance of being "real", and therefore can be used as an indicator to increase caution but NOT on it's own as a signal of guilt.
  • exe34 2y ago
    Why not just flip a coin then? It would be much cheaper?
    • satellite2 2y ago
      I imagine the solution space is much bigger. I think it could be an actually pretty good solution for this. A bit like Chaos Monkey but for human systems. Introducing error often enough that operators don't assume it never happens.
    • mcherm 2y ago
      If the expected rate of known shoplifters is 1 in 10,000 then the human security team can keep a close eye on 10,000 people (one of whom is probably a shoplifter) or on just 2 people (one of whom is probably a shoplifter). The latter is much more efficient. Without the intentional false positives they could simply throw out 1 individual with no need to keep an eye on anyone at all. That's enen MORE efficient, but it utterly ruins the lives of anyone falsely identified.
      • exe34 2y ago
        they'll just throw both out.