7 ms·

This article, and in particular the title, is horrible and shows no understanding of the math in play. The facts do not indicate that the dog "is barely more ac

• lisper 11y ago
  On your quality metric, the best result would come from a two-headed coin that indicated positive 100% of the time. But that seems to me to miss the point rather badly.
  • harryh 11y ago
    Obviously the best quality metric would be 0 false positives and 0 false negatives. A two-headed coin would not achieve that.
    • lisper 11y ago
      You have completely missed the point. Zero error is not achievable. You have to choose to accept either false positives or false negatives. In a free society, false positives are supposed to be considered the greater evil.
      • harryh 11y ago
        I have made no judgement about the relative evilness of false positives vs false negatives. I have merely stated that the dog's performance is dramatically different from that of a fair coin and that making that comparison is a misrepresentation of the facts.
        • lisper 11y ago
          It's not different in the way that matters: both produce a lot of false positives. That makes it a fair comparison despite the fact that some of the details are different.
          • harryh 11y ago
            On your quality metric, the best result would come from a two-tailed coin that indicated positive 0% of the time. But that seems to me to miss the point rather badly. ;-)
            • lisper 11y ago
              Actually, for drug offenses I'd take the two-tailed coin without hesitation. Prohibition doesn't work. Never has. Never will.
              • harryh 11y ago
                Then it sounds like your opinion on the drug war is incorrectly influencing your opinion on issues surrounding the acceptable standards for a probable cause search. While the particular example here is drugs the same questions apply on searches for guns, stolen property or any other contraband. It's important to keep orthogonal issues separate so you don't cross contaminate and introduce unintended outcomes.
                • lisper 11y ago
                  > Then it sounds like your opinion on the drug war is incorrectly influencing your opinion on issues surrounding the acceptable standards for a probable cause search Well, you set up a straw man, so the rules of logic had already been implicitly suspended in that branch of the conversation. > the same questions apply on searches for guns, stolen property or any other contraband No, they don't. Dogs can't generally sniff out guns or distinguish between stolen and non-stolen property, and so dogs can't be used as a pretense in those situations. Using a dog to sniff out stolen property (unless that stolen property happens to be a salami) would make it obvious that the dog is just a pretense. > It's important to keep orthogonal issues separate Indeed. So let's review: the problem is that although dogs can sniff out drugs, unless some very strict protocols are followed, they generally don't sniff out drugs and instead just tell their handler what they want to hear, i.e. that there are drugs (though, of course, the dog has no idea that this is how its actions are being interpreted. All it knows is that if it signals, it gets a pat on the head.) As a result, dogs are not an effective crime fighting tool, but instead a de facto pretense for doing an end-run around the fourth amendment. The same effect could be produced by other means, e.g. flipping a biased coin, but that would defeat the purpose because that would make the bias obvious, which is exactly the rhetorical point that the original author wanted to make. Do you really want to disagree with any of that?
      • niels_olson 11y ago
        Choosing between false positive and false negatives has a lot to do with the consequences. If you miss a drug dealer on a screening test, you missed the chance to catch the bad guy. Depending on the drug and social dynamics, this could be fairly benign (pot) or lethal (cocaine, meth, benzos, opioids). You've also communicated to the bad guy that he can get away with it. Which, given what we know of male behaviour, suggests he will push more drug more brazenly next time.
        • lisper 11y ago
          Or you could just repeal prohibition. Then drug dealers wouldn't be "bad guys" any more and you wouldn't need to catch them.
          • niels_olson 11y ago
            You have perhaps not autopsied many overdoses.
• elif 11y ago
  When i was moving and literally everything i owned and cared about was pulled out of my car at 2am and strewn on the sidewalk in the rain for hours, the "extra 155 false positives" seemed to matter a little more to me than it does to you. This isn't a clinical study where the stats are everything. This is a real world situation where people are being profiled and it is legalized.
  • harryh 11y ago
    I was not arguing whether dogs with this level of success should be used or not. I was pointing out that the argument made in the essay misrepresented the facts. See my last sentence: "We can argue about whether that is good enough but comparing it to a coinflip makes no sense."
    • elif 11y ago
      Your point is that it is not comparable from a "I caught more bad guys" standpoint, which is fair. But I think the other side: to say that a coin flip would punish less innocent people, makes it a fair comparison.
      • niels_olson 11y ago
        Er, no. A coin flip would punish more innocent people. This is an argument of specificity, not sensitivity. The "screening" test, the "sensitive" test, was when the officer decided to involve Lex.
        • merpnderp 11y ago
          Not if they've already been screened by the police. IE, if the cops only flip the coin once they would have brought in the dog, then it is no better than a coin flip.
• nharada 11y ago
  I want to see the ROC curve
• geggam 11y ago
  So that is after the officers filter of society. Officer picks 100 people to be suspected drug dealers out of N ... Out of the 100 the officer picks the dog is choosing 93%... but is only right 55%... the dog correlates to the officers choice and both are only right about half the time
• rayiner 11y ago
  The article's whole analysis is fucked up because it ignores the prior probability. > In U.S. v. Bentley, we see just how damaging the Harris decision really was. Lex, the drug dog that searched Bentley’s car, had a 93 percent alert rate. That is, when Lex was called to search a car, he alerted 93 percent of the time. He was basically a probable cause generator. His success rate was much lower, at 59 percent. In other words, drugs were found in 55% of cars in which he was brought in.[1] The base rate for drug possession in a car is almost certainly a tiny fraction of that. So between the human judgment to bring in Lex, and Lex's sniffing, the police are able to find drugs at probabilities that are multiples of the base rate. In contrast, if you searched cars based on a coin flip (i.e. randomly), you'd expect to find drugs exactly at the base rate. [1] Being successful > 50% of the time is pretty much the definition of probable cause.
  • harryh 11y ago
    Indeed you are completely correct.
  • ethagknight 11y ago
    I wonder if some percentage of Lex's false positives were from vehicles that had drugs in them at one point but not during the search?
  • CPLX 11y ago
    Not really. You have conflated a couple things here: There is some aggregate base rate, which is the percent of a random sample of motorists that are carrying drugs. Then there is the rate of those who are actually carrying drugs within the a group that the police have pulled over, and suspect are carrying drugs, and have also presumably asserted their rights and refused a consensual search by the police and also do not have some other attribute (such as smell, items visible in plain view) that would provide probable cause. You argue that "the base rate for drug possession in a car is almost certainly a tiny fraction of that" but in fact the base rate within this much narrower second group should be extremely high in comparison to the general population. And the problem is that searching that group, regardless of what you think of them, is illegal. In order to make it legal you have to be able to show that you're going to do something specific that will provide probable cause. Showing that the dog is anything short of a clearly reliable and accurate means of determining that a law is being violated means that the dog is not providing probable cause. A dog that alerts to drugs more 90% of the time when only 60% or so of the population in question has the drugs, has shown results that aren't sufficiently correlated to the data it is supposed to discern. When that is true it is just a tool for violating laws that prohibit searches.
• davik 11y ago
  The coin flip comparison is bad, but I think the article is saying that the precision is very low (like half) but the recall is possibly high. https://en.wikipedia.org/wiki/Precision_and_recall https://en.wikipedia.org/wiki/Precision_and_recall
  • harryh 11y ago
    I think that's what the article is trying to say too, it just does a very very bad job of it.
• DanBC 11y ago
  Most people have no idea how to use percentages. Most of the population has no hope of understanding what's actually going on with this article. That becomes scary when you're talking to a doctor about a test that returned a positive result. Does that doctor know how likely it is you actually have what you tested positive for? Are you going to get treatment for this thing? Gerd Gigerenzer has a nice book about this: http://www.amazon.com/Reckoning-Risk-Learning-Live-Uncertainty/dp/0140297863 http://www.amazon.com/Reckoning-Risk-Learning-Live-Uncertain... Here's one example about breast cancer testing: http://imgur.com/zO4zkl4 http://imgur.com/zO4zkl4
  • harryh 11y ago
    Indeed, I agree with your assessment of the statistical reasoning ability of the general public. I think that articles like this one, if they were better, could move things (even if just a little bit) in a positive direction by educating the public. Unfortunately when they are as poorly written as this I think they do the exact opposite. Oh well.
• shkkmo 11y ago
  Yes, it doesn't make sense to compare to a coin flip. However, I would argue that the article's understanding of the math in play is at least as good as the courts who are deciding these cases. However my reading is the the 'coin flip' is being used because the dog's accuracy is near false positive rate 50% and they are trying to compare the dog's accuracy vs. a randomly accurate instrument to illustrate the following point to non statisticians: "If a drug dog isn’t eliminating any innocent people, if it’s validating the suspicions of the police nine out of ten times resulting in searches for which up to half or more of the suspects are innocent, then descriptors such as finely tuned and incredible precision no longer apply."
  • harryh 11y ago
    Keep in mind that the dog isn't putting people in jail. It's assertion is merely being used as evidence for probable cause and a more in depth search. As rayiner notes elsewhere in this thread being right over half the time is generally considered the acceptable standard for probable cause.
    • shkkmo 11y ago
      > being right over half the time is generally considered the acceptable standard for probable cause. Care to cite that? Because that means that in a neighborhood where 51% of the houses contain narcotics, the police have probable cause to search every single house.
      • rayiner 11y ago
        See Section IIA of: http://scholarship.law.duke.edu/cgi/viewcontent.cgi?article=1577&context=lcp http://scholarship.law.duke.edu/cgi/viewcontent.cgi?article=.... It explains that probable cause is generally considered a lower standard than preponderance of the evidence (which is a 51% standard).
        • shkkmo 11y ago
          Actually it sort of directly contradicts you right at the start of that section: >We do not know exactly what the phrase “probable cause” means, in strict numerical terms. We do, however, know what it does not mean: “probably.” That is, probable cause does not—in the context of Fourth Amendment law—mean that the police must have evidence sufficient to conclude that a suspect is probably guilty or that she probably has evidence of a crime hidden inside her home. I didn't read the whole paper, but the gist of it seems to being exploring the difference between 'concrete' and 'statistical' and evidence of effects (with a non-conclusive, but general indication that the supreme court leans towards 'concrete' over 'statistical') Basically if you have a 3 people in a car that contains drugs and thus know 1 of them is guilty, you have probable cause to arrest all of them. (Concrete) However, if you know that 1/3 of the black people in a certain neighborhood posses drugs, that does not mean you have probably cause to arrest all of them. (Statistical) Thus my argument is that the low accuracy rate of the dog vs. the baseline of the searched people (which in this case is only known because of the high alert rate) should not allow it to constitute 'concrete' evidence of probable cause. Additionally, the overall success rate of the dog/officer combination vs. the general population baseline constitutes 'statistical' evidence which is also not sufficient for probable cause. Generally, my argument would be that allowing the dogs to serve as a final determination of probable cause without requiring high accuracy rates leads to unnecessary harm to the general populace in the form of the unwarranted searched that could be prevented by encouraging effective training and usage.
• analyst74 11y ago
  You don't know about the remaining 70 are true negatives or not, so you can't use that data for calculation. If you only consider the 930 cases where there is verification, you do come up to 59% accuracy. Even using your generous assumption of the 70 cases being true negatives, the dog only come up to 62% accuracy.