11 ms·
Court rules drug dog barely more accurate than a coin flip is good enough
- fiatmoney 11y agoThe dog isn't even "as accurate as a coin flip". Given a 93% alert rate, regardless of the subsequent true positive rate, information gain from an alert is approximately zero. Edit: "accurate as a coin flip" is a bad construction, but I wouldn't exactly expect better from a newspaper. The information gain from a device that activates randomly 50% of the time, 0% of the time, and 100% of the time are all the same, which is to say zero.
- JoeAltmaier 11y agoBe fair. False positives and false negatives are very different things. A drug dog that finds drugs even sometimes, is still useful. Imagine if it was bombs they were sniffing for. We'd be happy if they found even one more bomb.
- gruez 11y ago>A drug dog that finds drugs even sometimes, is still useful. Why even bother with the dog? why not just do random searches? Oh wait, because that would be unconstitutional. The dog exists solely to justify otherwise unconstitutional searches.
- JoeAltmaier 11y agoIgnoring the fact that false positives don't preclude real positives...it works. People are fond of cute doggies, and often their resentment evaporates.
- ossreality 11y agoOh boy, you've gotta be joking, right?
- zajd 11y ago[citation needed]
- elektromekatron 11y agoFalse positives only don't preclude real positives if time and resources are not constrained. Time and resources are always constrained.
- elektromekatron 11y agoIf it was explosives and a dog got a 93% alert rate irrespective of there being any explosives, then that would be useless and nobody in their right mind would use that dog, as it would have them search 93% of everything. At that point you might as well give up on faffing about with a sniffer dog and just search the remaining 7%.
- pavel_lishin 11y agoThen we should simply search 100% of the bags. Or maybe just 93%.
- protonfish 11y agoIf I had a coin that showed heads 93% of the time, I'd think it might be rigged.
- DSMan195276 11y agoIMO, it doesn't seem remotely comparable to a coin flip because the odds aren't 50/50. The dogs are only brought out in certain circumstances when they think a positive might be possible, meaning that lots of people who would be negatives aren't tested and thus not included in that 93%. The cases making up the 93% only consist of people whom they thought may get a positive, meaning the sample is biased to having lots of positives. I'd compare it to the fact that lots of very hard certification tests actually have fairly high pass-rates (80% or above), simply because people who don't think they could pass don't take the test to begin with - The 80% pass-rate is not representative of the average person's chances of passing the test, nor the difficulty of the test itself. What's really important is the percentage of those positives which were true positives, vs. false positives. In this case, you would expect the number of positives to be high - The number of true positives vs. false positives would tell you how accurate the dog actually was in the positive case. If this number is also high, then the dog is doing a reasonable job. Of course, to derive a real argument from this, you'd need to know the number of true and false negatives as well, but obviously we don't have that information.
- slapresta 11y agoSure, because a coin is evaluated on its capacity to map to an even distribution of outcomes. Drug possessions, however, are almost surely not an even distribution. If, say, 95% of the people that the dog evaluates have drugs, then raising an alarm 93% of the time is probably pretty good. In other words: If we wanted a drug dog to be as close as possible to a coin flip, we'd just flip a coin instead.
- gburt 11y agoThis isn't being fair. I don't really think the post title is fair either. That 93% isn't comparable to the 50% you're probably thinking for the coin flip, or even the 0/100% mentioned in your edit. --- 93% alert rate means: 93% = false positive + true positive 7% = true negative + false negative --- We're given "His success rate was much lower, at 59 percent." Assuming that means that 59% are true positives, we know the false positive rate is 34%. Without data on how many passengers are carrying drugs, we can't say anything about the rest of that result. A plausible world where the drug dog may be valuable is when he unquestionably rules out that 7% of the population that he doesn't alert on, as in false negative = 0. I believe the reason this number is so high (59%) is that the officer has already used his suspicion to determine filter out most people. This makes the value of the "ruling out" plausibility even more valuable: the drug dog reduces searches compared to trusting the cop. Ideally, they'd provide the receiver operating characteristics curve [0] or the confusion matrix [1]. [0] https://en.wikipedia.org/wiki/Receiver_operating_characteristic https://en.wikipedia.org/wiki/Receiver_operating_characteris... [1] https://en.wikipedia.org/wiki/Confusion_matrix https://en.wikipedia.org/wiki/Confusion_matrix
- nickff 11y agoThe real question is whether the dog is better than the officer at guessing who has drugs. If the dog is much better than the officer, it would suggest that a dog 'alert' should be given more weight than the officer's guess. If the dog is no better than the officer, the dog is useless. I say this because it is possible (and indeed likely) that the officer is also better at guessing who has drugs than a coin toss. Full Disclosure: I have never used or trafficked contraband, but I do not support random searches or the use of drug dogs in public without a warrant, regardless of their effectiveness.
- gburt 11y agoI have no opinion on the legal and ethical issues, I was commenting only on the classifier. I don't think your first question is appropriately described as "the real question" though, as one of the coolest results of machine learning is that there are techniques for combining multiple bad classifiers [0] in to one really good one. Even more so, I think that is actually what is happening here. An officer is acting as a tier 1 filter, and then the drug dog is acting as a tier 2 filter, further reducing the number of false positives, which in a "legal and ethical" sense, we generally feel are worse than false negatives. [0] https://en.wikipedia.org/wiki/Ensemble_learning https://en.wikipedia.org/wiki/Ensemble_learning
- merrywhether 11y agoI think the coin-flip analogy comes from the idea that once an officer thought you had drugs, he/she was about 55% right (not accounting for false negatives). The dog alerted most of the time, thus allowing for search to confirm or deny the suspicions. Thus the dog gave no real information and the officer was the coin, but the reporter conflated these success rates.
- deutronium 11y agoI wonder if we could use bees instead then, as there seemed to be some interesting research on training them to detect drugs and presumably they wouldn't be biased by human behaviour especially. https://en.wikipedia.org/wiki/Hymenoptera_training https://en.wikipedia.org/wiki/Hymenoptera_training Alternatively are there any portable mass spectrometers now?
- jonlucc 11y agoSurely a portable mass spec would be an actual search?
- eli 11y agoYou'd hope so, especially since the court ruled that pointing thermal imaging cameras at your house from the street constitutes a search.
- DanBC 11y agoAnother thing Bees were being used to detect was explosive, but researchers discovered that bees detected the impurities in explosives. If you gave the bees pure samples of explosive they didn't do very well. Professor Jackie Akhavan (Professor of Explosive Chemistry) mentioned this on the BBC programme "The Life Scientific", where a scientist talks to other scientists about their work. It's for a general audience so nothing particularly technical, but still interesting.
- deutronium 11y agoCheers, I'll see if I can grab that.
- DanBC 11y agoSorry! I forget to include the link. Here's that specific episode: http://www.bbc.co.uk/programmes/b04jk360 http://www.bbc.co.uk/programmes/b04jk360 > Jackie Akhavan, Professor of Explosive Chemistry, tells Jim al-Khalili all about the science of explosives. She explains exactly what explosives are and how to make them safer to handle. > She started by working on how to make fireworks safer and has been involved in research with bees to see whether they can be used smell different types of explosives. Her current project involves testing the rocket fuel that will be used in Bloodhound, the British designed and built supersonic car that aims to reach a speed of 1,000mph. > Her work involves finding out how to best detect explosives in airports and elsewhere, teaching security professionals how to differentiate between false alarms and the real thing. She also works on explosives used in warfare and discusses the ethical issues involved.
- javajosh 11y agoSo drug dogs become a kind of police "search anything" power totem. (This as opposed to the "kill anything" power totem that is the cops brain and tongue via I thought he had a gun or I thought my life was in danger; or the "break into people's home and terrorize them" power totem of the knock-less warrant; or the "track anyone, listen to anyone" power totem of the Patriot Act, etc. at the federal level.)
- joeax 11y agoYou forgot "he reached for my gun" or in the case of Zachary Hammond, "he tried to kill me by ramming me with his car" (by pulling into a parking spot).
- merpnderp 11y agoBefore the US revolutionary war they were called "General Warrants" and allowed British soldiers to search and seize as they pleased.
- jimrandomh 11y agoThe article says: > In 2013, the Supreme Court made things worse in Florida v. Harris. In that case, the court unanimously ruled that mere certification of a drug dog was enough to establish a presumption that a drug dog is reliable, regardless of the reputation of the certifying organization, regardless of whether that organization understands and appreciates the importance of training dogs to ignore their handlers’ suspicions, and regardless of the dog’s performance in the real world. But the decision in Florida v. Harris says: > A defendant, however, must have an opportunity to challenge such evidence of a dog's reliability, whether by cross-examining the testifying officer or by introducing his own fact or expert witnesses. The defendant, for example, may contest the adequacy of a certification or training program, perhaps asserting that its standards are too lax or its methods faulty. ... > In short, a probable-cause hearing focusing on a dog's alert should proceed much like any other. The court should allow the parties to make their best case, consistent with the usual rules of criminal procedure. And the court should then evaluate the proffered evidence to decide what all the circumstances demonstrate. If the State has produced proof from controlled [7] settings that a dog performs reliably in detecting drugs, and the defendant has not contested that showing, then the court should find probable cause. If, in contrast, the defendant has challenged the State's case (by disputing the reliability of the dog overall or of a particular alert), then the court should weigh the competing evidence. ... > Harris, as also noted above, declined to challenge in the trial court any aspect of Aldo's training. See supra, at 3. To be sure, Harris's briefs in this Court raise questions about that training's adequacy — for example, whether the programs simulated sufficiently diverse environments and whether they used enough blind testing (in which the handler does not know the location of drugs and so cannot cue the dog). See Brief for Respondent 57-58. Similarly, Harris here queries just how well Aldo performed in controlled testing. See id., at 58. But Harris never voiced those doubts in the trial court, and cannot do so for the first time here. So what that supreme court case actually* says is that the validity of a drug-sniffing dog as probable cause for a search can be challenged in court, but the challenge must be made in trial court rather than on appeal.
- piker 11y agoThe article says that certification establishes a presumption of reliability. That means that certification can shift the burden of proof to the defendant. In that case, Bentley would have to prove to the jury that the dog is unreliable. It's correct. It seems Bentley's trial counsel erred by not presenting evidence that the dog was unreliable, and hence did not meet Bentley's burden. [Edit: clarity]
- mangeletti 11y agohttps://en.wikipedia.org/wiki/Base_rate_fallacy https://en.wikipedia.org/wiki/Base_rate_fallacy
- daenz 11y agoAdaBoost is an algorithm for taking weak classifiers (performing only slightly better than random chance) and boosting/weighing them such that together they are a strong classifier. https://en.wikipedia.org/wiki/AdaBoost https://en.wikipedia.org/wiki/AdaBoost To play the devil's advocate, if the dog alerting to drugs is only one of the classifiers in play (others being the suspects look/act under the influence), and if it's weighed correctly in an officer's mind, it is a key part of the strong classifier.
- saint_fiasco 11y agoHumans don't properly implement AdaBoost on their heads. If a dog alerts an officer, confirmation bias will cause the officer to think "now that I look at him closely, he sure seems to be acting strange..." For bad implementations of adaptive boosting, weak information can be worse than no information.
- jqm 11y agoIf we could convince the suspects to repeatedly drive by the same officer and dog n number of times so they could be pulled over and a trial run, that might just work.
- shkkmo 11y agoBut if you are going to use that reasoning, then you need to remove the dog from the equation and only judge the officers on their actual success rate, utilizing any method at their disposal. What should this rate be? 50%? Any officer who falls below that rate should be forced to get approval from another officer with a higher success rate? They can then choose to use dogs to supplement certain officer's accuracy without giving them incentive to train the dogs to be "probable cause generators". Using a weak classifier to serve as the basis for getting around a constitutional right is pretty messed up.
- daenz 11y ago> Using a weak classifier to serve as the basis for getting around a constitutional right is pretty messed up. I was playing the devil's advocate, in case you missed that part, and my point was that a classifier being weak does not necessarily make it bad, as evidenced by AdaBoost.
- rdtsc 11y agoDogs are a 100% proof way to always let the cops search your car. If the court takes that way, they'll have to invent some other legal loophole. Handler trains their dog to alert on cues such as saying "Good boy" or "Find..." and so on. The dog understands what to do. Alerts and gets a treat or patted on the back. I am wondering what are statistics of "dog alerted, search performed, but no drugs found", does anyone keep that? It would be interesting as it would reveal the scheme pretty well. (They can of course say, well at some point in the past the car had drugs in it...)
- harryh 11y agoThis article, and in particular the title, is horrible and shows no understanding of the math in play. The facts do not indicate that the dog "is barely more accurate than a coinflip" if 59% of the indications in question turn out to actually have drugs. Out of 1000 samples the dog is positively indicating for 930 of the (93% of the time) of which 549 are true positives and 381 are false positives. Let's be generous and say that all 70 remaining are true negatives. If the dog was subbed out for a coin flip we would have a positive indicator 500 times of which 274 would be true positives and 226 would be false positives. We would then have 275 false negatives and 225 true negatives. So subbing in the dog for a coin catches 275 drug carriers at the expense of an extra 155 false positives. We can argue about whether that is good enough but comparing it to a coinflip makes no sense.
- lisper 11y agoOn your quality metric, the best result would come from a two-headed coin that indicated positive 100% of the time. But that seems to me to miss the point rather badly.
- harryh 11y agoObviously the best quality metric would be 0 false positives and 0 false negatives. A two-headed coin would not achieve that.
- lisper 11y agoYou have completely missed the point. Zero error is not achievable. You have to choose to accept either false positives or false negatives. In a free society, false positives are supposed to be considered the greater evil.
- harryh 11y agoI have made no judgement about the relative evilness of false positives vs false negatives. I have merely stated that the dog's performance is dramatically different from that of a fair coin and that making that comparison is a misrepresentation of the facts.
- ryguytilidie 11y agoAt that point why not just have officers approach a suspects car with a box with a button and two lights, press the button and if the light turns green, there is "reasonable suspicion"? Just have the light turn green everytime and they can always search you and F the 4th Amendment.
- joesmo 11y agoThis is hardly news. Until we put in real consequences for both police and judges who intentionally misbehave or oftentimes fail at their job, none of this will change. Here's an idea: penalize the cop for every false positive. Docking pay seems to be the best idea here. I guarantee drug dogs' accuracies will go up nationwide within weeks (or however long it takes to retrain dogs and their masters). Every 5 or 10 false positives, dock the cop extra pay or reprimand him. Problem solved. Here's another idea: hold judges accountable for police actions in the courtroom. If the police lie on the stand, and it's shown that the judge had "probable cause" the judge should be fired. The collusion problem between police and judges, what the article is referring to tough maybe not explicitly, will go away as well. Problem solved. tl;dr: The only way to keep the police in check is to make them suffer real consequences for fucking up. This will never happen in the US because the citizens here have no idea about the atrocities the police are comitting and the police prefer to keep it that way.
- cle 11y agoNot only is this an important civil rights issue, but it's a safety issue too. If the dog is used as a proxy for the bias/hunch of the officer, rather than objectively, then adversaries can use that to their advantage, e.g. to divert resources away from a real threat.
- shkkmo 11y agoThis headline is pretty terrible. Keep in mind when looking at these statics that the number for this dog is not based on a random sample. Therefor it is impossible to actually gage the dog's accuracy from these numbers, since we don't know the ration of false:true negatives, only the number of false:true positives. The ratios could be identical, which would make the dog's alerting behavior random. There could even be a higher ratio of false:true negatives, which would make the dog worse than random. The only meaningful number is the 93% ratio which basically means that the dog is very close to being a 'probably cause generator' as mentioned in the article.
- ericabiz 11y agoNot the first time, nor is this the worst offense perpetuated by drug dogs. A town in New Mexico used one drug dog with an expired certification, "Leo", in order to justify invasive anal probes on two men--neither of whom had drugs. To add insult to injury, both men were later billed by the hospitals for this non-consenting search. Timothy Young, 2012: http://krqe.com/2014/07/28/nm-driver-tells-story-of-sheriffs-invasive-body-search/ http://krqe.com/2014/07/28/nm-driver-tells-story-of-sheriffs... David Eckert, 2013: http://www.dailykos.com/story/2013/11/05/1253274/-Absolutely-unimaginable-this-could-happen-in-America http://www.dailykos.com/story/2013/11/05/1253274/-Absolutely... Both cases were eventually settled, leading to Young receiving a $900,000 settlement and Eckert receiving $1.6 million: http://articles.latimes.com/2014/jan/16/nation/la-na-nn-new-mexico-police-search-20140116 http://articles.latimes.com/2014/jan/16/nation/la-na-nn-new-... These two cases are an absolutely horrifying violation of human rights and I'm really glad to see drug dogs being taken to task as part of what led to both of these men being violated.
- piker 11y agoTo be clear, what this appellate court and the trial court were asked to consider are different questions. Trial courts find facts. For example, "this guy shot that guy". Mostly these are considered by a jury, but in preliminary things such as Bentley's probable cause hearing, the trial court judge can act as a fact-finder. Appellate courts interpret laws. For example, strip searching a middle school student for pain medication is an "unreasonble" search under the Fourth Amendment. Lex is demonstrably inaccurate. Bentley argued in appellate court that Lex's alert is insufficient to support probable cause under any circumstances--i.e., a "matter of law". Bentley did not argue in trial court that Lex's alert was unreliable in his situation--i.e., a "matter of fact". The ruling simply reject's Bentley's argument, but preserves future defendant's rights to make the latter argument.
- Zikes 11y agoStatistics should be a high school course.
- doomsells 11y agoUpvoted because drugs are good.
- Animats 11y agoHm. What would be the probable cause level required for a search of a politician's records, phone calls, and bank accounts for evidence of bribery. We could probably use big data to establish patterns which are indicative of bribe-taking that would be demonstrably more reliable than 50%.
- mikestew 11y agoI am not a professional dog trainer, meaning I do not have a CPDT certification nor do people pay me. However, I have extensive experience training dogs at a local dog shelter (minimum two hours/week) in a specific training atmosphere, and I have a fair amount of experience with "nose works" which is somewhat related to "drug sniffing dogs" (nose works is where a drug dog might start, for instance). In other words, if software didn't pay so darned well I could go jump through the hoops and hang out my dog trainer shingle. I also hang out with a lot of trainers who are certified and paid for their work. With my (albeit weak) credentials out of the way, my quasi-professional opinion is that drug dogs are the polygraph test of the drug war. I've watched bomb and drug dogs at work, and much like a polygraph, I think you can read anything you want into the dog's actions. And unless you're 100% consistent (and even pros aren't), the dog might mis-read the handler. There are retired women who devote way too much time to the "nose works" sport with their dog(s), and in competition even those dogs get it wrong a fair percentage of the time. For bomb sniffing and sniffing for bodies after a disaster, that's fine. Dog says there's explosives, turns out Fido was wrong, no biggie. A few folks got unnecessarily riled up, but no other harm done. Dog said it found a body, but there's nothing but rubble? Meh, did some digging we didn't have to do, but otherwise little harm done (though we could have been digging for a real person). Sniffing for drugs? False positives have a greater impact. And just because the dog hit, and there were drugs, doesn't mean the dog was right. Correlation and causation and all that, I suppose. Anyway, I'm explaining this poorly, but I hope the gist gets across. TLDR: I spend a lot of time around dogs trying to teach them things. They might be more reliable than a polygraph, but I'm not willing to ruin people's day or lives over the word of a dog.
- njharman 11y agoAll this doesn't matter until the root cause is fixed. Police will find a way. The root cause* is mentioned in the article, property seizure laws. As long as there is monetary incentive Police will be compelled to violate the constitution. * The real root cause is the criminalization of recreational drugs. But ending Prohibition is basically a non-starter.
- nosuchthing 11y agoMerits and repercussions of drug prohibition aside, the fact that the dogs can signal a false positive on command seems unsettling.
- jacobheller 11y agoHere's the full text of the case for those interested: https://casetext.com/case/united-states-v-bentley-13 https://casetext.com/case/united-states-v-bentley-13