6 ms·

This isn't being fair. I don't really think the post title is fair either. That 93% isn't comparable to the 50% you're probably thinking for the coin flip, or

by gburt 11y ago

This isn't being fair. I don't really think the post title is fair either. That 93% isn't comparable to the 50% you're probably thinking for the coin flip, or even the 0/100% mentioned in your edit. --- 93% alert rate means: 93% = false positive + true positive 7% = true negative + false negative --- We're given "His success rate was much lower, at 59 percent." Assuming that means that 59% are true positives, we know the false positive rate is 34%. Without data on how many passengers are carrying drugs, we can't say anything about the rest of that result. A plausible world where the drug dog may be valuable is when he unquestionably rules out that 7% of the population that he doesn't alert on, as in false negative = 0. I believe the reason this number is so high (59%) is that the officer has already used his suspicion to determine filter out most people. This makes the value of the "ruling out" plausibility even more valuable: the drug dog reduces searches compared to trusting the cop. Ideally, they'd provide the receiver operating characteristics curve [0] or the confusion matrix [1]. [0] https://en.wikipedia.org/wiki/Receiver_operating_characteristic https://en.wikipedia.org/wiki/Receiver_operating_characteris... [1] https://en.wikipedia.org/wiki/Confusion_matrix https://en.wikipedia.org/wiki/Confusion_matrix

nickff 11y ago
The real question is whether the dog is better than the officer at guessing who has drugs. If the dog is much better than the officer, it would suggest that a dog 'alert' should be given more weight than the officer's guess. If the dog is no better than the officer, the dog is useless. I say this because it is possible (and indeed likely) that the officer is also better at guessing who has drugs than a coin toss. Full Disclosure: I have never used or trafficked contraband, but I do not support random searches or the use of drug dogs in public without a warrant, regardless of their effectiveness.
- gburt 11y ago
  I have no opinion on the legal and ethical issues, I was commenting only on the classifier. I don't think your first question is appropriately described as "the real question" though, as one of the coolest results of machine learning is that there are techniques for combining multiple bad classifiers [0] in to one really good one. Even more so, I think that is actually what is happening here. An officer is acting as a tier 1 filter, and then the drug dog is acting as a tier 2 filter, further reducing the number of false positives, which in a "legal and ethical" sense, we generally feel are worse than false negatives. [0] https://en.wikipedia.org/wiki/Ensemble_learning https://en.wikipedia.org/wiki/Ensemble_learning
  - nickff 11y ago
    >" An officer is acting as a tier 1 filter, and then the drug dog is acting as a tier 2 filter, further reducing the number of false positives, which in a "legal and ethical" sense, we generally feel are worse than false negatives." If the officer is acting as a filter, then it seems to me like this is a search, and not allowable, because it would be subject to all the usual objections to random searches. You might be right, but if that is the only way for these searches to be better than the officer's guesses, it seems that drug dogs would not be allowed to 'sniff' in public without warrants.
- LordKano 11y ago
  If the dog is much better than the officer, it would suggest that a dog 'alert' should be given more weight than the officer's guess. If the dog is no better than the officer, the dog is useless. A very big problem is that drug dogs will alert when the handler gives it subtle clues. This is done to get probable cause for a search when there is nothing more than an officer's hunch or just a desire to hassle a particular individual.
  - mejari 11y ago
    This is something I see claimed a lot. I'd be very interested in any information you have about this. I find it very unlikely this kind of training is part of the official training, so I'm imagining it's some kind of unintentional conditioning? Or is it done on purpose by the officers?
    - lukeschlather 11y ago
      This is covered in the article, and it's quite straightforward. The handler was giving the dog treats for alerting, rather than giving treats for finding drugs. The primary trouble I see is that there is no penalty for re-training dogs like this.
      - mejari 11y ago
        The article only discusses that the handler gives the dog treats for alerting, it doesn't go into the motivations about that, whether they're because the officers aren't considering the Pavlovian conditioning involved and are accidentally reinforcing such behavior or if it's intentional.
        kd0amg 11y ago
        Does the motivation matter when we're talking about whether it establishes sufficient justification for searching someone's car? After a clear flaw is pointed out, a response of "We didn't mean to ruin the statistical power of this test" wouldn't do much to restore my confidence in its results.
brobdingnagian 11y ago
The dog has low precision and high recall. The dogs are capable of high precision and high recall, but aren't being trained to do that because other incentives are at play. I found it impressive that the dog that alerts 93% of the time was correct 40% of the time. That means the police are doing what seems like a good job of identifying targets that actually have drugs. With those kind of odds, you'd probably want to take a closer look at everyone just to make sure you aren't letting people with drugs get away (false negatives).
- deleted 11y ago
  [deleted]
okasaki 11y ago
Erm, let's say 10% of the people on whom drug dogs are called have drugs. Then for 1000 stopped, you'll have 0.59 * 0.1 * 1000 = 59 true positives and ... 0.34 * 0.9 * 1000 = 306 false positives. That's pretty terrible.
- harryh 11y ago
  Your math is wrong. The article indicates that of all the people the dog indicated had drugs 59% of them actually did. So for 1000 people whom drug dogs are called on 930 the dog indicated had drugs. Of those 549 were correct and 381 were false positives.
jeremysmyth 11y ago
It's in the article. The 93% in that study includes a 59% true positive (of 93%, presumably) success rate. We can't know how many true or false negatives there are because those searches aren't actually carried out, so out of every hundred cars stopped for a dogsniff, 55% (59% of 93%) are found to be carrying drugs. More worryingly, that also means that out of every hundred cars stopped, 38 are wastefully searched by police (after getting probable cause from the positive dog alert) without finding drugs. From the article, a search takes about 45 minutes and carries a bunch of other risks for occupants of suspected cars (e.g. civil forfeiture, overenthusiastically trigger-happy LEOs), so this is pretty bad situation, not just in wasted time and cost, for both the car occupants and the police force. One slightly positive thing I'll take from this: Over 50% of cars that are dogged have drugs, and I sincerely doubt that over 50% of cars (on average) have drugs, so in this particular study it appears that police have a better than random chance of guessing which cars have drugs. It's not a very positive thing though, because civilized countries that operate under the rule of law cannot permit law enforcement to operate on guesswork. This is underlined by the fact that the article mentions that the false positive rate rose to something like 80% for latino drivers.
- gburt 11y ago
  Sorry - I edited my post multiple times after posting it to clean it up and increase quality. Some of your response (especially the first paragraph) is now confusing. That was my fault. I agree with you mostly - especially on the discussion of what "probable cause" should be defined as and what constitutes a "random" search, as just an example of some of the ethical issues of interest here. That is a debate to have and even one I think this community is well equipped to discuss, but it was not something I was trying to comment on. I was only making the point that it wasn't fair to jump on the 93% as though it was representative of the problem. I am not taking any ethical or defensive position of drug dogs here.
mikeash 11y ago
It's plausible that the 7% is all true negatives. It's also plausible that the 7% is carrying drugs at the same rate as the 93%, and the drug dog is doing nothing at all. If that is the case, then flipping a biased coin would be just as effective as the drug dog.
deleted 11y ago
[deleted]

by gburt 11y ago

nickff 11y ago

The real question is whether the dog is better than the officer at guessing who has drugs. If the dog is much better than the officer, it would suggest that a dog 'alert' should be given more weight than the officer's guess. If the dog is no better than the officer, the dog is useless. I say this because it is possible (and indeed likely) that the officer is also better at guessing who has drugs than a coin toss. Full Disclosure: I have never used or trafficked contraband, but I do not support random searches or the use of drug dogs in public without a warrant, regardless of their effectiveness.

gburt 11y ago
I have no opinion on the legal and ethical issues, I was commenting only on the classifier. I don't think your first question is appropriately described as "the real question" though, as one of the coolest results of machine learning is that there are techniques for combining multiple bad classifiers [0] in to one really good one. Even more so, I think that is actually what is happening here. An officer is acting as a tier 1 filter, and then the drug dog is acting as a tier 2 filter, further reducing the number of false positives, which in a "legal and ethical" sense, we generally feel are worse than false negatives. [0] https://en.wikipedia.org/wiki/Ensemble_learning https://en.wikipedia.org/wiki/Ensemble_learning
- nickff 11y ago
  >" An officer is acting as a tier 1 filter, and then the drug dog is acting as a tier 2 filter, further reducing the number of false positives, which in a "legal and ethical" sense, we generally feel are worse than false negatives." If the officer is acting as a filter, then it seems to me like this is a search, and not allowable, because it would be subject to all the usual objections to random searches. You might be right, but if that is the only way for these searches to be better than the officer's guesses, it seems that drug dogs would not be allowed to 'sniff' in public without warrants.
LordKano 11y ago
If the dog is much better than the officer, it would suggest that a dog 'alert' should be given more weight than the officer's guess. If the dog is no better than the officer, the dog is useless. A very big problem is that drug dogs will alert when the handler gives it subtle clues. This is done to get probable cause for a search when there is nothing more than an officer's hunch or just a desire to hassle a particular individual.
- mejari 11y ago
  This is something I see claimed a lot. I'd be very interested in any information you have about this. I find it very unlikely this kind of training is part of the official training, so I'm imagining it's some kind of unintentional conditioning? Or is it done on purpose by the officers?
  - lukeschlather 11y ago
    This is covered in the article, and it's quite straightforward. The handler was giving the dog treats for alerting, rather than giving treats for finding drugs. The primary trouble I see is that there is no penalty for re-training dogs like this.
    - mejari 11y ago
      The article only discusses that the handler gives the dog treats for alerting, it doesn't go into the motivations about that, whether they're because the officers aren't considering the Pavlovian conditioning involved and are accidentally reinforcing such behavior or if it's intentional.
      - kd0amg 11y ago
        Does the motivation matter when we're talking about whether it establishes sufficient justification for searching someone's car? After a clear flaw is pointed out, a response of "We didn't mean to ruin the statistical power of this test" wouldn't do much to restore my confidence in its results.

brobdingnagian 11y ago

The dog has low precision and high recall. The dogs are capable of high precision and high recall, but aren't being trained to do that because other incentives are at play. I found it impressive that the dog that alerts 93% of the time was correct 40% of the time. That means the police are doing what seems like a good job of identifying targets that actually have drugs. With those kind of odds, you'd probably want to take a closer look at everyone just to make sure you aren't letting people with drugs get away (false negatives).

deleted 11y ago
[deleted]

okasaki 11y ago

Erm, let's say 10% of the people on whom drug dogs are called have drugs. Then for 1000 stopped, you'll have 0.59 * 0.1 * 1000 = 59 true positives and ... 0.34 * 0.9 * 1000 = 306 false positives. That's pretty terrible.

harryh 11y ago
Your math is wrong. The article indicates that of all the people the dog indicated had drugs 59% of them actually did. So for 1000 people whom drug dogs are called on 930 the dog indicated had drugs. Of those 549 were correct and 381 were false positives.

jeremysmyth 11y ago

It's in the article. The 93% in that study includes a 59% true positive (of 93%, presumably) success rate. We can't know how many true or false negatives there are because those searches aren't actually carried out, so out of every hundred cars stopped for a dogsniff, 55% (59% of 93%) are found to be carrying drugs. More worryingly, that also means that out of every hundred cars stopped, 38 are wastefully searched by police (after getting probable cause from the positive dog alert) without finding drugs. From the article, a search takes about 45 minutes and carries a bunch of other risks for occupants of suspected cars (e.g. civil forfeiture, overenthusiastically trigger-happy LEOs), so this is pretty bad situation, not just in wasted time and cost, for both the car occupants and the police force. One slightly positive thing I'll take from this: Over 50% of cars that are dogged have drugs, and I sincerely doubt that over 50% of cars (on average) have drugs, so in this particular study it appears that police have a better than random chance of guessing which cars have drugs. It's not a very positive thing though, because civilized countries that operate under the rule of law cannot permit law enforcement to operate on guesswork. This is underlined by the fact that the article mentions that the false positive rate rose to something like 80% for latino drivers.

gburt 11y ago
Sorry - I edited my post multiple times after posting it to clean it up and increase quality. Some of your response (especially the first paragraph) is now confusing. That was my fault. I agree with you mostly - especially on the discussion of what "probable cause" should be defined as and what constitutes a "random" search, as just an example of some of the ethical issues of interest here. That is a debate to have and even one I think this community is well equipped to discuss, but it was not something I was trying to comment on. I was only making the point that it wasn't fair to jump on the 93% as though it was representative of the problem. I am not taking any ethical or defensive position of drug dogs here.

mikeash 11y ago

It's plausible that the 7% is all true negatives. It's also plausible that the 7% is carrying drugs at the same rate as the 93%, and the drug dog is doing nothing at all. If that is the case, then flipping a biased coin would be just as effective as the drug dog.

deleted 11y ago

[deleted]