10 ms·

The dog isn't even "as accurate as a coin flip". Given a 93% alert rate, regardless of the subsequent true positive rate, information gain from an alert is app

The dog isn't even "as accurate as a coin flip". Given a 93% alert rate, regardless of the subsequent true positive rate, information gain from an alert is approximately zero. Edit: "accurate as a coin flip" is a bad construction, but I wouldn't exactly expect better from a newspaper. The information gain from a device that activates randomly 50% of the time, 0% of the time, and 100% of the time are all the same, which is to say zero.

JoeAltmaier 11y ago
Be fair. False positives and false negatives are very different things. A drug dog that finds drugs even sometimes, is still useful. Imagine if it was bombs they were sniffing for. We'd be happy if they found even one more bomb.
- gruez 11y ago
  >A drug dog that finds drugs even sometimes, is still useful. Why even bother with the dog? why not just do random searches? Oh wait, because that would be unconstitutional. The dog exists solely to justify otherwise unconstitutional searches.
  - JoeAltmaier 11y ago
    Ignoring the fact that false positives don't preclude real positives...it works. People are fond of cute doggies, and often their resentment evaporates.
    - ossreality 11y ago
      Oh boy, you've gotta be joking, right?
    - zajd 11y ago
      [citation needed]
    - elektromekatron 11y ago
      False positives only don't preclude real positives if time and resources are not constrained. Time and resources are always constrained.
- elektromekatron 11y ago
  If it was explosives and a dog got a 93% alert rate irrespective of there being any explosives, then that would be useless and nobody in their right mind would use that dog, as it would have them search 93% of everything. At that point you might as well give up on faffing about with a sniffer dog and just search the remaining 7%.
- pavel_lishin 11y ago
  Then we should simply search 100% of the bags. Or maybe just 93%.
protonfish 11y ago
If I had a coin that showed heads 93% of the time, I'd think it might be rigged.
- DSMan195276 11y ago
  IMO, it doesn't seem remotely comparable to a coin flip because the odds aren't 50/50. The dogs are only brought out in certain circumstances when they think a positive might be possible, meaning that lots of people who would be negatives aren't tested and thus not included in that 93%. The cases making up the 93% only consist of people whom they thought may get a positive, meaning the sample is biased to having lots of positives. I'd compare it to the fact that lots of very hard certification tests actually have fairly high pass-rates (80% or above), simply because people who don't think they could pass don't take the test to begin with - The 80% pass-rate is not representative of the average person's chances of passing the test, nor the difficulty of the test itself. What's really important is the percentage of those positives which were true positives, vs. false positives. In this case, you would expect the number of positives to be high - The number of true positives vs. false positives would tell you how accurate the dog actually was in the positive case. If this number is also high, then the dog is doing a reasonable job. Of course, to derive a real argument from this, you'd need to know the number of true and false negatives as well, but obviously we don't have that information.
- slapresta 11y ago
  Sure, because a coin is evaluated on its capacity to map to an even distribution of outcomes. Drug possessions, however, are almost surely not an even distribution. If, say, 95% of the people that the dog evaluates have drugs, then raising an alarm 93% of the time is probably pretty good. In other words: If we wanted a drug dog to be as close as possible to a coin flip, we'd just flip a coin instead.
gburt 11y ago
This isn't being fair. I don't really think the post title is fair either. That 93% isn't comparable to the 50% you're probably thinking for the coin flip, or even the 0/100% mentioned in your edit. --- 93% alert rate means: 93% = false positive + true positive 7% = true negative + false negative --- We're given "His success rate was much lower, at 59 percent." Assuming that means that 59% are true positives, we know the false positive rate is 34%. Without data on how many passengers are carrying drugs, we can't say anything about the rest of that result. A plausible world where the drug dog may be valuable is when he unquestionably rules out that 7% of the population that he doesn't alert on, as in false negative = 0. I believe the reason this number is so high (59%) is that the officer has already used his suspicion to determine filter out most people. This makes the value of the "ruling out" plausibility even more valuable: the drug dog reduces searches compared to trusting the cop. Ideally, they'd provide the receiver operating characteristics curve [0] or the confusion matrix [1]. [0] https://en.wikipedia.org/wiki/Receiver_operating_characteristic https://en.wikipedia.org/wiki/Receiver_operating_characteris... [1] https://en.wikipedia.org/wiki/Confusion_matrix https://en.wikipedia.org/wiki/Confusion_matrix
- nickff 11y ago
  The real question is whether the dog is better than the officer at guessing who has drugs. If the dog is much better than the officer, it would suggest that a dog 'alert' should be given more weight than the officer's guess. If the dog is no better than the officer, the dog is useless. I say this because it is possible (and indeed likely) that the officer is also better at guessing who has drugs than a coin toss. Full Disclosure: I have never used or trafficked contraband, but I do not support random searches or the use of drug dogs in public without a warrant, regardless of their effectiveness.
  - gburt 11y ago
    I have no opinion on the legal and ethical issues, I was commenting only on the classifier. I don't think your first question is appropriately described as "the real question" though, as one of the coolest results of machine learning is that there are techniques for combining multiple bad classifiers [0] in to one really good one. Even more so, I think that is actually what is happening here. An officer is acting as a tier 1 filter, and then the drug dog is acting as a tier 2 filter, further reducing the number of false positives, which in a "legal and ethical" sense, we generally feel are worse than false negatives. [0] https://en.wikipedia.org/wiki/Ensemble_learning https://en.wikipedia.org/wiki/Ensemble_learning
merrywhether 11y ago
I think the coin-flip analogy comes from the idea that once an officer thought you had drugs, he/she was about 55% right (not accounting for false negatives). The dog alerted most of the time, thus allowing for search to confirm or deny the suspicions. Thus the dog gave no real information and the officer was the coin, but the reporter conflated these success rates.

JoeAltmaier 11y ago

Be fair. False positives and false negatives are very different things. A drug dog that finds drugs even sometimes, is still useful. Imagine if it was bombs they were sniffing for. We'd be happy if they found even one more bomb.

gruez 11y ago
>A drug dog that finds drugs even sometimes, is still useful. Why even bother with the dog? why not just do random searches? Oh wait, because that would be unconstitutional. The dog exists solely to justify otherwise unconstitutional searches.
- JoeAltmaier 11y ago
  Ignoring the fact that false positives don't preclude real positives...it works. People are fond of cute doggies, and often their resentment evaporates.
  - ossreality 11y ago
    Oh boy, you've gotta be joking, right?
  - zajd 11y ago
    [citation needed]
  - elektromekatron 11y ago
    False positives only don't preclude real positives if time and resources are not constrained. Time and resources are always constrained.
elektromekatron 11y ago
If it was explosives and a dog got a 93% alert rate irrespective of there being any explosives, then that would be useless and nobody in their right mind would use that dog, as it would have them search 93% of everything. At that point you might as well give up on faffing about with a sniffer dog and just search the remaining 7%.
pavel_lishin 11y ago
Then we should simply search 100% of the bags. Or maybe just 93%.

protonfish 11y ago

If I had a coin that showed heads 93% of the time, I'd think it might be rigged.

DSMan195276 11y ago
IMO, it doesn't seem remotely comparable to a coin flip because the odds aren't 50/50. The dogs are only brought out in certain circumstances when they think a positive might be possible, meaning that lots of people who would be negatives aren't tested and thus not included in that 93%. The cases making up the 93% only consist of people whom they thought may get a positive, meaning the sample is biased to having lots of positives. I'd compare it to the fact that lots of very hard certification tests actually have fairly high pass-rates (80% or above), simply because people who don't think they could pass don't take the test to begin with - The 80% pass-rate is not representative of the average person's chances of passing the test, nor the difficulty of the test itself. What's really important is the percentage of those positives which were true positives, vs. false positives. In this case, you would expect the number of positives to be high - The number of true positives vs. false positives would tell you how accurate the dog actually was in the positive case. If this number is also high, then the dog is doing a reasonable job. Of course, to derive a real argument from this, you'd need to know the number of true and false negatives as well, but obviously we don't have that information.
slapresta 11y ago
Sure, because a coin is evaluated on its capacity to map to an even distribution of outcomes. Drug possessions, however, are almost surely not an even distribution. If, say, 95% of the people that the dog evaluates have drugs, then raising an alarm 93% of the time is probably pretty good. In other words: If we wanted a drug dog to be as close as possible to a coin flip, we'd just flip a coin instead.

gburt 11y ago

This isn't being fair. I don't really think the post title is fair either. That 93% isn't comparable to the 50% you're probably thinking for the coin flip, or even the 0/100% mentioned in your edit. --- 93% alert rate means: 93% = false positive + true positive 7% = true negative + false negative --- We're given "His success rate was much lower, at 59 percent." Assuming that means that 59% are true positives, we know the false positive rate is 34%. Without data on how many passengers are carrying drugs, we can't say anything about the rest of that result. A plausible world where the drug dog may be valuable is when he unquestionably rules out that 7% of the population that he doesn't alert on, as in false negative = 0. I believe the reason this number is so high (59%) is that the officer has already used his suspicion to determine filter out most people. This makes the value of the "ruling out" plausibility even more valuable: the drug dog reduces searches compared to trusting the cop. Ideally, they'd provide the receiver operating characteristics curve [0] or the confusion matrix [1]. [0] https://en.wikipedia.org/wiki/Receiver_operating_characteristic https://en.wikipedia.org/wiki/Receiver_operating_characteris... [1] https://en.wikipedia.org/wiki/Confusion_matrix https://en.wikipedia.org/wiki/Confusion_matrix

nickff 11y ago
The real question is whether the dog is better than the officer at guessing who has drugs. If the dog is much better than the officer, it would suggest that a dog 'alert' should be given more weight than the officer's guess. If the dog is no better than the officer, the dog is useless. I say this because it is possible (and indeed likely) that the officer is also better at guessing who has drugs than a coin toss. Full Disclosure: I have never used or trafficked contraband, but I do not support random searches or the use of drug dogs in public without a warrant, regardless of their effectiveness.
- gburt 11y ago
  I have no opinion on the legal and ethical issues, I was commenting only on the classifier. I don't think your first question is appropriately described as "the real question" though, as one of the coolest results of machine learning is that there are techniques for combining multiple bad classifiers [0] in to one really good one. Even more so, I think that is actually what is happening here. An officer is acting as a tier 1 filter, and then the drug dog is acting as a tier 2 filter, further reducing the number of false positives, which in a "legal and ethical" sense, we generally feel are worse than false negatives. [0] https://en.wikipedia.org/wiki/Ensemble_learning https://en.wikipedia.org/wiki/Ensemble_learning

merrywhether 11y ago

I think the coin-flip analogy comes from the idea that once an officer thought you had drugs, he/she was about 55% right (not accounting for false negatives). The dog alerted most of the time, thus allowing for search to confirm or deny the suspicions. Thus the dog gave no real information and the officer was the coin, but the reporter conflated these success rates.