5 ms·

Another somewhat similar (in that the result is unintuitive) problem about false positives: Imagine that you have an HIV test that is 95% accurate (false posit

by rplnt 8y ago

Another somewhat similar (in that the result is unintuitive) problem about false positives: Imagine that you have an HIV test that is 95% accurate (false positive rate of 5%). and around 2% of tested population is infected with HIV. What is the probability that you actually have HIV when your test comes back positive? Is it 95%? Nope, actually just 29%. You do the test again. Numbers from wiki about the paradox: https://en.wikipedia.org/wiki/False_positive_paradox https://en.wikipedia.org/wiki/False_positive_paradox

nabla9 8y ago
Same issue comes up when you discuss effectiveness of ethnic profiling of Muslims/brown skinned people in airports. Sam Harris was unable to wrap his head around this and was sticking to his common sense intuition. Strong profiling is not mathematically optimal for discovering rare malfeasors http://www.pnas.org/content/106/6/1716.full?sid=3bc684ec-b593-41e9-b03e-2e3f32bc42b0 http://www.pnas.org/content/106/6/1716.full?sid=3bc684ec-b59...
- LeifCarrotson 8y ago
  The Sam Harris article referenced, for anyone else who was unfamiliar with the controversy: https://samharris.org/in-defense-of-profiling/ https://samharris.org/in-defense-of-profiling/
  - deleted 8y ago
    [deleted]
  - deleted 8y ago
    [deleted]
  - zouhair 8y ago
    Just wow.
- Bartweiss 8y ago
  And more recently, UK police have discovered that surveilling crowds with facial recognition software produces almost entirely false positives, even if the software is supposedly quite accurate at picking out criminals. Not that they care or plan to stop using it, of course. https://arstechnica.com/tech-policy/2018/05/uk-police-say-92-percent-false-positive-facial-recognition-is-no-big-deal/ https://arstechnica.com/tech-policy/2018/05/uk-police-say-92...
- nixarian 8y ago
  All you're saying is that any Muslim picked up is more likely to not be a criminal, which seems expected. The problem with "strong sampling", is, perhaps obviously, that they focus too strongly on the 'brown skinned people' and therefore might miss other criminals. The article you link seems to suggest some profiling is good, but that strong profiling AND random sampling are equally inefficient. "A mathematically optimal strategy would be 'square-root biased sampling,' the geometric mean between strong profiling and uniform sampling, with secondary screenings distributed broadly, although not uniformly, over the population."
Bartweiss 8y ago
Depressingly, this is apparently unfamiliar even to doctors who actually face this exact situation on a regular basis. Given multiple choice tests, they get the right rate at (or a bit below) random chance, and a large plurality think that the test accuracy equals the chance that a positive result is correct. Those numbers are from 2007 - given that I learned this result in an entry-level undergraduate stats course, I'd like to think things are improving. But I'm not especially optimistic, and even if they are it still means generations of practicing doctors weren't taught this. https://www.bbc.com/news/magazine-28166019 https://www.bbc.com/news/magazine-28166019
DanBC 8y ago
Gerd Gigerenzer has a good book called Reckoning With Risk that talks about this, and gives several real examples of people taking action after being given information by medical staff.
charmides 8y ago
Wouldn't you need to know the false negative rate too?

by rplnt 8y ago

nabla9 8y ago

Same issue comes up when you discuss effectiveness of ethnic profiling of Muslims/brown skinned people in airports. Sam Harris was unable to wrap his head around this and was sticking to his common sense intuition. Strong profiling is not mathematically optimal for discovering rare malfeasors http://www.pnas.org/content/106/6/1716.full?sid=3bc684ec-b593-41e9-b03e-2e3f32bc42b0 http://www.pnas.org/content/106/6/1716.full?sid=3bc684ec-b59...

LeifCarrotson 8y ago
The Sam Harris article referenced, for anyone else who was unfamiliar with the controversy: https://samharris.org/in-defense-of-profiling/ https://samharris.org/in-defense-of-profiling/
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
- zouhair 8y ago
  Just wow.
Bartweiss 8y ago
And more recently, UK police have discovered that surveilling crowds with facial recognition software produces almost entirely false positives, even if the software is supposedly quite accurate at picking out criminals. Not that they care or plan to stop using it, of course. https://arstechnica.com/tech-policy/2018/05/uk-police-say-92-percent-false-positive-facial-recognition-is-no-big-deal/ https://arstechnica.com/tech-policy/2018/05/uk-police-say-92...
nixarian 8y ago
All you're saying is that any Muslim picked up is more likely to not be a criminal, which seems expected. The problem with "strong sampling", is, perhaps obviously, that they focus too strongly on the 'brown skinned people' and therefore might miss other criminals. The article you link seems to suggest some profiling is good, but that strong profiling AND random sampling are equally inefficient. "A mathematically optimal strategy would be 'square-root biased sampling,' the geometric mean between strong profiling and uniform sampling, with secondary screenings distributed broadly, although not uniformly, over the population."

Bartweiss 8y ago

Depressingly, this is apparently unfamiliar even to doctors who actually face this exact situation on a regular basis. Given multiple choice tests, they get the right rate at (or a bit below) random chance, and a large plurality think that the test accuracy equals the chance that a positive result is correct. Those numbers are from 2007 - given that I learned this result in an entry-level undergraduate stats course, I'd like to think things are improving. But I'm not especially optimistic, and even if they are it still means generations of practicing doctors weren't taught this. https://www.bbc.com/news/magazine-28166019 https://www.bbc.com/news/magazine-28166019

DanBC 8y ago

Gerd Gigerenzer has a good book called Reckoning With Risk that talks about this, and gives several real examples of people taking action after being given information by medical staff.

charmides 8y ago

Wouldn't you need to know the false negative rate too?