4 ms·

Any time I see "accuracy" reported as a single number, I assume it's B.S. and "for marketing purposes only". If you're serious, give me the precision and recall

Any time I see "accuracy" reported as a single number, I assume it's B.S. and "for marketing purposes only". If you're serious, give me the precision and recall.

the_snooze 5y ago
>If you're serious, give me the precision and recall. The thing is decision-makers have no idea (and no care to know) what those are.
- wcerfgba 5y ago
  Maybe these people shouldn't be decision-makers then. We need both greater transparency in decision-making processes, and greater education amongst the population about statistics, causal inference, and the scientific method. In order for the citizenry to hold decision-makers to account and obtain a higher standard we need: 1. Citizens to have the time and motivation to seek out and examine primary evidence. 2. Citizens to have the skills to understand and critique that evidence. 3. Robust mechanisms to allow citizens to replace people in power.
  - kube-system 5y ago
    This is an inherent problem with the specialization of labor. Those who run police departments have more relevant specializations to their job role than information science. This isn't a problem with statistics or science in particular -- it's an issue with every specialization. This is only going to get worse as popular opinion of the role of higher education is starting to shift from a more liberal-arts focus to a job-placement focus.
- anigbrowl 5y ago
  Correct and that's unlikely to change, but it should be grounds to throw out the prosecution. Yes, this will make it harder to convict real criminals. Too bad.
drc500free 5y ago
Precision is an attribute of the deployed system. The same tech in an affluent suburb will have different precision deployed in an inner city. Ran into this exact problem in biometrics, where NIST publishes tech error rates like FNMR and FMR, and system owners don't even know precision is a thing. They expect the vendor to tell them if it will work, when it depends how many true and false events you throw at the system.
abstrakraft 5y ago
I've never understood why ML folks insist on using precision, which depends on the base rate. Why do they not use probability of false alarm as in traditional detection theory?
- deleted 5y ago
  [deleted]
- alexilliamson 5y ago
  It doesn't really depend on the base rate though. The idea is that a high precision model will have all the targets at one end of the distribution, regardless of base rate.
  - abstrakraft 5y ago
    Precision is P(T|R), where R is the event that a thing is reported, and T is the event that the thing is true. By Bayes' rule, P(T|R) = P(T)/(1+Pfa/Pd). So in terms of the traditional statistics Pd and Pfa, it is proportional to the base rate.

Any time I see "accuracy" reported as a single number, I assume it's B.S. and "for marketing purposes only". If you're serious, give me the precision and recall.

the_snooze 5y ago

>If you're serious, give me the precision and recall. The thing is decision-makers have no idea (and no care to know) what those are.

wcerfgba 5y ago
Maybe these people shouldn't be decision-makers then. We need both greater transparency in decision-making processes, and greater education amongst the population about statistics, causal inference, and the scientific method. In order for the citizenry to hold decision-makers to account and obtain a higher standard we need: 1. Citizens to have the time and motivation to seek out and examine primary evidence. 2. Citizens to have the skills to understand and critique that evidence. 3. Robust mechanisms to allow citizens to replace people in power.
- kube-system 5y ago
  This is an inherent problem with the specialization of labor. Those who run police departments have more relevant specializations to their job role than information science. This isn't a problem with statistics or science in particular -- it's an issue with every specialization. This is only going to get worse as popular opinion of the role of higher education is starting to shift from a more liberal-arts focus to a job-placement focus.
anigbrowl 5y ago
Correct and that's unlikely to change, but it should be grounds to throw out the prosecution. Yes, this will make it harder to convict real criminals. Too bad.

drc500free 5y ago

Precision is an attribute of the deployed system. The same tech in an affluent suburb will have different precision deployed in an inner city. Ran into this exact problem in biometrics, where NIST publishes tech error rates like FNMR and FMR, and system owners don't even know precision is a thing. They expect the vendor to tell them if it will work, when it depends how many true and false events you throw at the system.

abstrakraft 5y ago

I've never understood why ML folks insist on using precision, which depends on the base rate. Why do they not use probability of false alarm as in traditional detection theory?

deleted 5y ago
[deleted]
alexilliamson 5y ago
It doesn't really depend on the base rate though. The idea is that a high precision model will have all the targets at one end of the distribution, regardless of base rate.
- abstrakraft 5y ago
  Precision is P(T|R), where R is the event that a thing is reported, and T is the event that the thing is true. By Bayes' rule, P(T|R) = P(T)/(1+Pfa/Pd). So in terms of the traditional statistics Pd and Pfa, it is proportional to the base rate.