3 ms·

Regarding precision/recall, I've a background in financial econometrics and this is the first time I encounter the terms.

by Bootvis 10y ago

halflings 10y ago
That's OK. The article was talking about somebody interviewing for a search-related position (where precision and recall are usually what you are optimizing for). I guess they might be called differently in econometrics?
kenjackson 10y ago
I think the problem is that certain subfields use different terminology to mean similar or identical concepts. For example, while I'm in software, I tend to hear the terms sensitivity and specificity. They are historically medical terms. They aren't identical to recall/precision, but I think you can derive one set from the other.
- Bootvis 10y ago
  That certainly seems likely and a good thing to keep in mind when you're giving or taking an interview!
- antognini 10y ago
  The fundamental thing to know is the confusion matrix. There are about a dozen terms for various descriptors of the matrix, but they all can be calculated if you know the confusion matrix. The Wikipedia page has a great table to describe them all: https://en.wikipedia.org/wiki/Confusion_matrix https://en.wikipedia.org/wiki/Confusion_matrix You can see from that that sensitivity and recall are the same thing, but specificity and precision are not.
jknoepfler 10y ago
It's literally the first thing you learn in data science / machine learning coursework about evaluating model performance. It would probably be better to ask the candidate to whiteboard a set of metrics for evaluating model performance rather than ask for the definition of a pair of words, but the concept is practically the for-loop of data science. Edit: note that I'm not saying you need this to add roi as an analyst for a business!
- Bootvis 10y ago
  I haven't taken a lot of data science classes but I'm not sure that's true. If you start with linear regression the mean squared error would make more sense. I actually searched through "The Elements of Statistical Learning" and the word 'recall' is not used in this sense at all.
  - mjn 10y ago
    The jargon does vary by subfield and community, along with the actual measures used (sometimes it's just a different name, but sometimes practices are different as well). Precision/recall are terms from information retrieval that migrated into the CS-flavored portion of machine learning, but are not as common in the stats-flavored portion of ML, in part because some statisticians consider them poor measures of classifier performance [1]. Hence they don't show up in the Hastie/Tibshirani/Friedman book you mention, which is written by three authors solidly on the stats side of ML. It does occasionally mention some equivalent terms, e.g. Ctrl+F'ing through a PDF, I see that in Chapter 9 it borrows the sensitivity/specificity metrics used in medical statistics, where sensitivity is a synonym for recall (but specificity is not the same thing as precision). It looks like the book more often uses ROC curves, though, which have their own adherents and detractors. [1] This paper is the one that most often gets cited as background by people who don't like recall/precision as metrics: http://dspace2.flinders.edu.au/xmlui/bitstream/handle/2328/27165/Powers%20Evaluation.pdf http://dspace2.flinders.edu.au/xmlui/bitstream/handle/2328/2...
  - mehaveaccount 10y ago
    People don't pay for linear regressions. They pay for discrete things: what is my best option among my three clear courses of action. Linear regression can be a tiny piece of a larger argument in favor or against one option or the other, but that alone doesn't make money.
    - Bootvis 10y ago
      That's obvious but not at all what I responded to in my post. I responded to the claim that ML courses start with the definition of precision and recall. In my admittedly limited experience those courses start with linear regression and mean squared errors. After that, there is so much generalization possible and that doesn't include precision/recall. You make money by solving someone's problems, making money by stating definitions is only done on TV quizzes.

Regarding precision/recall, I've a background in financial econometrics and this is the first time I encounter the terms.

by Bootvis 10y ago

Regarding precision/recall, I've a background in financial econometrics and this is the first time I encounter the terms.

halflings 10y ago

That's OK. The article was talking about somebody interviewing for a search-related position (where precision and recall are usually what you are optimizing for). I guess they might be called differently in econometrics?

kenjackson 10y ago

I think the problem is that certain subfields use different terminology to mean similar or identical concepts. For example, while I'm in software, I tend to hear the terms sensitivity and specificity. They are historically medical terms. They aren't identical to recall/precision, but I think you can derive one set from the other.

Bootvis 10y ago
That certainly seems likely and a good thing to keep in mind when you're giving or taking an interview!
antognini 10y ago
The fundamental thing to know is the confusion matrix. There are about a dozen terms for various descriptors of the matrix, but they all can be calculated if you know the confusion matrix. The Wikipedia page has a great table to describe them all: https://en.wikipedia.org/wiki/Confusion_matrix https://en.wikipedia.org/wiki/Confusion_matrix You can see from that that sensitivity and recall are the same thing, but specificity and precision are not.

jknoepfler 10y ago

It's literally the first thing you learn in data science / machine learning coursework about evaluating model performance. It would probably be better to ask the candidate to whiteboard a set of metrics for evaluating model performance rather than ask for the definition of a pair of words, but the concept is practically the for-loop of data science. Edit: note that I'm not saying you need this to add roi as an analyst for a business!

Bootvis 10y ago
I haven't taken a lot of data science classes but I'm not sure that's true. If you start with linear regression the mean squared error would make more sense. I actually searched through "The Elements of Statistical Learning" and the word 'recall' is not used in this sense at all.
- mjn 10y ago
  The jargon does vary by subfield and community, along with the actual measures used (sometimes it's just a different name, but sometimes practices are different as well). Precision/recall are terms from information retrieval that migrated into the CS-flavored portion of machine learning, but are not as common in the stats-flavored portion of ML, in part because some statisticians consider them poor measures of classifier performance [1]. Hence they don't show up in the Hastie/Tibshirani/Friedman book you mention, which is written by three authors solidly on the stats side of ML. It does occasionally mention some equivalent terms, e.g. Ctrl+F'ing through a PDF, I see that in Chapter 9 it borrows the sensitivity/specificity metrics used in medical statistics, where sensitivity is a synonym for recall (but specificity is not the same thing as precision). It looks like the book more often uses ROC curves, though, which have their own adherents and detractors. [1] This paper is the one that most often gets cited as background by people who don't like recall/precision as metrics: http://dspace2.flinders.edu.au/xmlui/bitstream/handle/2328/27165/Powers%20Evaluation.pdf http://dspace2.flinders.edu.au/xmlui/bitstream/handle/2328/2...
- mehaveaccount 10y ago
  People don't pay for linear regressions. They pay for discrete things: what is my best option among my three clear courses of action. Linear regression can be a tiny piece of a larger argument in favor or against one option or the other, but that alone doesn't make money.
  - Bootvis 10y ago
    That's obvious but not at all what I responded to in my post. I responded to the claim that ML courses start with the definition of precision and recall. In my admittedly limited experience those courses start with linear regression and mean squared errors. After that, there is so much generalization possible and that doesn't include precision/recall. You make money by solving someone's problems, making money by stating definitions is only done on TV quizzes.