4 ms·

Two things are missing here. The first the the null model result, if a model simply predicted the most common class, we would get accuracy of 0.649. So the CAR

Two things are missing here. The first the the null model result, if a model simply predicted the most common class, we would get accuracy of 0.649. So the CART model has increased performance by ~14%. A good start. The second is that in many scenarios in business and especially medical scenarios, you also need to know sensitivity (how good are we at predicting a person with diabetes does have diabetes) and specificity (how good are we at predicting a person without diabetes does NOT have it). Always calculate these metrics and a confusion matrix for binary classification. If a simple model predicted you had low vitamin-C, a doctor in my country would simply prescribe it even if the model had terrible specificity because the treatment is harmless and cheaper than a blood test. On the other hand if a simple model predicted cancer, they would refer you a variety of scans and tests and not make a diagnosis until they were quite sure.

Phemist 4y ago
Take it one step further. A test _never_ has a binary outcome, this is _always_ the result of some score thresholded against something else (could be another score). Sensitivity and specificity are a function of your data to validate the test, as well as this tunable threshold! The tunability of the threshold should be a core concern in the reporting of the performance of a test. Look at ROC or DET curve and their performance measures (EER, False accept @ some given False reject rate, or False Reject @ given False Accept rate, AUC). This allows applications to understand the costs and benefits of tuning this thresholding parameter, and understanding what happens when you accept a higher false reject rate, than a higher false accept rate. Just reporting a specificity and sensitivity totally misses the point that the threshold is tunable and different applications can have fundamentally different needs for this parameter. The vitamin-c case is a case where false accepts (model predicts I have some deficiency where in reality I do not) are essentially cost-free, so you can take the threshold where FAR==100%, and then report on the FRR at this point
- teruakohatu 4y ago
  > Take it one step further. A test _never_ has a binary outcome, this is _always_ the result of some score thresholded against something else (could be another score). By test do you mean model/method? Because not all models have a score eg. discrete classifiers. CART classifiers are simply large if/else statements. There is no tunable threshold and while some people sort of draw something that looks like a ROC, it's not quite as useful. But of course you are right, in the real world you would be comparing many different models built with different methods and hyper params using a variety of metrics.
  - igorkraw 4y ago
    Unless you are using some very exotic method i don't know, you are wrong about discrete classifier not having a score. Everything I know extracts an integer from a vector of scores over options, with the most common being argmax over likelihood trained with CrossEntropy
    - fjkdlsjflkds 4y ago
      Nope. Decision trees typically output binary (or multinomial) values directly. Sure, you could tune the node thresholds (or optimize them during tree construction), to vary the trade-off between specificity and selectivity, but this may not be straightforward for deep trees or those with many leaf nodes, and it will never be as simple as simply tuning a single threshold (unless you are using decision stumps, rather than decision trees).
      - igorkraw 4y ago
        How do you construct your decision tree though? All methods I'm familiar with use a criterion which comes back down to using a score, which also ties down to the set theoretical foundations of probability theory: assign a score and then normalise to to obtain a probability.
      - radarsat1 4y ago
        > Decision trees typically output binary (or multinomial) values directly. Is there some Bayesian version of decision trees where you model each decision in the tree as a beta distribution or something? Then you could simply choose the leaf or path with the highest confidence. I just tried a cursory search about this but came up with a lot of theory justifying the use of thresholds as an approximation to a Bayesian interpretation, but I couldn't find any actual implementations that propagate a confidence measure or probability distribution through the tree, so I don't know if this is standard methodology. I can imagine having confidence in the final decision to also be useful for model ensembling. Curious what the right way to do this might be, or to know if there are no advantages vs. simple threshold trees.

Phemist 4y ago

Take it one step further. A test _never_ has a binary outcome, this is _always_ the result of some score thresholded against something else (could be another score). Sensitivity and specificity are a function of your data to validate the test, as well as this tunable threshold! The tunability of the threshold should be a core concern in the reporting of the performance of a test. Look at ROC or DET curve and their performance measures (EER, False accept @ some given False reject rate, or False Reject @ given False Accept rate, AUC). This allows applications to understand the costs and benefits of tuning this thresholding parameter, and understanding what happens when you accept a higher false reject rate, than a higher false accept rate. Just reporting a specificity and sensitivity totally misses the point that the threshold is tunable and different applications can have fundamentally different needs for this parameter. The vitamin-c case is a case where false accepts (model predicts I have some deficiency where in reality I do not) are essentially cost-free, so you can take the threshold where FAR==100%, and then report on the FRR at this point

teruakohatu 4y ago
> Take it one step further. A test _never_ has a binary outcome, this is _always_ the result of some score thresholded against something else (could be another score). By test do you mean model/method? Because not all models have a score eg. discrete classifiers. CART classifiers are simply large if/else statements. There is no tunable threshold and while some people sort of draw something that looks like a ROC, it's not quite as useful. But of course you are right, in the real world you would be comparing many different models built with different methods and hyper params using a variety of metrics.
- igorkraw 4y ago
  Unless you are using some very exotic method i don't know, you are wrong about discrete classifier not having a score. Everything I know extracts an integer from a vector of scores over options, with the most common being argmax over likelihood trained with CrossEntropy
  - fjkdlsjflkds 4y ago
    Nope. Decision trees typically output binary (or multinomial) values directly. Sure, you could tune the node thresholds (or optimize them during tree construction), to vary the trade-off between specificity and selectivity, but this may not be straightforward for deep trees or those with many leaf nodes, and it will never be as simple as simply tuning a single threshold (unless you are using decision stumps, rather than decision trees).
    - igorkraw 4y ago
      How do you construct your decision tree though? All methods I'm familiar with use a criterion which comes back down to using a score, which also ties down to the set theoretical foundations of probability theory: assign a score and then normalise to to obtain a probability.
    - radarsat1 4y ago
      > Decision trees typically output binary (or multinomial) values directly. Is there some Bayesian version of decision trees where you model each decision in the tree as a beta distribution or something? Then you could simply choose the leaf or path with the highest confidence. I just tried a cursory search about this but came up with a lot of theory justifying the use of thresholds as an approximation to a Bayesian interpretation, but I couldn't find any actual implementations that propagate a confidence measure or probability distribution through the tree, so I don't know if this is standard methodology. I can imagine having confidence in the final decision to also be useful for model ensembling. Curious what the right way to do this might be, or to know if there are no advantages vs. simple threshold trees.