4 ms·

Take it one step further. A test _never_ has a binary outcome, this is _always_ the result of some score thresholded against something else (could be another sc

Take it one step further. A test _never_ has a binary outcome, this is _always_ the result of some score thresholded against something else (could be another score). Sensitivity and specificity are a function of your data to validate the test, as well as this tunable threshold! The tunability of the threshold should be a core concern in the reporting of the performance of a test. Look at ROC or DET curve and their performance measures (EER, False accept @ some given False reject rate, or False Reject @ given False Accept rate, AUC). This allows applications to understand the costs and benefits of tuning this thresholding parameter, and understanding what happens when you accept a higher false reject rate, than a higher false accept rate. Just reporting a specificity and sensitivity totally misses the point that the threshold is tunable and different applications can have fundamentally different needs for this parameter. The vitamin-c case is a case where false accepts (model predicts I have some deficiency where in reality I do not) are essentially cost-free, so you can take the threshold where FAR==100%, and then report on the FRR at this point

teruakohatu 4y ago
> Take it one step further. A test _never_ has a binary outcome, this is _always_ the result of some score thresholded against something else (could be another score). By test do you mean model/method? Because not all models have a score eg. discrete classifiers. CART classifiers are simply large if/else statements. There is no tunable threshold and while some people sort of draw something that looks like a ROC, it's not quite as useful. But of course you are right, in the real world you would be comparing many different models built with different methods and hyper params using a variety of metrics.
- igorkraw 4y ago
  Unless you are using some very exotic method i don't know, you are wrong about discrete classifier not having a score. Everything I know extracts an integer from a vector of scores over options, with the most common being argmax over likelihood trained with CrossEntropy
  - fjkdlsjflkds 4y ago
    Nope. Decision trees typically output binary (or multinomial) values directly. Sure, you could tune the node thresholds (or optimize them during tree construction), to vary the trade-off between specificity and selectivity, but this may not be straightforward for deep trees or those with many leaf nodes, and it will never be as simple as simply tuning a single threshold (unless you are using decision stumps, rather than decision trees).
    - igorkraw 4y ago
      How do you construct your decision tree though? All methods I'm familiar with use a criterion which comes back down to using a score, which also ties down to the set theoretical foundations of probability theory: assign a score and then normalise to to obtain a probability.
    - radarsat1 4y ago
      > Decision trees typically output binary (or multinomial) values directly. Is there some Bayesian version of decision trees where you model each decision in the tree as a beta distribution or something? Then you could simply choose the leaf or path with the highest confidence. I just tried a cursory search about this but came up with a lot of theory justifying the use of thresholds as an approximation to a Bayesian interpretation, but I couldn't find any actual implementations that propagate a confidence measure or probability distribution through the tree, so I don't know if this is standard methodology. I can imagine having confidence in the final decision to also be useful for model ensembling. Curious what the right way to do this might be, or to know if there are no advantages vs. simple threshold trees.
      - fjkdlsjflkds 4y ago
        > Is there some Bayesian version of decision trees where you model each decision in the tree as a beta distribution or something? Then you could simply choose the leaf or path with the highest confidence. Sure... there's many things you could do "force" a decision tree to output a real scalar/vector (that you can then threshold): for example, you could use a regression tree where leafs are logistic/multinomial GLM. My point is that most types of decision trees (including CART decision trees) do not work like that by default (and they are considered pretty standard classifiers in ML literature), unlike what was claimed by igorkraw. > I can imagine having confidence in the final decision to also be useful for model ensembling. Curious what the right way to do this might be, or to know if there are no advantages vs. simple threshold trees. Sure, you could use ensembles of decision trees (e.g. random forests) to estimate a continuous value that perhaps (but not surely) correlates with "confidence" or "probability", which you can then threshold to obtain a binary value. But, again, this is beyond what a "vanilla" decision tree is attempting to do (note: a decision tree is usually not explicitly trying to model the probability of each sample belonging to class X or Y).

teruakohatu 4y ago

> Take it one step further. A test _never_ has a binary outcome, this is _always_ the result of some score thresholded against something else (could be another score). By test do you mean model/method? Because not all models have a score eg. discrete classifiers. CART classifiers are simply large if/else statements. There is no tunable threshold and while some people sort of draw something that looks like a ROC, it's not quite as useful. But of course you are right, in the real world you would be comparing many different models built with different methods and hyper params using a variety of metrics.

igorkraw 4y ago
Unless you are using some very exotic method i don't know, you are wrong about discrete classifier not having a score. Everything I know extracts an integer from a vector of scores over options, with the most common being argmax over likelihood trained with CrossEntropy
- fjkdlsjflkds 4y ago
  Nope. Decision trees typically output binary (or multinomial) values directly. Sure, you could tune the node thresholds (or optimize them during tree construction), to vary the trade-off between specificity and selectivity, but this may not be straightforward for deep trees or those with many leaf nodes, and it will never be as simple as simply tuning a single threshold (unless you are using decision stumps, rather than decision trees).
  - igorkraw 4y ago
    How do you construct your decision tree though? All methods I'm familiar with use a criterion which comes back down to using a score, which also ties down to the set theoretical foundations of probability theory: assign a score and then normalise to to obtain a probability.
  - radarsat1 4y ago
    > Decision trees typically output binary (or multinomial) values directly. Is there some Bayesian version of decision trees where you model each decision in the tree as a beta distribution or something? Then you could simply choose the leaf or path with the highest confidence. I just tried a cursory search about this but came up with a lot of theory justifying the use of thresholds as an approximation to a Bayesian interpretation, but I couldn't find any actual implementations that propagate a confidence measure or probability distribution through the tree, so I don't know if this is standard methodology. I can imagine having confidence in the final decision to also be useful for model ensembling. Curious what the right way to do this might be, or to know if there are no advantages vs. simple threshold trees.
    - fjkdlsjflkds 4y ago
      > Is there some Bayesian version of decision trees where you model each decision in the tree as a beta distribution or something? Then you could simply choose the leaf or path with the highest confidence. Sure... there's many things you could do "force" a decision tree to output a real scalar/vector (that you can then threshold): for example, you could use a regression tree where leafs are logistic/multinomial GLM. My point is that most types of decision trees (including CART decision trees) do not work like that by default (and they are considered pretty standard classifiers in ML literature), unlike what was claimed by igorkraw. > I can imagine having confidence in the final decision to also be useful for model ensembling. Curious what the right way to do this might be, or to know if there are no advantages vs. simple threshold trees. Sure, you could use ensembles of decision trees (e.g. random forests) to estimate a continuous value that perhaps (but not surely) correlates with "confidence" or "probability", which you can then threshold to obtain a binary value. But, again, this is beyond what a "vanilla" decision tree is attempting to do (note: a decision tree is usually not explicitly trying to model the probability of each sample belonging to class X or Y).