3 ms·

> Is there some Bayesian version of decision trees where you model each decision in the tree as a beta distribution or something? Then you could simply choose t

> Is there some Bayesian version of decision trees where you model each decision in the tree as a beta distribution or something? Then you could simply choose the leaf or path with the highest confidence. Sure... there's many things you could do "force" a decision tree to output a real scalar/vector (that you can then threshold): for example, you could use a regression tree where leafs are logistic/multinomial GLM. My point is that most types of decision trees (including CART decision trees) do not work like that by default (and they are considered pretty standard classifiers in ML literature), unlike what was claimed by igorkraw. > I can imagine having confidence in the final decision to also be useful for model ensembling. Curious what the right way to do this might be, or to know if there are no advantages vs. simple threshold trees. Sure, you could use ensembles of decision trees (e.g. random forests) to estimate a continuous value that perhaps (but not surely) correlates with "confidence" or "probability", which you can then threshold to obtain a binary value. But, again, this is beyond what a "vanilla" decision tree is attempting to do (note: a decision tree is usually not explicitly trying to model the probability of each sample belonging to class X or Y).

radarsat1 4y ago
> (note: a decision tree is usually not explicitly trying to model the probability of each sample belonging to class X or Y). I realize that. Maybe my question wasn't clear. I was specifically trying to ask if there exist interesting or useful models beyond standard decision trees that might do so.
- fjkdlsjflkds 4y ago
  I think I got your question right (but perhaps I was not totally clear in my answer). In a nutshell: sure, there are lots of such possibly useful models that use trees (you can even come up with new ones yourself). Random forests or GLM regression trees are two examples. My point is that, because standard decision trees do not attempt to explicitly model probabilities (they just try to partition the data in "homogeneous groups", in some sense), you have no pre-baked assurances that your "score" will actually be well calibrated (i.e., no assurances that it will actually be a good predictor of the raw probability/confidence). Sure, you could come up with some "decision tree"-like (i.e., partitioning algorithm) Bayesian method that would (maybe) ensure reasonable calibration of "confidence"... but, at that point, are you even using "decision trees" anymore? Probably not.

radarsat1 4y ago

> (note: a decision tree is usually not explicitly trying to model the probability of each sample belonging to class X or Y). I realize that. Maybe my question wasn't clear. I was specifically trying to ask if there exist interesting or useful models beyond standard decision trees that might do so.

fjkdlsjflkds 4y ago
I think I got your question right (but perhaps I was not totally clear in my answer). In a nutshell: sure, there are lots of such possibly useful models that use trees (you can even come up with new ones yourself). Random forests or GLM regression trees are two examples. My point is that, because standard decision trees do not attempt to explicitly model probabilities (they just try to partition the data in "homogeneous groups", in some sense), you have no pre-baked assurances that your "score" will actually be well calibrated (i.e., no assurances that it will actually be a good predictor of the raw probability/confidence). Sure, you could come up with some "decision tree"-like (i.e., partitioning algorithm) Bayesian method that would (maybe) ensure reasonable calibration of "confidence"... but, at that point, are you even using "decision trees" anymore? Probably not.