4 ms·

I like decision trees and this helps support my case for using them. I often go even further and don't use an algorithm to build the trees but build trees mysel

I like decision trees and this helps support my case for using them. I often go even further and don't use an algorithm to build the trees but build trees myself along intuitive causal lines and use the data to train its parameters. I sometimes build a few models manually and see which fits better with the data. Prior knowledge can prevent the pitfalls of automatically built models. Trees may be better than NNs because they overfit less but you can overfit even less with a bespoke model. For example, I've seen an automatically generated tree made to tune the efficiency of a factory end up using as a main feature "be after a specific date" because a machine was upgraded on that date and so the learning algorithm latched on to that unactionable piece of data as a main predictor for the model. This was an easy fix, to not feed timestamp data to the model but there are lots of more subtle cases like this and I've seen people spend much time cleaning and "tweaking" the input data to get the answers they want out of their ML models. If you have to try to make your ML model behave by manually selecting what data to feed it, you might as well go all the way and build a clean causal model yourself that reflect your priors and domain knowledge about the subject. I have an ML background but I often get more performance out of my models by doing something along the lines of what a Bayesian statistician would do. Of course with highly dimensional data like pixels in images you almost have no choice to use NNs. There's no way to hand-build these models.

hackerlight 4y ago
I want to know more about your process. How do you optimise the split point. How do you choose which feature to split on first
- BenoitEssiambre 4y ago
  I start with common sense priors or "intuition" but also sometimes write code to try different combinations and assess accuracy. For example, not too long ago I was trying to estimate and predict ATM usage and considered using days of the week as a feature. The simplest case is a single node tree that assumes every day of the week has similar usage. You just calculate mean and variance. But then, a lot of ATMs have different usage patterns on weekends than weekdays. So I considered a split into two groups, calculating different mean/variance for weekend days and week days. I could have tried different groupings of days but those two were easy to understand for users, covered most cases I was interested in with less risk of overfitting given the amount of data that I was working with (with more data I might have considered more splits even take into account special days like holidays). The more splits the more data you need to train the model, especially if you want not just a point estimate but an idea of variance (more parameters to estimate require more data). The decision whether to split or not can be done on a per ATM basis cross validating the splitting strategy by assessing a fleet of ATMs. People optimize different things in these kinds of problem. Here https://www.saedsayad.com/decision_tree_reg.htm https://www.saedsayad.com/decision_tree_reg.htm for example they try to reduce prediction variance. Oftentimes decision trees try to optimize information gain or gain ratio (https://en.wikipedia.org/wiki/Information_gain_(decision_tree) https://en.wikipedia.org/wiki/Information_gain_(decision_tre...) Gain ratio seems like a pretty ad-hoc technique to me so to prevent over-fitting and select the correct model (to split or not), I sometimes use a relative entropy penalty on the trained parameters kinda like described here: http://www.inference.org.uk/mackay/itprnn/ps/343.355.pdf http://www.inference.org.uk/mackay/itprnn/ps/343.355.pdf. I've never used it but apparently if you have problems where you can't calculate that parameter relative entropy, there are approximation techniques such as https://en.wikipedia.org/wiki/Bayesian_information_criterion https://en.wikipedia.org/wiki/Bayesian_information_criterion. Then there's things I sometimes do with hyperparameters to improve performance like mixing into individual ATM data fake data points that represent the prior for a wider group or class of ATMs. Again using common sense to adjust those priors (and always cross validating my tweaks against real data).
  - BenoitEssiambre 4y ago
    To add to this. One of the best thing about this approach is how little computational power it requires compared to NNs. I was able to finish this project in little time without having to get extra hardware approved. One core of our DB is now constantly used to feed real time data to constantly retrain the models but that, plus a partial core doing the analysis is all it takes to analyze billions in transactions per year in near real time and keep our predictions up to date and live on our portal. I'm also not too worried that I could get better performance using NNs as Bayesian methods are basically proven to be near optimal under some fairly reasonable assumptions and the more automatic methods overfitting or latching on artifacts of imperfect inputs actually have worse performance as the article points out. Until you get to really high dimensional problems, this is a better approach imo.
    - hackerlight 4y ago
      With these partially hand built trees, are you using only 1 or 2 at inference time? Or is it still a full on large forest of 200 trees? To elaborate: Are you coding up this bayesian logic as part of the training process, and then you run this code 200 times to end up with 200 trees? That code mixes the hand built rules with the normal approach to tree construction (random split location, random feature choice out of N features, etc)
      - BenoitEssiambre 4y ago
        In this case I ended up with two very simple trees which I train for each ATM (with a prior trained on the larger fleet). You could try a mix though. I didn't have time to try too many things. The following step which I didn't mention, was to use the trees for generating predictions into the future. Little monte carlo simulations using one of the tree (selected using that relative entropy methods, you could even use a weighted mix of both trees) that I run a few thousand times for each ATM and to generate a forecasted range of when the ATM is going to run out of cash.

hackerlight 4y ago

I want to know more about your process. How do you optimise the split point. How do you choose which feature to split on first

BenoitEssiambre 4y ago
I start with common sense priors or "intuition" but also sometimes write code to try different combinations and assess accuracy. For example, not too long ago I was trying to estimate and predict ATM usage and considered using days of the week as a feature. The simplest case is a single node tree that assumes every day of the week has similar usage. You just calculate mean and variance. But then, a lot of ATMs have different usage patterns on weekends than weekdays. So I considered a split into two groups, calculating different mean/variance for weekend days and week days. I could have tried different groupings of days but those two were easy to understand for users, covered most cases I was interested in with less risk of overfitting given the amount of data that I was working with (with more data I might have considered more splits even take into account special days like holidays). The more splits the more data you need to train the model, especially if you want not just a point estimate but an idea of variance (more parameters to estimate require more data). The decision whether to split or not can be done on a per ATM basis cross validating the splitting strategy by assessing a fleet of ATMs. People optimize different things in these kinds of problem. Here https://www.saedsayad.com/decision_tree_reg.htm https://www.saedsayad.com/decision_tree_reg.htm for example they try to reduce prediction variance. Oftentimes decision trees try to optimize information gain or gain ratio (https://en.wikipedia.org/wiki/Information_gain_(decision_tree) https://en.wikipedia.org/wiki/Information_gain_(decision_tre...) Gain ratio seems like a pretty ad-hoc technique to me so to prevent over-fitting and select the correct model (to split or not), I sometimes use a relative entropy penalty on the trained parameters kinda like described here: http://www.inference.org.uk/mackay/itprnn/ps/343.355.pdf http://www.inference.org.uk/mackay/itprnn/ps/343.355.pdf. I've never used it but apparently if you have problems where you can't calculate that parameter relative entropy, there are approximation techniques such as https://en.wikipedia.org/wiki/Bayesian_information_criterion https://en.wikipedia.org/wiki/Bayesian_information_criterion. Then there's things I sometimes do with hyperparameters to improve performance like mixing into individual ATM data fake data points that represent the prior for a wider group or class of ATMs. Again using common sense to adjust those priors (and always cross validating my tweaks against real data).
- BenoitEssiambre 4y ago
  To add to this. One of the best thing about this approach is how little computational power it requires compared to NNs. I was able to finish this project in little time without having to get extra hardware approved. One core of our DB is now constantly used to feed real time data to constantly retrain the models but that, plus a partial core doing the analysis is all it takes to analyze billions in transactions per year in near real time and keep our predictions up to date and live on our portal. I'm also not too worried that I could get better performance using NNs as Bayesian methods are basically proven to be near optimal under some fairly reasonable assumptions and the more automatic methods overfitting or latching on artifacts of imperfect inputs actually have worse performance as the article points out. Until you get to really high dimensional problems, this is a better approach imo.
  - hackerlight 4y ago
    With these partially hand built trees, are you using only 1 or 2 at inference time? Or is it still a full on large forest of 200 trees? To elaborate: Are you coding up this bayesian logic as part of the training process, and then you run this code 200 times to end up with 200 trees? That code mixes the hand built rules with the normal approach to tree construction (random split location, random feature choice out of N features, etc)
    - BenoitEssiambre 4y ago
      In this case I ended up with two very simple trees which I train for each ATM (with a prior trained on the larger fleet). You could try a mix though. I didn't have time to try too many things. The following step which I didn't mention, was to use the trees for generating predictions into the future. Little monte carlo simulations using one of the tree (selected using that relative entropy methods, you could even use a weighted mix of both trees) that I run a few thousand times for each ATM and to generate a forecasted range of when the ATM is going to run out of cash.