I start with common sense priors or "intuition" but also sometimes write code to try different combinations and assess accuracy. For example, not too long ago

I start with common sense priors or "intuition" but also sometimes write code to try different combinations and assess accuracy. For example, not too long ago I was trying to estimate and predict ATM usage and considered using days of the week as a feature. The simplest case is a single node tree that assumes every day of the week has similar usage. You just calculate mean and variance. But then, a lot of ATMs have different usage patterns on weekends than weekdays. So I considered a split into two groups, calculating different mean/variance for weekend days and week days. I could have tried different groupings of days but those two were easy to understand for users, covered most cases I was interested in with less risk of overfitting given the amount of data that I was working with (with more data I might have considered more splits even take into account special days like holidays). The more splits the more data you need to train the model, especially if you want not just a point estimate but an idea of variance (more parameters to estimate require more data). The decision whether to split or not can be done on a per ATM basis cross validating the splitting strategy by assessing a fleet of ATMs. People optimize different things in these kinds of problem. Here https://www.saedsayad.com/decision_tree_reg.htm https://www.saedsayad.com/decision_tree_reg.htm for example they try to reduce prediction variance. Oftentimes decision trees try to optimize information gain or gain ratio (https://en.wikipedia.org/wiki/Information_gain_(decision_tree) https://en.wikipedia.org/wiki/Information_gain_(decision_tre...) Gain ratio seems like a pretty ad-hoc technique to me so to prevent over-fitting and select the correct model (to split or not), I sometimes use a relative entropy penalty on the trained parameters kinda like described here: http://www.inference.org.uk/mackay/itprnn/ps/343.355.pdf http://www.inference.org.uk/mackay/itprnn/ps/343.355.pdf. I've never used it but apparently if you have problems where you can't calculate that parameter relative entropy, there are approximation techniques such as https://en.wikipedia.org/wiki/Bayesian_information_criterion https://en.wikipedia.org/wiki/Bayesian_information_criterion. Then there's things I sometimes do with hyperparameters to improve performance like mixing into individual ATM data fake data points that represent the prior for a wider group or class of ATMs. Again using common sense to adjust those priors (and always cross validating my tweaks against real data).