>> He'll happily throw in your zip code and credit card number into the mix, not realizing that those two features have super high entropy You will be surprise

>> He'll happily throw in your zip code and credit card number into the mix, not realizing that those two features have super high entropy You will be surprised: you can make significant gains by including the zip code - i've seen that happen in a competitive setting. Where you live probably contains some signal about your credit worthiness. Having said that, of course it doesn't make sense to simply feed the raw zip code to the tree. An appropriate encoding (most people would use a one-hot encoding, though there exist better ones) of the zip code will be key to extracting signal in a robust way. >> ... the tree will have serious overfitting issues Isn't it an almost standard practise now to use an ensemble of trees, such as a random forest? Decision trees have long been known to be prone to overfitting.