Also (5) it doesn't really work or make sense. The data isn't necessarily representative of the domain theory, in fact in a lot of domains the data isn't, beca

Also (5) it doesn't really work or make sense. The data isn't necessarily representative of the domain theory, in fact in a lot of domains the data isn't, because the domains are so large that you can't capture the whole of it in a tractable training set, for example : images. Other data doesn't capture the domain because the data is generated in a regime that isn't operant when gathered - for example bull runs vs. bear runs in the markets. Bayesian analysis is attractive, we can include informative priors that capture our knowledge that in circumstances outside of the data other determining behaviours exist. This is also one of the reasons why deep networks can outperform support vector machines; deep networks can learn to prefer domain theories that are not the minimal descriptive one. The other thing that is interesting about MDL is where does the idea that the minimal theory is the right theory come from? Most people say "oh it's Occam's razor" but where did Occam's razor come from - who was Mr (Fr.) Occam?? Well, he was a 13th century philosopher - part of the Cambridge school and part of the tradition of Scotus invented to construct a story that supported the Trinitarian God... and this is why we prefer the idea that "entities will not multiply beyond necessity" because it says that you have a Trinity because The University ABSOLUTELY cannot work without it, and that's why you have three and not two and not four. I am happy with all this but why should we think it's a good way to do machine learning? After all there are lots of examples of theories that were simple but don't work as well as complex alternatives - Gravity is a good one.