Some ideas like the Minimum Description Length principle derived from information theory turn out to be just rephrasings of already-known statistical ideas. Th

Some ideas like the Minimum Description Length principle derived from information theory turn out to be just rephrasings of already-known statistical ideas. This can occasionally provide more insight, but also can lead to ridiculously irrelevant things like looking in detail at how one might produce a code for data, rather than just saying the code would have length equal to log2 of the probability of the data, which of course leads to just forgetting about the code altogether and looking at probabilities instead. The maximum entropy idea is just wrong (in general), in that there is no good argument for doing it. Actually, it's "not even wrong", since maximizing the entropy subject to the observed values of some expectations is just not possible, since we do not observe expectations, but rather particular finite data sets.