16 ms·
There's a tendency for anyone who learns information theory before statistics to think that information theory is tremendously relevant - but no, rephrasing max
by gone35 9y ago
There's a tendency for anyone who learns information theory before statistics to think that information theory is tremendously relevant - but no, rephrasing maximum likelihood or Bayesian methods in information theory terms may sometimes be slightly helpful in thinking about them, but doesn't really add anything fundamental.
And no, there's nothing particularly special or interesting about distributions that maximize entropy subject to some (generally arbitrarily selected) constraint.
Wow, some two heavyweight opinions there. Care to elaborate?
- radford-neal 9y agoSome ideas like the Minimum Description Length principle derived from information theory turn out to be just rephrasings of already-known statistical ideas. This can occasionally provide more insight, but also can lead to ridiculously irrelevant things like looking in detail at how one might produce a code for data, rather than just saying the code would have length equal to log2 of the probability of the data, which of course leads to just forgetting about the code altogether and looking at probabilities instead. The maximum entropy idea is just wrong (in general), in that there is no good argument for doing it. Actually, it's "not even wrong", since maximizing the entropy subject to the observed values of some expectations is just not possible, since we do not observe expectations, but rather particular finite data sets.