4 ms·

One thing I'd add to this is that this kind of thinking makes your coordinate system really matter. Consider a measurement of some uncertainly sized cubes. You

One thing I'd add to this is that this kind of thinking makes your coordinate system really matter. Consider a measurement of some uncertainly sized cubes. You could describe them with their edge length or their volume. Learning one tells you the other. They're equivalent data. However a maximum entropy distribution on one isn't a maximum entropy distribution on the other. Pragmatically, there's always something you can do (e.g. a Jeffreys prior), but philosophically, this has always made me uneasy with justifications about max entropy that don't also have justifications of the choice of coordinate system.

canjobear 6y ago
This seems important. Is there somewhere where this example is worked out in more detail?
- lambdatronics 6y ago
  http://bayes.wustl.edu/etj/articles/prior.pdf http://bayes.wustl.edu/etj/articles/prior.pdf
  - yetanothermonk 6y ago
    Thank you!
blackbear_ 6y ago
Could this be solved by putting a max-entropy distribution on their joint probability?
- contravariant 6y ago
  Their joint distribution is degenerate, so it becomes a bit unclear how to even define entropy. Personally I'm of the view that the Kullback-Leibler divergence which is defined for arbitrary probability measures (with no special treatment for continuous ones) and which is independent of the choice of coordinates is the true measure of information. Its downside is that you can only compare 2 distributions that way. For the discrete case you can just pick the uniform distribution as your non-informative base. The issues with the entropy definition for continuous distributions boil down to the problem of picking a uniform distribution for the real numbers.
  - kgwgk 6y ago
    You're right, a proper extension of Shannon's entropy to the continuous case requires a reference measure. Jaynes discusses this in the Brandeis lectures I linked to in another comment: https://bayes.wustl.edu/etj/articles/brandeis.pdf https://bayes.wustl.edu/etj/articles/brandeis.pdf
    - contravariant 6y ago
      I see, of course in physics you have the advantage that the phase space already has a canonical volume form, which coincides with the normal uniform measure if you use canonical coordinates and, amazingly, is preserved by the equations of motion. In most statistical problems you don't have such a nice measure, and it is always good to keep in mind that choosing a uniform 'improper prior', even implicitly, will mean your choice of coordinates will influence your result. Edit: Hang on, I just noticed the names on the lecture notes you sent me. Uhlenbeck, Wheeler and Schwinger? That's one heck of a line-up. With the part you linked by Uhlenbeck it seems, I'm going to set aside some time to read that one more carefully.
      - kgwgk 6y ago
        The pdf I linked to contains only pages 181-218 from Vol III of the 1962 lectures (Jaynes). I’ve not read the others, I don’t know if they may be available online.

canjobear 6y ago

This seems important. Is there somewhere where this example is worked out in more detail?

lambdatronics 6y ago
http://bayes.wustl.edu/etj/articles/prior.pdf http://bayes.wustl.edu/etj/articles/prior.pdf
- yetanothermonk 6y ago
  Thank you!

blackbear_ 6y ago

Could this be solved by putting a max-entropy distribution on their joint probability?

contravariant 6y ago
Their joint distribution is degenerate, so it becomes a bit unclear how to even define entropy. Personally I'm of the view that the Kullback-Leibler divergence which is defined for arbitrary probability measures (with no special treatment for continuous ones) and which is independent of the choice of coordinates is the true measure of information. Its downside is that you can only compare 2 distributions that way. For the discrete case you can just pick the uniform distribution as your non-informative base. The issues with the entropy definition for continuous distributions boil down to the problem of picking a uniform distribution for the real numbers.
- kgwgk 6y ago
  You're right, a proper extension of Shannon's entropy to the continuous case requires a reference measure. Jaynes discusses this in the Brandeis lectures I linked to in another comment: https://bayes.wustl.edu/etj/articles/brandeis.pdf https://bayes.wustl.edu/etj/articles/brandeis.pdf
  - contravariant 6y ago
    I see, of course in physics you have the advantage that the phase space already has a canonical volume form, which coincides with the normal uniform measure if you use canonical coordinates and, amazingly, is preserved by the equations of motion. In most statistical problems you don't have such a nice measure, and it is always good to keep in mind that choosing a uniform 'improper prior', even implicitly, will mean your choice of coordinates will influence your result. Edit: Hang on, I just noticed the names on the lecture notes you sent me. Uhlenbeck, Wheeler and Schwinger? That's one heck of a line-up. With the part you linked by Uhlenbeck it seems, I'm going to set aside some time to read that one more carefully.
    - kgwgk 6y ago
      The pdf I linked to contains only pages 181-218 from Vol III of the 1962 lectures (Jaynes). I’ve not read the others, I don’t know if they may be available online.