4 ms·

As someone who knows a little information theory and enough differential geometry to be dangerous, I have to admit that I've tried numerous to learn about Infor

As someone who knows a little information theory and enough differential geometry to be dangerous, I have to admit that I've tried numerous to learn about Information Geometry but haven't been able to penetrate the surface. The introductory articles like this tend to get bogged down in the definitions, and research-level articles are pretty impenetrable. If anyone here has a little more insight into this field, I'd be really interested to know more about the impact IG has had on other areas (be it applications to the real world, or to helping organize our understanding of another field of math).

scottlocklin 6y ago
The most intriguing use cases I've seen is for when it gets used to create better estimators (using the geometry of the Fisher information matrix). I guess it works particularly well on things like the EM algorithm, but there are probably lots of signal processing-ey applications. People babbling about neural nets; that's probably wrong -never seen an actual application there, though TDA does very interesting things for neural approaches for reasons that should be obvious. I kind of wish I had done the GR course with Carlo Rovelli and Ezra Newman back when I had the chance; the first semester was all differential geometry, and those dudes were great at explaining math. Since the Einstein equations are more nasty than the Fisher information matrix, it would have made all the info-geo stuff pretty accessible.
credit_guy 6y ago
Let me take a stab at this. A while ago I read about a guy's innovative idea to classify languages using zip (the archiving software). This idea is cute in itself, if you are curious you can check [1]. My rough summary would be like this: when you zip a French text, the zip facility first spends some time to train itself, and finds the letter frequencies and then compresses the text using these frequencies. If you train zip to a French text, but then use the frequencies to compress a Spanish text, you'll do worse than if you use the frequencies of the Spanish language itself. However, if you use the French frequencies to compress Hungarian text, you will do much, much worse. Then you ask the question: if for a certain language A, a zipped text is 1 MB using the language's own frequencies but 1.3MB using the frequencies form a different language B, then you say the distance between A and B is 0.3. If it takes 1.05 MB, then the distance is 0.05. If two languages are very similar like Nrwegian and Swedish, this distance is very small, if they are very different, the distance could be quite large. This concept of distance is essentially the KL-divergence [2]. It's not symmetric, so the distance between A and B is different compared to the distance between B and A, so it's not exactly what mathematicians call a "distance". But it's very useful nonetheless. When you train a classifier for example, many times you measure the quality of the classifier using the KL-divergence, but in that context people use the term cross-entropy. You can for example do a cluster analysis and find groups of languages that are related, like the Romance languages, or Slavic languages, and this without knowing anything about these languages at all. Now, how many languages are out there? Let's say 5000. Imagine you want to create one of those cool graphs where each language is linked with its closes neighbors and the length of that edge is equal to this "distance". You won't be able to do this exactly, because you are constrained by the geometry of the flat plane, but if you allow yourself to go in higher dimensions, you can do that ( * ). This graph will then have some shape, it's going to be somewhat curved. For example, if you restrict yourself to the Romance languages, and you put Latin in the center and put the current languages (Spanish, Portuguese, Catalan, French, Italian, Romanian, there are probably a few more) around it, you get something that looks like a circle. Is the circumference of this "circle" higher, or lower than 2piradius? If it's lower, then you say the space has positive curvature, if it's higher, negative. So, information geometry is 1. taking care of the various details I brushed off (like the fact that you actually use the square root of the KL-divergence to define the distance) and 2. studying the geometric properties of the metric space you produce. ( * ) This fact is known as the Nash embedding theorem, the same Nash who got a Nobel for the Nash equilibrium. [1] https://link.springer.com/chapter/10.1007/978-3-540-31865-1_22 https://link.springer.com/chapter/10.1007/978-3-540-31865-1_... [2]https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_diver...

scottlocklin 6y ago

The most intriguing use cases I've seen is for when it gets used to create better estimators (using the geometry of the Fisher information matrix). I guess it works particularly well on things like the EM algorithm, but there are probably lots of signal processing-ey applications. People babbling about neural nets; that's probably wrong -never seen an actual application there, though TDA does very interesting things for neural approaches for reasons that should be obvious. I kind of wish I had done the GR course with Carlo Rovelli and Ezra Newman back when I had the chance; the first semester was all differential geometry, and those dudes were great at explaining math. Since the Einstein equations are more nasty than the Fisher information matrix, it would have made all the info-geo stuff pretty accessible.

credit_guy 6y ago

Let me take a stab at this. A while ago I read about a guy's innovative idea to classify languages using zip (the archiving software). This idea is cute in itself, if you are curious you can check [1]. My rough summary would be like this: when you zip a French text, the zip facility first spends some time to train itself, and finds the letter frequencies and then compresses the text using these frequencies. If you train zip to a French text, but then use the frequencies to compress a Spanish text, you'll do worse than if you use the frequencies of the Spanish language itself. However, if you use the French frequencies to compress Hungarian text, you will do much, much worse. Then you ask the question: if for a certain language A, a zipped text is 1 MB using the language's own frequencies but 1.3MB using the frequencies form a different language B, then you say the distance between A and B is 0.3. If it takes 1.05 MB, then the distance is 0.05. If two languages are very similar like Nrwegian and Swedish, this distance is very small, if they are very different, the distance could be quite large. This concept of distance is essentially the KL-divergence [2]. It's not symmetric, so the distance between A and B is different compared to the distance between B and A, so it's not exactly what mathematicians call a "distance". But it's very useful nonetheless. When you train a classifier for example, many times you measure the quality of the classifier using the KL-divergence, but in that context people use the term cross-entropy. You can for example do a cluster analysis and find groups of languages that are related, like the Romance languages, or Slavic languages, and this without knowing anything about these languages at all. Now, how many languages are out there? Let's say 5000. Imagine you want to create one of those cool graphs where each language is linked with its closes neighbors and the length of that edge is equal to this "distance". You won't be able to do this exactly, because you are constrained by the geometry of the flat plane, but if you allow yourself to go in higher dimensions, you can do that ( * ). This graph will then have some shape, it's going to be somewhat curved. For example, if you restrict yourself to the Romance languages, and you put Latin in the center and put the current languages (Spanish, Portuguese, Catalan, French, Italian, Romanian, there are probably a few more) around it, you get something that looks like a circle. Is the circumference of this "circle" higher, or lower than 2piradius? If it's lower, then you say the space has positive curvature, if it's higher, negative. So, information geometry is 1. taking care of the various details I brushed off (like the fact that you actually use the square root of the KL-divergence to define the distance) and 2. studying the geometric properties of the metric space you produce. ( * ) This fact is known as the Nash embedding theorem, the same Nash who got a Nobel for the Nash equilibrium. [1] https://link.springer.com/chapter/10.1007/978-3-540-31865-1_22 https://link.springer.com/chapter/10.1007/978-3-540-31865-1_... [2]https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_diver...