6 ms·

Does anyone really find it surprising that you can do contortions around KL divergence among candidate probabilistic models to the point of shoe-horning some an

by mlthoughts2018 8y ago

Does anyone really find it surprising that you can do contortions around KL divergence among candidate probabilistic models to the point of shoe-horning some analogue of thermodynamic laws and equilibria into machine learning? I don’t find this either surprising, illuminating, or interesting. It reminds me of the “information geometry of boosting” section of [0]. [0]: https://pdfs.semanticscholar.org/2fad/679058e465fc07f942cfedd215cedbe09c6d.pdf https://pdfs.semanticscholar.org/2fad/679058e465fc07f942cfed...

gog-ma-gog 8y ago
...yes? Why is it not surprising?
danielmorozoff 8y ago
Fundamentally, thermodynamics and information theory share a slew of similar ideas. Because in a lot of ways information theory came out of a thermodynamic thought process applied to signals. Even more recent work in ml like Boltzmann machines are directly taken from this overlap. That being said stronger theoretical connections are well worth the effort as these fields do have some different ideas that if shown to be similar can yield discoveries and push our understanding forward. Some thoughts that come to mind are compression, model generalizability/ convergence.
- kgwgk 8y ago
  Minor nitpick: Boltzmann machines are hardly recent. They were introduced in the eighties.
  - danielmorozoff 8y ago
    Definitely. I meant more recent than the foundations of information theory
    - kgwgk 8y ago
      My bad, I understood "more recent work in ml" as being recent in ml terms, not as work in ml which is more recent than the initial use (and abuse [1]) of thermodynamic concepts in information theory. [1] Shannon, according to Tribus and McIrvine: "My greatest concern was what to call it. I thought of calling it ‘information’, but the word was overly used, so I decided to call it ‘uncertainty’. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, ‘You should call it entropy, for two reasons: In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage."
WhitneyLand 8y ago
Does anyone? What % of people make any connection at all between the fields? Firstly this is not a CS or Physics only forum so I would guess the subject could easily have never come up for many (if so for the simpler/non-ML connections see https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_information_theory https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_...). Secondly I think general awareness beyond formal education has increased in a non-linear way only over the last 20 years only because there has been so much science press around the celebrity of Steven Hawking. His celebrity alone raises awareness, however it was compounded by drama depicted between himself and others and some really interesting observations along the way (https://en.wikipedia.org/wiki/Holographic_principle https://en.wikipedia.org/wiki/Holographic_principle). When all of this publicity has various inevitable Kevin Bacon trails back to the fundamental connections, I’d say the population is not only not everyone but could be much less if things unfolded a little differently.
- mlthoughts2018 8y ago
  I mean it more from a Joel Spolsky sort of too much abstraction point of view, e.g. from [0]: > “When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. Those are both sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more. And if you go too far up, abstraction-wise, you run out of oxygen. Sometimes smart thinkers just don’t know when to stop, and they create these absurd, all-encompassing, high-level pictures of the universe that are all good and fine, but don’t actually mean anything at all.” [0]: < https://www.joelonsoftware.com/2000/07/22/microsoft-goes-bonkers/ https://www.joelonsoftware.com/2000/07/22/microsoft-goes-bon... >
  - WhitneyLand 8y ago
    I can relate to your point more in that way. Joel has a gift for exlplaining intangibles. Agree with his point also having had to spend time debating whether an abstraction choice adds value. Still worth noting I think though that he does say “sometimes”, because no one is going anywhere without abstractions or generalizations. The problem is they’re hard to do really well and we know historically really smart people can make small mistakes and fall into the not cogent/no value/Joel trap #234.
orbifold 8y ago
Whether you find something like this surprising largely depends on your prior knowledge. There are quite different sets of people that work with statistics, theoretical physicists are one of them. So a recent advance in one field (machine learning), will inevitably draw in the hyenas from other fields, especially if one of their favorite toys (building increasingly complicated high energy particle physics models) has been largely taken away from them given the recent zero results at the LHC. Which is how you get review articles like this one (https://arxiv.org/abs/1803.08823 https://arxiv.org/abs/1803.08823).
- kgwgk 8y ago
  Thanks for the link, it seems a nice review.
c3534l 8y ago
I doubt even 1% of the population can decode that sentence, let alone find it so blindingly obvious it need not be said.
- mlthoughts2018 8y ago
  It seems clear that this paper (and many papers like it that try to produce higher level abstractions casting one field’s concepts into theorems of some other field) is only meant for that 1% you mention. In fact this is why I don’t get excited about these sorts of results. For people outside the main fields, it ends up sounding like some dramatic philosophical discovery, but for people in the field it’s borderline meaningless. If you play around with definitions long enough, you can always find some notion of X that looks like “equilibrium” or “convergence” or “entropy” or “orthogonality” or some decomposition or some transformation for some other concept Y. Finding these results is almost always just about rejiggering definitions so that something mundane can be reinterpreted as something significant “in the right space.”

gog-ma-gog 8y ago

...yes? Why is it not surprising?

danielmorozoff 8y ago

Fundamentally, thermodynamics and information theory share a slew of similar ideas. Because in a lot of ways information theory came out of a thermodynamic thought process applied to signals. Even more recent work in ml like Boltzmann machines are directly taken from this overlap. That being said stronger theoretical connections are well worth the effort as these fields do have some different ideas that if shown to be similar can yield discoveries and push our understanding forward. Some thoughts that come to mind are compression, model generalizability/ convergence.

kgwgk 8y ago
Minor nitpick: Boltzmann machines are hardly recent. They were introduced in the eighties.
- danielmorozoff 8y ago
  Definitely. I meant more recent than the foundations of information theory
  - kgwgk 8y ago
    My bad, I understood "more recent work in ml" as being recent in ml terms, not as work in ml which is more recent than the initial use (and abuse [1]) of thermodynamic concepts in information theory. [1] Shannon, according to Tribus and McIrvine: "My greatest concern was what to call it. I thought of calling it ‘information’, but the word was overly used, so I decided to call it ‘uncertainty’. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, ‘You should call it entropy, for two reasons: In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage."

WhitneyLand 8y ago

Does anyone? What % of people make any connection at all between the fields? Firstly this is not a CS or Physics only forum so I would guess the subject could easily have never come up for many (if so for the simpler/non-ML connections see https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_information_theory https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_...). Secondly I think general awareness beyond formal education has increased in a non-linear way only over the last 20 years only because there has been so much science press around the celebrity of Steven Hawking. His celebrity alone raises awareness, however it was compounded by drama depicted between himself and others and some really interesting observations along the way (https://en.wikipedia.org/wiki/Holographic_principle https://en.wikipedia.org/wiki/Holographic_principle). When all of this publicity has various inevitable Kevin Bacon trails back to the fundamental connections, I’d say the population is not only not everyone but could be much less if things unfolded a little differently.

mlthoughts2018 8y ago
I mean it more from a Joel Spolsky sort of too much abstraction point of view, e.g. from [0]: > “When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. Those are both sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more. And if you go too far up, abstraction-wise, you run out of oxygen. Sometimes smart thinkers just don’t know when to stop, and they create these absurd, all-encompassing, high-level pictures of the universe that are all good and fine, but don’t actually mean anything at all.” [0]: < https://www.joelonsoftware.com/2000/07/22/microsoft-goes-bonkers/ https://www.joelonsoftware.com/2000/07/22/microsoft-goes-bon... >
- WhitneyLand 8y ago
  I can relate to your point more in that way. Joel has a gift for exlplaining intangibles. Agree with his point also having had to spend time debating whether an abstraction choice adds value. Still worth noting I think though that he does say “sometimes”, because no one is going anywhere without abstractions or generalizations. The problem is they’re hard to do really well and we know historically really smart people can make small mistakes and fall into the not cogent/no value/Joel trap #234.

orbifold 8y ago

Whether you find something like this surprising largely depends on your prior knowledge. There are quite different sets of people that work with statistics, theoretical physicists are one of them. So a recent advance in one field (machine learning), will inevitably draw in the hyenas from other fields, especially if one of their favorite toys (building increasingly complicated high energy particle physics models) has been largely taken away from them given the recent zero results at the LHC. Which is how you get review articles like this one (https://arxiv.org/abs/1803.08823 https://arxiv.org/abs/1803.08823).

kgwgk 8y ago
Thanks for the link, it seems a nice review.

c3534l 8y ago

I doubt even 1% of the population can decode that sentence, let alone find it so blindingly obvious it need not be said.

mlthoughts2018 8y ago
It seems clear that this paper (and many papers like it that try to produce higher level abstractions casting one field’s concepts into theorems of some other field) is only meant for that 1% you mention. In fact this is why I don’t get excited about these sorts of results. For people outside the main fields, it ends up sounding like some dramatic philosophical discovery, but for people in the field it’s borderline meaningless. If you play around with definitions long enough, you can always find some notion of X that looks like “equilibrium” or “convergence” or “entropy” or “orthogonality” or some decomposition or some transformation for some other concept Y. Finding these results is almost always just about rejiggering definitions so that something mundane can be reinterpreted as something significant “in the right space.”