4 ms·
Fundamentally, thermodynamics and information theory share a slew of similar ideas. Because in a lot of ways information theory came out of a thermodynamic thou
by danielmorozoff 8y ago
Fundamentally, thermodynamics and information theory share a slew of similar ideas. Because in a lot of ways information theory came out of a thermodynamic thought process applied to signals. Even more recent work in ml like Boltzmann machines are directly taken from this overlap. That being said stronger theoretical connections are well worth the effort as these fields do have some different ideas that if shown to be similar can yield discoveries and push our understanding forward. Some thoughts that come to mind are compression, model generalizability/ convergence.
- kgwgk 8y agoMinor nitpick: Boltzmann machines are hardly recent. They were introduced in the eighties.
- danielmorozoff 8y agoDefinitely. I meant more recent than the foundations of information theory
- kgwgk 8y agoMy bad, I understood "more recent work in ml" as being recent in ml terms, not as work in ml which is more recent than the initial use (and abuse [1]) of thermodynamic concepts in information theory. [1] Shannon, according to Tribus and McIrvine: "My greatest concern was what to call it. I thought of calling it ‘information’, but the word was overly used, so I decided to call it ‘uncertainty’. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, ‘You should call it entropy, for two reasons: In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage."