3 ms·
Probably one of the dozen or so most important publications of the twentieth century. Ironically though it would be Norbert Wiener's interpretation of informati
by noema 6y ago
Probably one of the dozen or so most important publications of the twentieth century. Ironically though it would be Norbert Wiener's interpretation of information (exactly Shannon's times minus one) that would cement itself in the popular lexicon due to it being far more intuitive. Where Wiener posits information as negative-entropy (or "order"), Shannon interprets it as representing a degree of freedom, or uncertainty. The problem with Wiener's is that the underlying epistemology carries a subjective bent, where information is equivocal to "meaning" or semantic content. Shannon's, meanwhile is ultimately superior as an objective metric, though far less intuitive (under his interpretation, a random string is technically the most information-saturated construct possible, because it possesses the highest degrees of freedom).
- ssivark 6y agoBut isn't the information gained from a message necessarily subjective? It depends on just on this message, but from the distribution from which this message was drawn, and more appropriately, the distribution anticipated by the observer will determine the value (information content) of the message. That is exactly how it becomes useful in practice. This "surprisal" can be parametrized by relative entropy (Kullback-Liebler divergence), which reduces to the Shannon entropy formula only when the observer has the correct anticipated distribution.
- noema 6y agoI think there is a distinction to be made between information in the ontological sense and information in the pragmatic sense. The former does not have any purchase on "meaning" or "use", only a metric of informational entropy or degrees of freedom, irrespective of observation. However, the sense in which you speak of information is the pragmatic dimension, where an observer receives a datum of semantic content against a given backdrop of pre-established "meaning". This sense is problematic as an objective metric.
- GoatOfAplomb 6y agoI'm trying to find a quote I read about this paper once. To paraphrase, not only did this paper create a new field of academic inquiry, it answered most of that new field's interesting questions, too.
- kkaranth 6y agoFrom Jon Gertner's "Idea Factory": > With Shannon’s startling ideas on information, it was one of the rare moments in history, an academic would later point out, “where somebody founded a field, stated all the major results, and proved most of them all pretty much at once.” It is attributed to a documentary on Claude Shannon: https://www.youtube.com/watch?v=z2Whj_nL-x8 https://www.youtube.com/watch?v=z2Whj_nL-x8
- eric1293 6y agoTo say Wiener and Shannon founded information theory is like saying Lorenz and Einstein founded speial relativity. I don't think physicists would consider Lorenz having made a major advance, and in fact even after the relativity was explained some people still didn't understand it. Wiener made important contributions to mathematics but not to information theory. He wrote a book saying that entropy is related to "information" and is maximized for a Gaussian. That's about his involvement. The paper attached goes way beyond that.
- noema 6y agoI didn't claim Wiener founded it, only that his popularization of the term is more ubiquitous than Shannon's formulation.