3 ms·
"From the earliest days of information theory it has been appreciated that information per se is not a good measure of message value. For example, a typical seq
by dri_ft 8y ago
"From the earliest days of information theory it has been appreciated that information per se is not a good measure of message value. For example, a typical sequence of coin tosses has high information content but little value; an ephemeris, giving the positions of the moon and planets every day for a hundred years, has no more information than the equations of motion and initial conditions from which it was calculated, but saves its owner the effort of recalculating these positions. The value of a message thus appears to reside not in its information (its absolutely unpredictable parts), nor in its obvious redundancy (verbatim repetitions, unequal digit frequencies), but rather in what might be called its buried redundancy--parts predictable only with difficulty, things the receiver could in principle have figured out without being told, but only at considerable cost in money, time, or computation. In other words, the value of a message is the amount of mathematical or other work plausibly done by its originator, which its receiver is saved from having to repeat."
—Bennett, Charles H. "Logical depth and physical complexity." The Universal Turing Machine: A Half-Century Survey.
- emanueldima 8y agoIs there a mathematical theory trying to quantify this value?
- deleted 8y ago[deleted]
- antidesitter 8y agoBennett’s logical depth.
- sgentle 8y agoYou might find relative entropy, aka information gain, aka Kullback-Liebler divergence interesting: https://en.m.wikipedia.org/wiki/Kullback–Leibler_divergence https://en.m.wikipedia.org/wiki/Kullback–Leibler_divergence Also this treatment of K-L divergence as a measure of "Bayesian surprise": http://ilab.usc.edu/surprise/ http://ilab.usc.edu/surprise/ That's all based on Shannon entropy (probabilistic), not Kolmogorov complexity (algorithmic), but there are a lot of connections between them. This paper is a pretty thorough summary: https://homepages.cwi.nl/~paulv/papers/info.pdf https://homepages.cwi.nl/~paulv/papers/info.pdf And here's a paper defining algorithmic relative complexity by analogy to relative entropy: http://www.mdpi.com/1099-4300/13/4/902/pdf-vor http://www.mdpi.com/1099-4300/13/4/902/pdf-vor "We define the cross-complexity of an object x with respect to another object y as the amount of computational resources needed to specify x in terms of y, and the complexity of x related to y as the compression power which is lost when adopting such a description for x, compared to the shortest representation of x"
- ppod 8y agoMight not be exactly what you're asking, but Landauer's principles that information is physical and necessarily requires some energy to compute is relevant. The 'cost' Bennett is talking about here might be the energy required to change the entropy.
- yters 8y agoThe sort of information we care about is mutual information, which cannot be generated by randomness or algorithms, per the law of information nongrowth. Which raises the question, where does this mutual information come from?
- badrabbit 8y agoIsn't the value of information given to it by the observer? Coin tosses can have a lot of value for some observers(odd but useful rng source for example).