7 ms·

Jane Austen's concept of information (Not Claude Shannon's) (2013)

73 points by benbreen 6y ago

whatshisface 6y ago
What about this is specific to Austen? All of the quotes in the article are examples of information used as a normal English word, to convey its normal meaning, which everyone knows.
- jxy 6y ago
  It is a truth universally acknowledged
breck 6y ago
In the past few years I introduced a new concept that provides a simple and universal mathematical way to represent information AND meaning. The basic idea is to recursively build grammars as trees in 2-D Cartesian space. Then you just count the elements. https://github.com/treenotation/research/blob/master/papers/paper3/countingComplexity.pdf https://github.com/treenotation/research/blob/master/papers/... It's still in its infancy, but it is very pragmatic and very useful.
- pineconewarrior 6y ago
  As someone who is completely ignorant to this field, I found this very readable and interesting. Thanks for your work!
2-tpg 6y ago
Recent work in information theory has formalized the concept of semantic information, where Shannon deliberately avoided semantics, and focused purely on the information content. They define semantic information as: Information which helps an object stay away from thermodynamic equilibrium (no different from entropic noise). Information then has meaning for a receiver if it helps them stay alive. The very same message may be completely "meaningless" to another receiver. https://arxiv.org/abs/1806.08053 https://arxiv.org/abs/1806.08053 Semantic information, autonomous agency, and nonequilibrium statistical physics > Shannon information theory provides various measures of so-called "syntactic information", which reflect the amount of statistical correlation between systems. In contrast, the concept of "semantic information" refers to those correlations which carry significance or "meaning" for a given system. Semantic information plays an important role in many fields, including biology, cognitive science, and philosophy, and there has been a long-standing interest in formulating a broadly applicable and formal theory of semantic information. In this paper we introduce such a theory. We define semantic information as the syntactic information that a physical system has about its environment which is causally necessary for the system to maintain its own existence. "Causal necessity" is defined in terms of counter-factual interventions which scramble correlations between the system and its environment, while "maintaining existence" is defined in terms of the system's ability to keep itself in a low entropy state. We also use recent results in nonequilibrium statistical physics to analyze semantic information from a thermodynamic point of view. Our framework is grounded in the intrinsic dynamics of a system coupled to an environment, and is applicable to any physical system, living or otherwise. It leads to formal definitions of several concepts that have been intuitively understood to be related to semantic information, including "value of information", "semantic content", and "agency".
- yters 6y ago
  What about webpages on suicide techniques? Do those count as information?
  - mhh__ 6y ago
    I guess death could be viewed as the ultimate move away from the equilibrium (of being alive)?
    - yters 6y ago
      Doesn't death allow the body to achieve maximal entropy? Also, this conversation here has nothing to do with keeping me alive, yet I find it informative. Seems the concept of sematic information is missing something, or a lot.
      - 2-tpg 6y ago
        For me, semantic information is information which allows you to predict future world states, or remove uncertainty about current world states, while optimizing for your utility function. If your utility function is to remove all sense data, then a website with suicide techniques helps you predict future world states (if X then Death), and aligns with your utility function, so it is meaningful information for you. But then you get into heterophenomenology territory: someone may state they are not hungry, while desperate for sustenance (or vice versa). People may temporarily act counter to their typical utility function. Do we want to allow such craziness to influence the formal value of semantic information? > has nothing to do with keeping me alive, yet I find it informative. Imagine having to read the exact same information every day. It won't allow you to adapt to a changing environment or be exposed to new ideas, so you can form new ideas yourself. You'd be stuck in a rut, and as good as non-alive. Soon you'll probably perish out of boredom.
        yters 6y ago
        I spend long periods of time without ingesting sources of information without getting bored and literally dying. Conversely, if I was constantly bombarded with useful information, I would still go crazy, and that could very well lead to literal death. I think staying alive is one piece of meaningful information out of a very large set of meaningful information, most of which has nothing to do with staying alive. In other words, meaningfulness is orthogonal to staying alive.
mhh__ 6y ago
I'm still reading this, so I may be getting the wrong end of the stick, but this seems to be conflating Shannon stating that his work is the study of the behaviour of information in the abstract and the notion that information theory is totally disinterested in anything else. This piece just seems a bit dull to my eyes - maybe there's something interesting trying to get out, but currently it seems like it's fumbling around the edges of Shannon's work and trying to find a dichotomy where there fundamentally isn't one. The connection with Jane Austen in particular seems to be completely spurious - let's say that the information of finding this in Austen's work doesn't seem to be very high.
- inimino 6y ago
  Conpletely agreed. There was just this: > But [Shannon's] choice of the label "information" in his publications seems to have confused many highly intelligent people. But there was nothing further about who was confused or where or when. The article seems to be clearing up a confusion that doesn't exist.
yters 6y ago
It's a very important point the paper makes, which is the disconnect between what is commonly called Shannon information (he actually called it entropy) and what we call information in normal parlance. A channel may transmit a high amount of Shannon entropy, but contain zero information of the regular sort, since the transmission is just random noise. A concept that is closer to our intuition is mutual information, so named by Shannon's colleague precisely because Shannon entropy is not concerned with semantics.
deleted 6y ago
[deleted]
teleforce 6y ago
Very interesting but nothing really new here, this what IoT and big data is all about. I think the article is even misleading by saying that Shannon's work is not about the content of the information. The statement of not focusing on content the article is referring to is due the scope and context of the communication paper. Of course it will not focusing on the content at the physical layer but his other works really do (see networking layers below). He is highly regarded as the pioneer of information theory and AI including the earlier machine learning e.g chess machine, mouse thru mazes, etc. For more complete hierarchy of knowledge it is the data -> information -> knowledge -> wisdom. For more meaningful in the age of AI and data analytics point of view it is the descriptive - > diagnostic -> prediction -> prescriptive [1]. As you can see the information content referred to by Jane Austin's novels mentioned in the article is even at the lower part of the hierarchy. It is the job of the information theorists and practitioners to develop inference engines (e.g. GPT-3) that can provide the higher level of understanding and proper prescriptions. In addition even if the inference engines are not that capable yet, human-in-the-loop system can be deployed to significantly reduce the cognitive workloads of the human expert (e.g. cardiologist) to perform the better diagnostics for more accurate and timely prediction and prescription. From data centric point of view in electronics and communication engineering Shannon's famous theorems on sampling and communication is a subset of data conditioning (to remove/overcome/mitigate noises on system) and data transmission (to remove/overcome/mitigate noises during transmission). From TCP/IP networking point of view Shannon's theorem of communication, albeit a very important concept, is mainly at physical layer not even at the data link layer and the layers above (networking transport, application). The more universal information theory and entropy partly developed by Shannon is mainly belong to application layer including the AI and data analytics. For more complete treatment refer to the book by the late Dr. David J. C. MacKay entitled Information Theory, Inference and Learning Algorithms [2]. The author has made the book available from the website. [1]https://www.ciscopress.com/store/iot-fundamentals-networking-technologies-protocols-9781587144561 https://www.ciscopress.com/store/iot-fundamentals-networking... [2]http://www.inference.org.uk/mackay/itila/book.html http://www.inference.org.uk/mackay/itila/book.html
benlivengood 6y ago
Cryptography, error coding, and lossy compression are particularly concerned with what the author identifies as Jane Austen's concept of information but also have a very close relationship to Shannon entropy. Basically in cryptography the problem is that plaintext messages reveal important information, and the probability distribution of plaintexts is roughly known. While "attack at dawn" and "attack at noon" may have a lot of letters those two messages may be the only expected messages, and they each may be close to 50% likely, reducing the Shannon entropy to nearly a single bit. Parties wishing to communicate secretly would like to obscure their communication by revealing no information about the actual probability distribution of the plaintexts, so Shannon entropy is maximized (with respect to tractability) when generating a ciphertext. Successful cryptography transforms a message channel into one where e.g. 2^N ciphertexts are all equally likely, hiding the semantic content of both the actual plaintext messages and their probabilities. On the other hand error codes strengthen the semantic meaning of the underlying message by significantly increasing the distribution while adjusting the probabilities toward favoring the important messages. Lossy compression (images, audio, video) is kind of the inverse of error coding. A reduction in the distribution by choosing which parts of the message are least semantically important and removing the distinction between them and making them indistinguishable. Roughly, Austen-information is limited by the Shannon entropy as an upper bound and also by a computable, tractable function of the available Shannon information (which we might as well call the semantic function which translates between bits and useful human representation). In cryptography's case P = D_K(C) is the particular function that turns an N-bit ciphertext from 0 bits of Austen information into N bits of Austin information with the proper key. Lossy compression turns an N-bit message into a semantically useful approximation of an N+M-bit message, excluding the production of a large class of semantically indistinguishable resulting messages in the process. There's another layer that maybe the author was getting at, and that's the human (or any agent) interpretation and response to sensory inputs. At that semantic layer it's again a function from inputs to experiences, responses, or actions. Information theory suggests that such functions are probably most useful if they can extract the most efficiency from all available channels, e.g. distinguish between important states of the world with the fewest bits possible. Going beyond that is a digression into intelligence itself where we have to determine what the most important states of the world are and how to respond.
marttt 6y ago
The site where this article stems from seems like an awesome rabbit hole, a definite flagship for Good Old, Dense and Messy Web: http://www.cs.bham.ac.uk/research/projects/cogaff/ http://www.cs.bham.ac.uk/research/projects/cogaff/ Same counts for Aaron Sloman's web site: https://www.cs.bham.ac.uk/~axs/ https://www.cs.bham.ac.uk/~axs/ I added both links to the wiby.me search engine.
sytelus 6y ago
Just hate these long form articles that takes forever to get to the point. They are designed to have your eye balls kept on page as long as possible as opposed to transfer maximum aloof information in given time.

73 points by benbreen 6y ago

What about this is specific to Austen? All of the quotes in the article are examples of information used as a normal English word, to convey its normal meaning, which everyone knows.

jxy 6y ago
It is a truth universally acknowledged

breck 6y ago

In the past few years I introduced a new concept that provides a simple and universal mathematical way to represent information AND meaning. The basic idea is to recursively build grammars as trees in 2-D Cartesian space. Then you just count the elements. https://github.com/treenotation/research/blob/master/papers/paper3/countingComplexity.pdf https://github.com/treenotation/research/blob/master/papers/... It's still in its infancy, but it is very pragmatic and very useful.

pineconewarrior 6y ago
As someone who is completely ignorant to this field, I found this very readable and interesting. Thanks for your work!

2-tpg 6y ago

Recent work in information theory has formalized the concept of semantic information, where Shannon deliberately avoided semantics, and focused purely on the information content. They define semantic information as: Information which helps an object stay away from thermodynamic equilibrium (no different from entropic noise). Information then has meaning for a receiver if it helps them stay alive. The very same message may be completely "meaningless" to another receiver. https://arxiv.org/abs/1806.08053 https://arxiv.org/abs/1806.08053 Semantic information, autonomous agency, and nonequilibrium statistical physics > Shannon information theory provides various measures of so-called "syntactic information", which reflect the amount of statistical correlation between systems. In contrast, the concept of "semantic information" refers to those correlations which carry significance or "meaning" for a given system. Semantic information plays an important role in many fields, including biology, cognitive science, and philosophy, and there has been a long-standing interest in formulating a broadly applicable and formal theory of semantic information. In this paper we introduce such a theory. We define semantic information as the syntactic information that a physical system has about its environment which is causally necessary for the system to maintain its own existence. "Causal necessity" is defined in terms of counter-factual interventions which scramble correlations between the system and its environment, while "maintaining existence" is defined in terms of the system's ability to keep itself in a low entropy state. We also use recent results in nonequilibrium statistical physics to analyze semantic information from a thermodynamic point of view. Our framework is grounded in the intrinsic dynamics of a system coupled to an environment, and is applicable to any physical system, living or otherwise. It leads to formal definitions of several concepts that have been intuitively understood to be related to semantic information, including "value of information", "semantic content", and "agency".

yters 6y ago
What about webpages on suicide techniques? Do those count as information?
- mhh__ 6y ago
  I guess death could be viewed as the ultimate move away from the equilibrium (of being alive)?
  - yters 6y ago
    Doesn't death allow the body to achieve maximal entropy? Also, this conversation here has nothing to do with keeping me alive, yet I find it informative. Seems the concept of sematic information is missing something, or a lot.
    - 2-tpg 6y ago
      For me, semantic information is information which allows you to predict future world states, or remove uncertainty about current world states, while optimizing for your utility function. If your utility function is to remove all sense data, then a website with suicide techniques helps you predict future world states (if X then Death), and aligns with your utility function, so it is meaningful information for you. But then you get into heterophenomenology territory: someone may state they are not hungry, while desperate for sustenance (or vice versa). People may temporarily act counter to their typical utility function. Do we want to allow such craziness to influence the formal value of semantic information? > has nothing to do with keeping me alive, yet I find it informative. Imagine having to read the exact same information every day. It won't allow you to adapt to a changing environment or be exposed to new ideas, so you can form new ideas yourself. You'd be stuck in a rut, and as good as non-alive. Soon you'll probably perish out of boredom.
      - yters 6y ago
        I spend long periods of time without ingesting sources of information without getting bored and literally dying. Conversely, if I was constantly bombarded with useful information, I would still go crazy, and that could very well lead to literal death. I think staying alive is one piece of meaningful information out of a very large set of meaningful information, most of which has nothing to do with staying alive. In other words, meaningfulness is orthogonal to staying alive.

mhh__ 6y ago

I'm still reading this, so I may be getting the wrong end of the stick, but this seems to be conflating Shannon stating that his work is the study of the behaviour of information in the abstract and the notion that information theory is totally disinterested in anything else. This piece just seems a bit dull to my eyes - maybe there's something interesting trying to get out, but currently it seems like it's fumbling around the edges of Shannon's work and trying to find a dichotomy where there fundamentally isn't one. The connection with Jane Austen in particular seems to be completely spurious - let's say that the information of finding this in Austen's work doesn't seem to be very high.

inimino 6y ago
Conpletely agreed. There was just this: > But [Shannon's] choice of the label "information" in his publications seems to have confused many highly intelligent people. But there was nothing further about who was confused or where or when. The article seems to be clearing up a confusion that doesn't exist.

yters 6y ago

It's a very important point the paper makes, which is the disconnect between what is commonly called Shannon information (he actually called it entropy) and what we call information in normal parlance. A channel may transmit a high amount of Shannon entropy, but contain zero information of the regular sort, since the transmission is just random noise. A concept that is closer to our intuition is mutual information, so named by Shannon's colleague precisely because Shannon entropy is not concerned with semantics.

deleted 6y ago

[deleted]

teleforce 6y ago

Very interesting but nothing really new here, this what IoT and big data is all about. I think the article is even misleading by saying that Shannon's work is not about the content of the information. The statement of not focusing on content the article is referring to is due the scope and context of the communication paper. Of course it will not focusing on the content at the physical layer but his other works really do (see networking layers below). He is highly regarded as the pioneer of information theory and AI including the earlier machine learning e.g chess machine, mouse thru mazes, etc. For more complete hierarchy of knowledge it is the data -> information -> knowledge -> wisdom. For more meaningful in the age of AI and data analytics point of view it is the descriptive - > diagnostic -> prediction -> prescriptive [1]. As you can see the information content referred to by Jane Austin's novels mentioned in the article is even at the lower part of the hierarchy. It is the job of the information theorists and practitioners to develop inference engines (e.g. GPT-3) that can provide the higher level of understanding and proper prescriptions. In addition even if the inference engines are not that capable yet, human-in-the-loop system can be deployed to significantly reduce the cognitive workloads of the human expert (e.g. cardiologist) to perform the better diagnostics for more accurate and timely prediction and prescription. From data centric point of view in electronics and communication engineering Shannon's famous theorems on sampling and communication is a subset of data conditioning (to remove/overcome/mitigate noises on system) and data transmission (to remove/overcome/mitigate noises during transmission). From TCP/IP networking point of view Shannon's theorem of communication, albeit a very important concept, is mainly at physical layer not even at the data link layer and the layers above (networking transport, application). The more universal information theory and entropy partly developed by Shannon is mainly belong to application layer including the AI and data analytics. For more complete treatment refer to the book by the late Dr. David J. C. MacKay entitled Information Theory, Inference and Learning Algorithms [2]. The author has made the book available from the website. [1]https://www.ciscopress.com/store/iot-fundamentals-networking-technologies-protocols-9781587144561 https://www.ciscopress.com/store/iot-fundamentals-networking... [2]http://www.inference.org.uk/mackay/itila/book.html http://www.inference.org.uk/mackay/itila/book.html

benlivengood 6y ago

Cryptography, error coding, and lossy compression are particularly concerned with what the author identifies as Jane Austen's concept of information but also have a very close relationship to Shannon entropy. Basically in cryptography the problem is that plaintext messages reveal important information, and the probability distribution of plaintexts is roughly known. While "attack at dawn" and "attack at noon" may have a lot of letters those two messages may be the only expected messages, and they each may be close to 50% likely, reducing the Shannon entropy to nearly a single bit. Parties wishing to communicate secretly would like to obscure their communication by revealing no information about the actual probability distribution of the plaintexts, so Shannon entropy is maximized (with respect to tractability) when generating a ciphertext. Successful cryptography transforms a message channel into one where e.g. 2^N ciphertexts are all equally likely, hiding the semantic content of both the actual plaintext messages and their probabilities. On the other hand error codes strengthen the semantic meaning of the underlying message by significantly increasing the distribution while adjusting the probabilities toward favoring the important messages. Lossy compression (images, audio, video) is kind of the inverse of error coding. A reduction in the distribution by choosing which parts of the message are least semantically important and removing the distinction between them and making them indistinguishable. Roughly, Austen-information is limited by the Shannon entropy as an upper bound and also by a computable, tractable function of the available Shannon information (which we might as well call the semantic function which translates between bits and useful human representation). In cryptography's case P = D_K(C) is the particular function that turns an N-bit ciphertext from 0 bits of Austen information into N bits of Austin information with the proper key. Lossy compression turns an N-bit message into a semantically useful approximation of an N+M-bit message, excluding the production of a large class of semantically indistinguishable resulting messages in the process. There's another layer that maybe the author was getting at, and that's the human (or any agent) interpretation and response to sensory inputs. At that semantic layer it's again a function from inputs to experiences, responses, or actions. Information theory suggests that such functions are probably most useful if they can extract the most efficiency from all available channels, e.g. distinguish between important states of the world with the fewest bits possible. Going beyond that is a digression into intelligence itself where we have to determine what the most important states of the world are and how to respond.

marttt 6y ago

The site where this article stems from seems like an awesome rabbit hole, a definite flagship for Good Old, Dense and Messy Web: http://www.cs.bham.ac.uk/research/projects/cogaff/ http://www.cs.bham.ac.uk/research/projects/cogaff/ Same counts for Aaron Sloman's web site: https://www.cs.bham.ac.uk/~axs/ https://www.cs.bham.ac.uk/~axs/ I added both links to the wiby.me search engine.

sytelus 6y ago

Just hate these long form articles that takes forever to get to the point. They are designed to have your eye balls kept on page as long as possible as opposed to transfer maximum aloof information in given time.