6 ms·

This is a bit surprising since it doesn't treat information as a totally human concept. It discusses 'semantic interpretation' but for some reason concerns itse

This is a bit surprising since it doesn't treat information as a totally human concept. It discusses 'semantic interpretation' but for some reason concerns itself with implementation, which I think should be irrelevant. I discovered this independently and soon after found it echoed in an important textbook (Datalogi, Hans Lunell) around these parts, around 20 years ago. In short; only data exists in the machine. The patterns we coax out of the machine are only meaningful to us, as humans, after we have interpreted the data and turned it into information. A computer never processes information; only data. If one deeply desires a deeper structure to data one can look at processes of convergence, which naturally leads one to study artificial neural networks for their working examples. Implementation however must be irrelevant to information because they are not in a bijective relationship.

yters 7y ago
Information is objectively described by notions such as entropy, algorithmic information, randomness deficiency, etc. Information is mind independent and has empirically describable characteristics.
- omeze 7y ago
  Someone hands me a thumb drive and it consists of one byte, 0000 0001. On Monday, the owner of the thumb drive tells me the contents are a counter of how many apples they had in their kitchen on new year. On Tuesday, the owner tells me the contents are a boolean for whether they took a vacation in 2014. What is the entropy of the data?
  - fossuser 7y ago
    https://www.lesswrong.com/posts/7X2j8HAkWdmMoS8PE/disputing-definitions https://www.lesswrong.com/posts/7X2j8HAkWdmMoS8PE/disputing-... You can deduce that the order of some data implies it's likely information. A person can also have a small amount of data be information even if it would be hard to deduce otherwise. Seems like arguing about this is just arguing about the definition of a word.
    - frenchyatwork 7y ago
      Unfortunately, the reason why it seems like arguing about the definition of a word is that there are there are more concepts at play here than we have words to describe them. When people talk about information as in information theory, what they're talking about is devoid of actual semantic contact, but when normal people people talk about information, that normally implies some sort of semantic content.
      - yters 7y ago
        Information theory addresses semantics through mutual information. 'Meaning' entails reference, and when A refers to B they have mutual information. Mutual information is how Shannon measures the carrying capacity of a channel.
        westoncb 7y ago
        This is still just talking about a property of information that resembles one aspect of what's meant by "meaning". This is all still purely structural and independent of semantic content. The example given above about alternate interpretations of the same bit string on a USB drive gets at the problem. Semantics is about the process of interpreting data into something "meaningful" to some particular system (e.g. a human). It's essentially an importing process: converting data to a format that some system knows how to operate on (i.e. "understands). To keep it brief, meaning (or "information" in the non-jargon sense) is about a relation between some data and a system which that data operates on (or the system operates on the data, depending on how you look at it). "information" from information theory is an intrinsic property of data[0]: the information content doesn't change based on what's interpreting it. (At least that's my understanding.) [0] I'm using "data" here to refer to something like a serialization of a system into a form where information-theoretical analysis can be applied to it (e.g. like a bit string).
        yters 7y ago
        Like I mentioned previously, there are objective sufficient criteria to identify meaning. Even if the interpretive context is subjective. So, we can guarantee true positives, even if we cannot guarantee true negatives.
  - yters 7y ago
    A better example is the difference between the contents of a normal English book vs a randomly generated string of letters. The former has a lot of structural properties the latter lacks, plus shares a large amount of mutual information with a source independent from the physical properties of the book.
    - ganzuul 7y ago
      When we cast our randomness as a string we impose a lot of structure. Proportionally, getting to English from there might involve a lot less new structure than in the previous step. Often I wonder if I'm just spouting gibberish, saying things like this, but it does make sense to me.
      - yters 7y ago
        No, the information cost grows exponentially going from random letters to English sentences. We can measure this with crossentropy to get a probability of generating a well formed English sentence from a uniform distribution over the letters. To generate just a few hundred words we end up needing more trials than are available in the lifespan of our universe.
  - mattkrause 7y ago
    8 bits, by definition. The crux of information theory is that you can analyze a message without know what it means. Mutual information would help you quantify how much this changes your mental model of the owner's life. Maybe you already knew about the vacation (you went too?). It would let you quantify how much this change in definitions affected you.
    - goatlover 7y ago
      What makes those bits zeros and ones? What makes the thumb drive an information storage device? Why do we need to know the mental model of the owner's life to infer the information stored? For information to be objective, it needs to be understood as mind-independent feature of the world. If you have to reference human minds to make it meaningful, then it's not objective. It's a cultural construct.
      - mattkrause 7y ago
        I took the byte and thumb drive bit literally, but if it were some exotic ternary system, it can hold about ~12.7 bits. If they're decimal digits instead, that sequence has a capacity of 10 bans or 26.6 bits. The name "information theory" seems to throw people; imagine it were called coding theory instead and it might seem clearer that we can talk about the "capacity" of different codes and how different representations interact without worrying too much about what they mean. It's similar to algebra, in that we can make statement about `x + y = y + x` or `n + 0 = n` without worrying about whether x is measured in apples, lightyears, or anything at all.
        goatlover 7y ago
        Right, but if a future alien archaeologist were to dig up a thumb drive (preserved in a vault) after humans went extinct, what would indicate that it held bits?
        mattkrause 7y ago
        A 'bit' in information theory is just a unit of measurement, corresponding to the information conveyed by one yes-or-no question (where both answers were previously equally likely). Since it's a fairly natural unit, your alien archaeologist might even use it. However, if they had a weird taboo against binary, they could define a similar concept in other ways. In fact, we humans have too. Bletchley Park used "bans" or hartleys, which is the information carried by a single base-10 digit (without any a priori information). Nats use a base-e representation instead, which ties into statistical physics nicely. There's nothing stopping you from inventing a weird informational unit where 1 corresponds to "the answer to a seven way question were two of the answers are each twice as likely as the rest." So, suppose your alien digs up my thumbdrive. It examines it carefully and notices that each MOSFET is always in one of two states. That suffice to describe its information-carrying capacity in whatever system they use: bits, hartleys, nats, zorblaxen. A similar analysis of their own media would let them say "This thing is so sophisticated, it can hold the work of 10,000 scribes" or "This junk can barely contain a moment of our 16D videos." If your drive were packed with different types of media, they might even discover that some parts of the disk (e.g., ascii text) DON'T use all of the available capacity. None of this, of course, tells them what the data means. For that, you'd need to relate these data to some external source.
- ganzuul 7y ago
  Approximated perhaps, but I would argue that in evaluation it is subjective because it is always compared to something else. The issue abstracts to determinism vs. infinite and unbounded physical entropy because we can't separate ourselves from the system. Decoherence and mixing of quantum 'information' is part of the puzzle, and I would like to claim processes of convergence are another. It becomes difficult to not overload terms while discussing this topic.
  - yters 7y ago
    There are certainly limits to formal definitions of information, but I see them as lower bounds on semantics, such that we can identify meaningful as that which passes the bounds. Doesn't guarantee true negatives, but at least gives us true positives, so is a sufficient if not necessary criteria for meaning.
- sova 7y ago
  Nice, which mind-independent entity did all the observing to come to this mind-independent conclusion?
  - dodobirdlord 7y ago
    Mostly Claude Shannon.
hackinthebochs 7y ago
>The patterns we coax out of the machine are only meaningful to us, as humans, after we have interpreted the data and turned it into information. I disagree. The number of processes that do something computation-like out of the space of all processes is vanishingly small. That we can input some informative sequence of bits into a computer, and the computer transforms that sequence into output which is then a different informative sequence of bits, tells us that computers are intrinsically information processors, and that information is not in general subjective.
- ganzuul 7y ago
  I would argue this is currently a carefully maintained illusion which is shattered by malformed input. Perhaps a universal function exists, whereof all known programs are a special case of. Perhaps humans operate like this or perhaps we are Boltzmann brain apparitions. If not the latter then the subject turns into a discussion of communication through lossy channels and convergence, and the data are again just tokens. (Sorry if I ramble. It is late at night here.)
- goatlover 7y ago
  But what makes the sequence of bits for the input and output informative? The transformation doesn't do that.
  - hackinthebochs 7y ago
    The input bits are presumed informative (i.e. have some mutual information with an external structure). The transformative process is one such that information inherent in the input is preserved and transformed into some amount of new information. That this process preserves and transforms information, when almost all physical processes would destroy such information, demonstrates the coherence between the structure of the transformation (i.e. computation) and the information in the input. If it were the case that the computer plays no informative role in the semantics of the input and output, we would be able to make a computer out of any physical process. But we can't. I can't coax my wall into telling me how to invert a matrix, for example. That certain vanishingly rare physical processes can invert a matrix for me implies the coherence between the semantics of the operations and the semantics of the information its operating on.

yters 7y ago

Information is objectively described by notions such as entropy, algorithmic information, randomness deficiency, etc. Information is mind independent and has empirically describable characteristics.

omeze 7y ago
Someone hands me a thumb drive and it consists of one byte, 0000 0001. On Monday, the owner of the thumb drive tells me the contents are a counter of how many apples they had in their kitchen on new year. On Tuesday, the owner tells me the contents are a boolean for whether they took a vacation in 2014. What is the entropy of the data?
- fossuser 7y ago
  https://www.lesswrong.com/posts/7X2j8HAkWdmMoS8PE/disputing-definitions https://www.lesswrong.com/posts/7X2j8HAkWdmMoS8PE/disputing-... You can deduce that the order of some data implies it's likely information. A person can also have a small amount of data be information even if it would be hard to deduce otherwise. Seems like arguing about this is just arguing about the definition of a word.
  - frenchyatwork 7y ago
    Unfortunately, the reason why it seems like arguing about the definition of a word is that there are there are more concepts at play here than we have words to describe them. When people talk about information as in information theory, what they're talking about is devoid of actual semantic contact, but when normal people people talk about information, that normally implies some sort of semantic content.
    - yters 7y ago
      Information theory addresses semantics through mutual information. 'Meaning' entails reference, and when A refers to B they have mutual information. Mutual information is how Shannon measures the carrying capacity of a channel.
      - westoncb 7y ago
        This is still just talking about a property of information that resembles one aspect of what's meant by "meaning". This is all still purely structural and independent of semantic content. The example given above about alternate interpretations of the same bit string on a USB drive gets at the problem. Semantics is about the process of interpreting data into something "meaningful" to some particular system (e.g. a human). It's essentially an importing process: converting data to a format that some system knows how to operate on (i.e. "understands). To keep it brief, meaning (or "information" in the non-jargon sense) is about a relation between some data and a system which that data operates on (or the system operates on the data, depending on how you look at it). "information" from information theory is an intrinsic property of data[0]: the information content doesn't change based on what's interpreting it. (At least that's my understanding.) [0] I'm using "data" here to refer to something like a serialization of a system into a form where information-theoretical analysis can be applied to it (e.g. like a bit string).
        yters 7y ago
        Like I mentioned previously, there are objective sufficient criteria to identify meaning. Even if the interpretive context is subjective. So, we can guarantee true positives, even if we cannot guarantee true negatives.
- yters 7y ago
  A better example is the difference between the contents of a normal English book vs a randomly generated string of letters. The former has a lot of structural properties the latter lacks, plus shares a large amount of mutual information with a source independent from the physical properties of the book.
  - ganzuul 7y ago
    When we cast our randomness as a string we impose a lot of structure. Proportionally, getting to English from there might involve a lot less new structure than in the previous step. Often I wonder if I'm just spouting gibberish, saying things like this, but it does make sense to me.
    - yters 7y ago
      No, the information cost grows exponentially going from random letters to English sentences. We can measure this with crossentropy to get a probability of generating a well formed English sentence from a uniform distribution over the letters. To generate just a few hundred words we end up needing more trials than are available in the lifespan of our universe.
- mattkrause 7y ago
  8 bits, by definition. The crux of information theory is that you can analyze a message without know what it means. Mutual information would help you quantify how much this changes your mental model of the owner's life. Maybe you already knew about the vacation (you went too?). It would let you quantify how much this change in definitions affected you.
  - goatlover 7y ago
    What makes those bits zeros and ones? What makes the thumb drive an information storage device? Why do we need to know the mental model of the owner's life to infer the information stored? For information to be objective, it needs to be understood as mind-independent feature of the world. If you have to reference human minds to make it meaningful, then it's not objective. It's a cultural construct.
    - mattkrause 7y ago
      I took the byte and thumb drive bit literally, but if it were some exotic ternary system, it can hold about ~12.7 bits. If they're decimal digits instead, that sequence has a capacity of 10 bans or 26.6 bits. The name "information theory" seems to throw people; imagine it were called coding theory instead and it might seem clearer that we can talk about the "capacity" of different codes and how different representations interact without worrying too much about what they mean. It's similar to algebra, in that we can make statement about `x + y = y + x` or `n + 0 = n` without worrying about whether x is measured in apples, lightyears, or anything at all.
      - goatlover 7y ago
        Right, but if a future alien archaeologist were to dig up a thumb drive (preserved in a vault) after humans went extinct, what would indicate that it held bits?
        mattkrause 7y ago
        A 'bit' in information theory is just a unit of measurement, corresponding to the information conveyed by one yes-or-no question (where both answers were previously equally likely). Since it's a fairly natural unit, your alien archaeologist might even use it. However, if they had a weird taboo against binary, they could define a similar concept in other ways. In fact, we humans have too. Bletchley Park used "bans" or hartleys, which is the information carried by a single base-10 digit (without any a priori information). Nats use a base-e representation instead, which ties into statistical physics nicely. There's nothing stopping you from inventing a weird informational unit where 1 corresponds to "the answer to a seven way question were two of the answers are each twice as likely as the rest." So, suppose your alien digs up my thumbdrive. It examines it carefully and notices that each MOSFET is always in one of two states. That suffice to describe its information-carrying capacity in whatever system they use: bits, hartleys, nats, zorblaxen. A similar analysis of their own media would let them say "This thing is so sophisticated, it can hold the work of 10,000 scribes" or "This junk can barely contain a moment of our 16D videos." If your drive were packed with different types of media, they might even discover that some parts of the disk (e.g., ascii text) DON'T use all of the available capacity. None of this, of course, tells them what the data means. For that, you'd need to relate these data to some external source.
ganzuul 7y ago
Approximated perhaps, but I would argue that in evaluation it is subjective because it is always compared to something else. The issue abstracts to determinism vs. infinite and unbounded physical entropy because we can't separate ourselves from the system. Decoherence and mixing of quantum 'information' is part of the puzzle, and I would like to claim processes of convergence are another. It becomes difficult to not overload terms while discussing this topic.
- yters 7y ago
  There are certainly limits to formal definitions of information, but I see them as lower bounds on semantics, such that we can identify meaningful as that which passes the bounds. Doesn't guarantee true negatives, but at least gives us true positives, so is a sufficient if not necessary criteria for meaning.
sova 7y ago
Nice, which mind-independent entity did all the observing to come to this mind-independent conclusion?
- dodobirdlord 7y ago
  Mostly Claude Shannon.

hackinthebochs 7y ago

>The patterns we coax out of the machine are only meaningful to us, as humans, after we have interpreted the data and turned it into information. I disagree. The number of processes that do something computation-like out of the space of all processes is vanishingly small. That we can input some informative sequence of bits into a computer, and the computer transforms that sequence into output which is then a different informative sequence of bits, tells us that computers are intrinsically information processors, and that information is not in general subjective.

ganzuul 7y ago
I would argue this is currently a carefully maintained illusion which is shattered by malformed input. Perhaps a universal function exists, whereof all known programs are a special case of. Perhaps humans operate like this or perhaps we are Boltzmann brain apparitions. If not the latter then the subject turns into a discussion of communication through lossy channels and convergence, and the data are again just tokens. (Sorry if I ramble. It is late at night here.)
goatlover 7y ago
But what makes the sequence of bits for the input and output informative? The transformation doesn't do that.
- hackinthebochs 7y ago
  The input bits are presumed informative (i.e. have some mutual information with an external structure). The transformative process is one such that information inherent in the input is preserved and transformed into some amount of new information. That this process preserves and transforms information, when almost all physical processes would destroy such information, demonstrates the coherence between the structure of the transformation (i.e. computation) and the information in the input. If it were the case that the computer plays no informative role in the semantics of the input and output, we would be able to make a computer out of any physical process. But we can't. I can't coax my wall into telling me how to invert a matrix, for example. That certain vanishingly rare physical processes can invert a matrix for me implies the coherence between the semantics of the operations and the semantics of the information its operating on.