4 ms·

This describes how to summarize a language in a computer system. I think the storage requirements for the semantics is in particular questionable. If you can r

This describes how to summarize a language in a computer system. I think the storage requirements for the semantics is in particular questionable. If you can refer to a particular concept somewhere in the brain you only need a pointer to it. However, single pointers might not be sufficient The apparatus to learn a language might also not so easily distinguished from the structure to store the language or the structure to quickly recall the language. In that case taking about storage is like talking about compressing Plato's world of concepts.

coldtea 8y ago
>This describes how to summarize a language in a computer system. It describes how to summarize a language from a information theory standpoint (which is something different). There's a good reason to believe evolution would use something close to the most information theoretic efficient implementation (and even if not, it's a good lower bound)
- madhadron 8y ago
  > There's a good reason to believe evolution would use something close to the most information theoretic efficient implementation Do you have a citation? I would be very interested to see it. I hate to make analogies between computing and biological systems, but I immediately think of denormalization in databases, error correcting codes, and database replication as examples of optimizations that take you farther from information theoretic efficiency. Information applies very straightforwardly to the study of a spike train of a synapse (source, channel, and receiver). I remain unconvinced that it systems like memory make sense as information channels.
  - pmayrgundter 8y ago
    Shannon observed this in his work towards information theory. Humans are able to predict next letters in a stream of natural language with a probability approaching optimal. The original paper is here: https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf The summary at wikipedia: https://en.wikipedia.org/wiki/Entropy_(information_theory)#Entropy_as_information_content https://en.wikipedia.org/wiki/Entropy_(information_theory)#E... Notably, PPM, the best compression algorithm (at the time), was not able to predict as well. So that's a quantitative demonstration of optimality in language performance that includes a faculty for understanding syntax and semantics and presumably is linked to phonology, pragmatics etc. Chomsky discusses optimality in a 1998 talk here: https://youtu.be/7Sw15-vSY8E https://youtu.be/7Sw15-vSY8E, at about 1h 7m an onwards. He says language evolved rather quickly for such a significant function (estimating it arose quickly ~200kya). He says though many things in evolution are pretty messy, language is close to optimal design. He doesn't provide references, but I think (given the time and academic associations; anyone know?) he may be referring to Optimality Theory: https://en.wikipedia.org/wiki/Optimality_Theory https://en.wikipedia.org/wiki/Optimality_Theory Separately, our knowledge of anatomy of the language faculty is that it is highly localized in the Wernicke and Broca areas of the brain. This also suggests a small modification of an existing brain structure. Additionally, optimality in biological systems has been a productive research direction in a few areas: - Optimality Principles in Biology - https://link.springer.com/book/10.1007/978-1-4899-6419-9 https://link.springer.com/book/10.1007/978-1-4899-6419-9 - Near optimal energy use in the metabolism of cell division - https://arxiv.org/pdf/1209.1179.pdf https://arxiv.org/pdf/1209.1179.pdf - https://www.quantamagazine.org/the-math-that-tells-cells-what-they-are-20190313/ https://www.quantamagazine.org/the-math-that-tells-cells-wha...
    - madhadron 8y ago
      I think we may be talking about different things. You're talking about the language itself. I was talking about the neurological structures that produce the behaviors that are language. I poked through the optimality literature in biology pretty thoroughly a few years back. There is a great deal of interest, but except in a few cases where a simple piece of natural history is nicely described by an evolutionary game, the choice of criterion to optimize is generally arbitrary. Without an evolutionary mechanism and observations of forces driving that mechanism there is no reason beyond just-so stories to apply any particular criterion.
      - pmayrgundter 8y ago
        I see the distinction but I am following Chomsky, that they're aspects of the same thing; language is a faculty in a specific, recently evolved neurological structure. That's a bold theory, but it seems to have held up better than approaches where language is an ability learned through a general cognitive intelligence mechanism. Chomsky says in that lecture that it's separate (and compact and recent) and somehow coupled to the general intelligence and performative areas located elsewhere. I think we're at a point with language and its neural mechanism where by analogy it looks like the phenomenon of flight being closely determined by the exact form and use of a wing. Almost any shape will not do, so there is a form closely (efficiently) following function. Of course there were preadaptations for both which provide very inefficient flight or communication abilities, but something flips and the adaptive utility of the function rapidly sculpts and elaborates the form in that direction. About optimality, generally agree though I think following thermodynamic efficiency seems a deep inductive step as that allows measure of what is arbitrary or not, e.g. in the work of England that I cited.
        madhadron 8y ago
        > I am following Chomsky, that they're aspects of the same thing; language is a faculty in a specific, recently evolved neurological structure. That is a bold hypothesis, certainly, but not one that you could use as a basis for argument at this point in time. I also deeply doubt it when viewed in the context of Wittgenstein's language games. A human and a sheep dog can very effectively engage in a language game, even subtle ones such as naming. Nor is it pure behavioral conditioning, as what makes various dog breeds better at particular things is selection for incidence of particular behaviors. Then you go to a gorilla or a parrot, which can engage in quite abstract language games. Language games with parrots always make you aware of Wittgenstein's lion, though. For Chomsky's hypothesis to be true, you would need to have a particular inflection point where language games become language, and for that inflection point to be reified as the evolution of a specific neural structure instead of exaptation of lots of aspects in the brain. It feels too much like Watson and Crick's nonsense "one gene = one protein," which could have been discarded with a modest amount of thought before it was allowed to damage biology for decades. > I think following thermodynamic efficiency seems a deep inductive step as that allows measure of what is arbitrary or not Until you look at, say, bower birds and realize that conspicuous waste is a signalling mechanism for fitness. There you have an evolutionary game that rewards thermodynamic inefficiency.
- mannykannot 8y ago
  > There's a good reason to believe evolution would use something close to the most information theoretic efficient implementation. People used to think that about genomes, until we got to know in some detail what's in them.
  - stanfordkid 8y ago
    Explain.
    - lake99 8y ago
      As far as the purpose of DNA is to tell the rest of the cell what proteins to synthesize, over 85% percent of our DNA is never translated to protein [1]. But leave alone our knowledge of DNA, biology rarely works in a manner that's best suited for functional efficiency. For examples, look at how the vagus nerve evolved, or look at vestigial organs. The question for the gene is rarely "is this the most efficient way of doing things?" Rather, it is "is this strategy good enough for me (and my descendants) to pass on my genes?" [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3205562/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3205562/
      - stanfordkid 8y ago
        That's a pretty cursory understanding of how DNA works. The "Junk DNA" hypothesis has long been invalidated. The non-coding genome is a large part of what causes us to look different from a mouse or a fruit-fly. Gene regulation (what gene to transcribe, and when) is largely controlled by non-coding DNA: https://en.wikipedia.org/wiki/Regulation_of_gene_expression https://en.wikipedia.org/wiki/Regulation_of_gene_expression There are energy costs associated with useless information being stored and transcribed in the genome... so it is pretty reasonable to believe that information theory does play some (however minor) role in how data is stored.
        mannykannot 8y ago
        We are drifting away from the original issue here. 'However minor' is a long way from near-maximal efficiency, and in the context of the original claim, 'efficient' means compact encoding. The fact that the DNA involved in gene regulation is mostly non-coding does not mean that most non-coding DNA has a regulatory purpose. If it did, changes in non-coding DNA would be more important, in aggregate, than changes in coding DNA are, with respect to evolution and disease, on account of its prevalence.

coldtea 8y ago

>This describes how to summarize a language in a computer system. It describes how to summarize a language from a information theory standpoint (which is something different). There's a good reason to believe evolution would use something close to the most information theoretic efficient implementation (and even if not, it's a good lower bound)

madhadron 8y ago
> There's a good reason to believe evolution would use something close to the most information theoretic efficient implementation Do you have a citation? I would be very interested to see it. I hate to make analogies between computing and biological systems, but I immediately think of denormalization in databases, error correcting codes, and database replication as examples of optimizations that take you farther from information theoretic efficiency. Information applies very straightforwardly to the study of a spike train of a synapse (source, channel, and receiver). I remain unconvinced that it systems like memory make sense as information channels.
- pmayrgundter 8y ago
  Shannon observed this in his work towards information theory. Humans are able to predict next letters in a stream of natural language with a probability approaching optimal. The original paper is here: https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf The summary at wikipedia: https://en.wikipedia.org/wiki/Entropy_(information_theory)#Entropy_as_information_content https://en.wikipedia.org/wiki/Entropy_(information_theory)#E... Notably, PPM, the best compression algorithm (at the time), was not able to predict as well. So that's a quantitative demonstration of optimality in language performance that includes a faculty for understanding syntax and semantics and presumably is linked to phonology, pragmatics etc. Chomsky discusses optimality in a 1998 talk here: https://youtu.be/7Sw15-vSY8E https://youtu.be/7Sw15-vSY8E, at about 1h 7m an onwards. He says language evolved rather quickly for such a significant function (estimating it arose quickly ~200kya). He says though many things in evolution are pretty messy, language is close to optimal design. He doesn't provide references, but I think (given the time and academic associations; anyone know?) he may be referring to Optimality Theory: https://en.wikipedia.org/wiki/Optimality_Theory https://en.wikipedia.org/wiki/Optimality_Theory Separately, our knowledge of anatomy of the language faculty is that it is highly localized in the Wernicke and Broca areas of the brain. This also suggests a small modification of an existing brain structure. Additionally, optimality in biological systems has been a productive research direction in a few areas: - Optimality Principles in Biology - https://link.springer.com/book/10.1007/978-1-4899-6419-9 https://link.springer.com/book/10.1007/978-1-4899-6419-9 - Near optimal energy use in the metabolism of cell division - https://arxiv.org/pdf/1209.1179.pdf https://arxiv.org/pdf/1209.1179.pdf - https://www.quantamagazine.org/the-math-that-tells-cells-what-they-are-20190313/ https://www.quantamagazine.org/the-math-that-tells-cells-wha...
  - madhadron 8y ago
    I think we may be talking about different things. You're talking about the language itself. I was talking about the neurological structures that produce the behaviors that are language. I poked through the optimality literature in biology pretty thoroughly a few years back. There is a great deal of interest, but except in a few cases where a simple piece of natural history is nicely described by an evolutionary game, the choice of criterion to optimize is generally arbitrary. Without an evolutionary mechanism and observations of forces driving that mechanism there is no reason beyond just-so stories to apply any particular criterion.
    - pmayrgundter 8y ago
      I see the distinction but I am following Chomsky, that they're aspects of the same thing; language is a faculty in a specific, recently evolved neurological structure. That's a bold theory, but it seems to have held up better than approaches where language is an ability learned through a general cognitive intelligence mechanism. Chomsky says in that lecture that it's separate (and compact and recent) and somehow coupled to the general intelligence and performative areas located elsewhere. I think we're at a point with language and its neural mechanism where by analogy it looks like the phenomenon of flight being closely determined by the exact form and use of a wing. Almost any shape will not do, so there is a form closely (efficiently) following function. Of course there were preadaptations for both which provide very inefficient flight or communication abilities, but something flips and the adaptive utility of the function rapidly sculpts and elaborates the form in that direction. About optimality, generally agree though I think following thermodynamic efficiency seems a deep inductive step as that allows measure of what is arbitrary or not, e.g. in the work of England that I cited.
      - madhadron 8y ago
        > I am following Chomsky, that they're aspects of the same thing; language is a faculty in a specific, recently evolved neurological structure. That is a bold hypothesis, certainly, but not one that you could use as a basis for argument at this point in time. I also deeply doubt it when viewed in the context of Wittgenstein's language games. A human and a sheep dog can very effectively engage in a language game, even subtle ones such as naming. Nor is it pure behavioral conditioning, as what makes various dog breeds better at particular things is selection for incidence of particular behaviors. Then you go to a gorilla or a parrot, which can engage in quite abstract language games. Language games with parrots always make you aware of Wittgenstein's lion, though. For Chomsky's hypothesis to be true, you would need to have a particular inflection point where language games become language, and for that inflection point to be reified as the evolution of a specific neural structure instead of exaptation of lots of aspects in the brain. It feels too much like Watson and Crick's nonsense "one gene = one protein," which could have been discarded with a modest amount of thought before it was allowed to damage biology for decades. > I think following thermodynamic efficiency seems a deep inductive step as that allows measure of what is arbitrary or not Until you look at, say, bower birds and realize that conspicuous waste is a signalling mechanism for fitness. There you have an evolutionary game that rewards thermodynamic inefficiency.
mannykannot 8y ago
> There's a good reason to believe evolution would use something close to the most information theoretic efficient implementation. People used to think that about genomes, until we got to know in some detail what's in them.
- stanfordkid 8y ago
  Explain.
  - lake99 8y ago
    As far as the purpose of DNA is to tell the rest of the cell what proteins to synthesize, over 85% percent of our DNA is never translated to protein [1]. But leave alone our knowledge of DNA, biology rarely works in a manner that's best suited for functional efficiency. For examples, look at how the vagus nerve evolved, or look at vestigial organs. The question for the gene is rarely "is this the most efficient way of doing things?" Rather, it is "is this strategy good enough for me (and my descendants) to pass on my genes?" [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3205562/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3205562/
    - stanfordkid 8y ago
      That's a pretty cursory understanding of how DNA works. The "Junk DNA" hypothesis has long been invalidated. The non-coding genome is a large part of what causes us to look different from a mouse or a fruit-fly. Gene regulation (what gene to transcribe, and when) is largely controlled by non-coding DNA: https://en.wikipedia.org/wiki/Regulation_of_gene_expression https://en.wikipedia.org/wiki/Regulation_of_gene_expression There are energy costs associated with useless information being stored and transcribed in the genome... so it is pretty reasonable to believe that information theory does play some (however minor) role in how data is stored.
      - mannykannot 8y ago
        We are drifting away from the original issue here. 'However minor' is a long way from near-maximal efficiency, and in the context of the original claim, 'efficient' means compact encoding. The fact that the DNA involved in gene regulation is mostly non-coding does not mean that most non-coding DNA has a regulatory purpose. If it did, changes in non-coding DNA would be more important, in aggregate, than changes in coding DNA are, with respect to evolution and disease, on account of its prevalence.