3 ms·

No, not quite. This is a little hard to describe outside of ML terminology and it's pretty new, but I'll try. The structure I'm describing is at a different sc

No, not quite. This is a little hard to describe outside of ML terminology and it's pretty new, but I'll try. The structure I'm describing is at a different scale from the structures of the individual neurons. Actually, come to think of it, baby motion may be an even better example: When a baby is born, it (possibly) has no understanding of how to move it's arms and legs. But the newborn brain is structured in such a way as to encourage movement of limbs, such that the movement required for learning becomes a high probability event°. In this manner one need not encode all of the information necessary for grasping, merely the conditions necessary to learn it by leveraging information from the environment. If you connect neurons in a particular large scale pattern, and wire some up to ears, some to eyes, some to touch receptors, two things may happen: 1. Your brain is more likely to perform the initial actions required to learn 2. Your brain may more easily learn certain actions The second thing is inductive bias, and I think it will be extremely powerful in the world of ML once we figure out how to harness it. We know that neural nets are hugely wasteful now, there have been papers published on neural network optimizers which can shrink nets by some 80% while maintaining accuracy. Inductive bias is probably such an optimization.

thaumasiotes 7y ago
> In this manner one need not encode all of the information necessary for grasping, merely the conditions necessary to learn it by leveraging information from the environment. That is all of the information necessary for grasping. By your own stipulation, it is sufficient to enable grasping. Nothing more can be necessary. What you're talking about is the degree to which the information is compressed at rest. But index.html and index.html.gz don't contain different information; they contain the same information encoded differently.
- inimino 7y ago
  No, it's not compression at rest. What GP is talking about is the encoding of recipes for acquiring skills rather than encoding the skill itself. The end result is that the skill is acquired, but the skill is not latent in the genome, so your .gz analogy doesn't hold. It's a compression that has access to a vast existing dictionary and makes better use of it then some other. The information develops through the expression of the phenotype in the environment, but it can't be said to be latent in the genome. This is what one would expect given what we already know about biology and Turing machines. The difference is the Turing machine here doesn't start with an empty tape -- far from it.
  - thaumasiotes 7y ago
    > The information develops through the expression of the phenotype in the environment, but it can't be said to be latent in the genome. Why not? Consider birdsong. Many birds must learn "correct" song from models. If raised without good models, they still sing, but they sing a deformed song. This is, correctly, treated as evidence that the song of these birds is a cultural phenomenon -- a bird raised in isolation will sing in an identifiably different way than a bird raised normally. So is normal song arbitrary? Can we say that it isn't encoded in the birds' genetics? Those answers are no and no; when birds with deformed song raise children, those children learn normal song. Normal song is the only steady state of the bird population; any other state will inevitably become normal song. This is sufficient to say that normal song is latent in the birds' genome -- the genome will create it from nothing if necessary -- even though the genome cannot cause a single individual bird to sing correctly in isolation.
    - inimino 7y ago
      That's just one, particularly limited, way of compressing information by using the environment. The information is merely a fixed point of the "function" that is encoded. But what about all the other sources of signal that don't originate from the organism's own genome? If there are other ways to compress the information even further, should we not expect Nature to use all of them? Consider human language. Would humans reproduce spoken language given a generation or two? We can't do the experiment but there are well-known results that strongly suggest not. (Strictly speaking, the genome can't even create the bird from nothing, but I understand what you are getting at.)
      - thaumasiotes 7y ago
        > Would humans reproduce spoken language given a generation or two? We can't do the experiment but there are well-known results that strongly suggest not. Which? I would say the answer is clearly yes. The closest analogue we have is the innovation of sign language by deaf children who were brought into schools with each other. They created working sign language from -- again -- nothing; they could not learn spoken language due to being deaf, and they had no sign language model to learn from. This took only a single generation, even better than the birds. https://en.wikipedia.org/wiki/Nicaraguan_Sign_Language https://en.wikipedia.org/wiki/Nicaraguan_Sign_Language This is an exact match to your hypothetical question if you're willing to overlook the fact that sign languages are not "spoken".

thaumasiotes 7y ago

> In this manner one need not encode all of the information necessary for grasping, merely the conditions necessary to learn it by leveraging information from the environment. That is all of the information necessary for grasping. By your own stipulation, it is sufficient to enable grasping. Nothing more can be necessary. What you're talking about is the degree to which the information is compressed at rest. But index.html and index.html.gz don't contain different information; they contain the same information encoded differently.

inimino 7y ago
No, it's not compression at rest. What GP is talking about is the encoding of recipes for acquiring skills rather than encoding the skill itself. The end result is that the skill is acquired, but the skill is not latent in the genome, so your .gz analogy doesn't hold. It's a compression that has access to a vast existing dictionary and makes better use of it then some other. The information develops through the expression of the phenotype in the environment, but it can't be said to be latent in the genome. This is what one would expect given what we already know about biology and Turing machines. The difference is the Turing machine here doesn't start with an empty tape -- far from it.
- thaumasiotes 7y ago
  > The information develops through the expression of the phenotype in the environment, but it can't be said to be latent in the genome. Why not? Consider birdsong. Many birds must learn "correct" song from models. If raised without good models, they still sing, but they sing a deformed song. This is, correctly, treated as evidence that the song of these birds is a cultural phenomenon -- a bird raised in isolation will sing in an identifiably different way than a bird raised normally. So is normal song arbitrary? Can we say that it isn't encoded in the birds' genetics? Those answers are no and no; when birds with deformed song raise children, those children learn normal song. Normal song is the only steady state of the bird population; any other state will inevitably become normal song. This is sufficient to say that normal song is latent in the birds' genome -- the genome will create it from nothing if necessary -- even though the genome cannot cause a single individual bird to sing correctly in isolation.
  - inimino 7y ago
    That's just one, particularly limited, way of compressing information by using the environment. The information is merely a fixed point of the "function" that is encoded. But what about all the other sources of signal that don't originate from the organism's own genome? If there are other ways to compress the information even further, should we not expect Nature to use all of them? Consider human language. Would humans reproduce spoken language given a generation or two? We can't do the experiment but there are well-known results that strongly suggest not. (Strictly speaking, the genome can't even create the bird from nothing, but I understand what you are getting at.)
    - thaumasiotes 7y ago
      > Would humans reproduce spoken language given a generation or two? We can't do the experiment but there are well-known results that strongly suggest not. Which? I would say the answer is clearly yes. The closest analogue we have is the innovation of sign language by deaf children who were brought into schools with each other. They created working sign language from -- again -- nothing; they could not learn spoken language due to being deaf, and they had no sign language model to learn from. This took only a single generation, even better than the birds. https://en.wikipedia.org/wiki/Nicaraguan_Sign_Language https://en.wikipedia.org/wiki/Nicaraguan_Sign_Language This is an exact match to your hypothetical question if you're willing to overlook the fact that sign languages are not "spoken".