4 ms·

eh... Information theory wise, at best you can say you've compressed the knowledge to the action to the steps required to construct the geometry. But you've sti

eh... Information theory wise, at best you can say you've compressed the knowledge to the action to the steps required to construct the geometry. But you've still encoded it on some level. So there is an information-theoretical limit that is still worth thinking about. I think they kind of miss the boat on their discussion of Shannon. You've encoded the structure, and the actual behavior then arises from actually running the network. I think they got a little confused to be honest. And indeed, what people will find is exactly as you say: The DNA encodes how to grow the network such that it will respond with the evolutionary advantageous instinct. I don't know if I'd quite say that no one is looking into that. Obviously we know that there are NN that use deep networks. But also we can see that we can increase the computation power by removing nodes and edges that don't contribute to the correct answer. In evolution, this is a literal cost savings in that you need to feed less neurons.

allovernow 7y ago
I understand your point, and I'm having trouble articulating this thought, but I think with inductive bias you can structure your net such that it is primed to naturally learn certain behaviors that are more complex than the Shannon entropy required in the encoding DNA. The minimal structural information combined with environmental interaction can produce far richer information content, past the limit of the original DNA, but from a design perspective your actual code is very compact because you gain the rest of the knowledge required for innate behaviors naturally from the environment.
- asdff 7y ago
  It's interesting to think about all this. Everything DNA encodes is responses to the environment. On the molecular level, proteins are only functional at certain temperatures, pressures, and pH levels. On the cell level, certain proteins might only be found in specific areas, and essential amino acids might be harvested from the environment rather than produced by the cell itself. On the organism level, we are predisposed to shiver if it's too cold or sweat if it's too hot, an uncomfortable feeling, so innately we go to where the climate is milder and won't kill us. We clothe ourselves and that range is expanded, suddenly there is a selective advantage for being able to clothe yourself, just like there is to be a white beetle on a white sand beach instead of a green one visible to predators, or a cell that's able to just mooch someone elses amino acids rather than expend energy making their own.
- inimino 7y ago
  As the article points out, Shannon information really is the wrong benchmark. Kolmogorov complexity and the busy beaver sequence should demonstrate that there's very little reason to argue for any sort of genetic bottleneck on gene count grounds.
perl4ever 7y ago
"So there is an information-theoretical limit that is still worth thinking about" So, technically, sure, but doesn't the busy beaver sequence imply the potential of anything more complicated than a 5 state turing machine is literally unimaginable? I mean, unimaginably huge even as a multiple of the size of the universe. https://en.wikipedia.org/wiki/Busy_beaver https://en.wikipedia.org/wiki/Busy_beaver
- ggggtez 7y ago
  I think there is perhaps a dozen reasons why this is an incorrect comparison. If we want to measure the information limit of DNA, we can argue the DNA is the program input... sure, so far so good. But then what is the "tape"? The tape must be empty at the beginning of the program... So, I guess we really need to consider the "program" is the entire universe of atoms... Ok... but then... who said that the program ever Halts, exactly? Busy beaver needs to Halt. So, we actually broke two of the fundamental assumptions of the Busy Beaver problem. So I wouldn't make any guarantees about the complexity of DNA based on it.

allovernow 7y ago

I understand your point, and I'm having trouble articulating this thought, but I think with inductive bias you can structure your net such that it is primed to naturally learn certain behaviors that are more complex than the Shannon entropy required in the encoding DNA. The minimal structural information combined with environmental interaction can produce far richer information content, past the limit of the original DNA, but from a design perspective your actual code is very compact because you gain the rest of the knowledge required for innate behaviors naturally from the environment.

asdff 7y ago
It's interesting to think about all this. Everything DNA encodes is responses to the environment. On the molecular level, proteins are only functional at certain temperatures, pressures, and pH levels. On the cell level, certain proteins might only be found in specific areas, and essential amino acids might be harvested from the environment rather than produced by the cell itself. On the organism level, we are predisposed to shiver if it's too cold or sweat if it's too hot, an uncomfortable feeling, so innately we go to where the climate is milder and won't kill us. We clothe ourselves and that range is expanded, suddenly there is a selective advantage for being able to clothe yourself, just like there is to be a white beetle on a white sand beach instead of a green one visible to predators, or a cell that's able to just mooch someone elses amino acids rather than expend energy making their own.
inimino 7y ago
As the article points out, Shannon information really is the wrong benchmark. Kolmogorov complexity and the busy beaver sequence should demonstrate that there's very little reason to argue for any sort of genetic bottleneck on gene count grounds.

perl4ever 7y ago

"So there is an information-theoretical limit that is still worth thinking about" So, technically, sure, but doesn't the busy beaver sequence imply the potential of anything more complicated than a 5 state turing machine is literally unimaginable? I mean, unimaginably huge even as a multiple of the size of the universe. https://en.wikipedia.org/wiki/Busy_beaver https://en.wikipedia.org/wiki/Busy_beaver

ggggtez 7y ago
I think there is perhaps a dozen reasons why this is an incorrect comparison. If we want to measure the information limit of DNA, we can argue the DNA is the program input... sure, so far so good. But then what is the "tape"? The tape must be empty at the beginning of the program... So, I guess we really need to consider the "program" is the entire universe of atoms... Ok... but then... who said that the program ever Halts, exactly? Busy beaver needs to Halt. So, we actually broke two of the fundamental assumptions of the Busy Beaver problem. So I wouldn't make any guarantees about the complexity of DNA based on it.