Ok, interesting point. Ultimately, one could say that a fairly simple set of rules (the laws of physics), generate all complexity in the universe. To take the

Ok, interesting point. Ultimately, one could say that a fairly simple set of rules (the laws of physics), generate all complexity in the universe. To take the NN example: Clearly it applies to species, like humans, with a significant cultural and learning element to our being. Much of our complexity is learned socially or by interacting with our environment. In fact, this is done by training the NN in our brains. Edit: Note that our brains can store much more information than our DNA, making the upper bound on complexity for a human the sum of storage capacity in the DNA + the sum of storage capacity in the brain. But for species with little ability to learn, like jellyfish, wouldn't it be more appropriate to say that the "training of the NN" is analogous to evolution optimizing the DNA? That would mean that, for a given individual, their complexity is all included in their DNA at birth? In that case, one could argue that the length of the DNA string does create an upper bound on how much complexity it can contain. I suppose that is the case for the NN, too. The total number of bits in the weights of the network is an upper bound on the complexity a given, already trained, network can have. (Of course, the possible complexity of the network is way, way higher than that of the program that creates it.)