3 ms·
> once we have trained a large overparameterized neural network we can often compress it to a smaller one with very little loss. So why can't we learn the small
by actionfromafar 2y ago
> once we have trained a large overparameterized neural network we can often compress it to a smaller one with very little loss. So why can't we learn the smaller one directly?
I feel something similar goes on in us humans. Interesting to think about.
- rmnclmnt 2y agoIndeed, you have to over-engineer something before converge to a leaner solution to a problem
- brookst 2y agoAnd I think this is because the ideal complex system is one where all the subsystems and parts combine to produce adequate reliability. Exponentiation means it is more efficient to start by far exceeding the required reliability and then optimizing the most expensive subsystems/parts. It is less efficient and far more frustrating if multiple things have to be improved to meet requirements.
- actionfromafar 2y agoThis is really insightful.
- nopinsight 2y agoThere is indeed an analogous process in the brain. "The number of synapses in the brain reaches its peak around ages 2-3, with about 15,000 synapses per neuron. As adolescents, the brain undergoes synaptic pruning. In adulthood, the brain stabilizes at around 7,500 synapses per neuron, roughly half the peak in early childhood. This figure can vary based on individual experiences and learning." -- written by GPT-4o Confirmed by e.g. https://extension.umaine.edu/publications/4356e/ https://extension.umaine.edu/publications/4356e/
- patcon 2y agoAlso perhaps why the evolved pattern of death is important: a subnetwork is selected in a brain, which is suited to a specific geological, physical, biological and cognitive environment that the brain is navigating. But when the environment shifts beneath the organisim (as culture does and the living world in general does), then the subnetwork is no longer the correct one, and needs to be reinitialized. Or in other words, even in an information theoretic sense, it's true: you can't teach a old dogs new tricks. You need a new dog.
- yumong 2y agoFrom an evolutionary perspective, wouldn't we expect to have developed a way to "reset" parts of our brain then?
- sitkack 2y agoThat is what kids are for.
- actionfromafar 2y agoHaha, exactly. "We did".
- ianmcgowan 2y agoThis reminds me of a hacker news comment that blew my mind - basically "I" am really my genetic code, and this particular body "I" am in is just another computer that the code has been moved to, because the old one is scheduled to be decommissioned. So I am really just the latest instance of a program that has been running continuously since the first DNA/RNA molecules started to replicate.
- interroboink 2y agoIf you're interested in such things, then start layering on epigenetics. The "I" is a product not just of genes, but of your environment as you developed. I was just reading about bees' "royal jelly" recently, and how genetically identical larvae can become a queen or a worker based on their exposure to it. So the program is not just the zeroes and ones, so to speak, but also more nebulous real-time activity, passed on through time. Like a wave on the ocean.
- deleted 2y ago[deleted]
- seanhunter 2y agoYes. My naive intuition about this is you need the extra parameters precisely to do the learning because learning a thing is more complicated than doing the thing once you have learned how. There are lots of natural examples that fit this intuition eg in my mind "junk" DNA is needed because the evolutionary mechanism is learning the sequences which work in a similar way. You don't need all that extra DNA once you have it working but once you have it working there's little selection pressure to clean up/optimise the DNA sequence so the junk stays.