4 ms·

I don't think it is counter intuitive (maybe if you hold a strong opinion, it is.) There's a growing body of evidence that brains, to some degree, learn by remo

by sva_ 5y ago

I don't think it is counter intuitive (maybe if you hold a strong opinion, it is.) There's a growing body of evidence that brains, to some degree, learn by removing connections, and in some cases, add connections elsewhere. I think part of learning is, that we have some general idea about something, but it turns out to be inaccurate, so we need to specialize it (and get rid off the old connection.) I'm speculating though. I'm a bit confused by the "replanted in others" part though. I'd assume, overall, there is a net loss of synapses when learning (and hence learning becomes more difficult as we age - there just isn't enough entropy to prune away.) See also https://www.tcd.ie/news_events/articles/why-do-we-forget-new-theory-proposes-forgetting-is-actually-a-form-of-learning/ https://www.tcd.ie/news_events/articles/why-do-we-forget-new... And Discussion https://news.ycombinator.com/item?id=29940435 https://news.ycombinator.com/item?id=29940435 The article also has some older references - the idea isn't new.

igorkraw 5y ago
Artificial neural networks have something similar, called the lottery ticket hypothesis
- sva_ 5y ago
  I'm not sure it is quite comparable, as the lottery hypothesis postulates that for every Neural Net with n connections/weights, there is one with n-1 connections, that retains its accuracy. So it doesn't learn by pruning, just becomes more efficient by being smaller. That's at least how I understood it.
  - igorkraw 5y ago
    Your thing would imply a 0 connected neural net could compete with GPT3. I quote The Lottery Ticket Hypothesis: A randomly-initialized, dense neural network contains a subnetwork that is initialised such that — when trained in isolation — it can match the test accuracy of the original network after training for at most the same number of iterations. - Frankle & Carbin (2019, p.2)
    - sva_ 5y ago
      Yeah, that's why it is merely a hypothesis. I think the Lottery Ticket hypothesis goes further and states that this subnetwork also has a subnetwork which matches test accuracy, up to some degree. Else it wouldn't be interesting. Of course there must be a limit...
      - igorkraw 5y ago
        The interesting bit is that you don't need to tweak the subnet, only remove the other noisy connections. If you remove capacity, at some point test accuracy needs to go down. The hypothesis is about the idea that we maybe don't "train" the weights, we "find" them, not about unlimited nesting (as far as I understand it, but I have high confidence. Happy to be proven wrong tho)

by sva_ 5y ago

igorkraw 5y ago

Artificial neural networks have something similar, called the lottery ticket hypothesis

sva_ 5y ago
I'm not sure it is quite comparable, as the lottery hypothesis postulates that for every Neural Net with n connections/weights, there is one with n-1 connections, that retains its accuracy. So it doesn't learn by pruning, just becomes more efficient by being smaller. That's at least how I understood it.
- igorkraw 5y ago
  Your thing would imply a 0 connected neural net could compete with GPT3. I quote The Lottery Ticket Hypothesis: A randomly-initialized, dense neural network contains a subnetwork that is initialised such that — when trained in isolation — it can match the test accuracy of the original network after training for at most the same number of iterations. - Frankle & Carbin (2019, p.2)
  - sva_ 5y ago
    Yeah, that's why it is merely a hypothesis. I think the Lottery Ticket hypothesis goes further and states that this subnetwork also has a subnetwork which matches test accuracy, up to some degree. Else it wouldn't be interesting. Of course there must be a limit...
    - igorkraw 5y ago
      The interesting bit is that you don't need to tweak the subnet, only remove the other noisy connections. If you remove capacity, at some point test accuracy needs to go down. The hypothesis is about the idea that we maybe don't "train" the weights, we "find" them, not about unlimited nesting (as far as I understand it, but I have high confidence. Happy to be proven wrong tho)