4 ms·
I guess I can see some parallel the way you describe it, but it is not that tight of an analogy IMO. Correct me if I'm wrong but the Lottery Ticket Hypothesis m
by gota 5y ago
I guess I can see some parallel the way you describe it, but it is not that tight of an analogy IMO. Correct me if I'm wrong but the Lottery Ticket Hypothesis mainly means that, for many of our tasks, we start with needlessly huge networks, and thus a good portion of the learning process is just the network learning to "cope with the noise" from the extraneous connections
Hence why the lottery ticket network (being a smaller version of the same initial network) can achieve similar accuracy (but not 'better') in much shorter time - because you took away all the confounding stuff. That kind of 'counterfactual' doesn't really translate to a biological domain, right?
Also - the baby brain is not really a 'pruned', smaller version of the adult brain, right? Neither is the fetus brain compared to the baby brain (which is more to the point of the article). So the analogy breaks down there too
I guess what the article describes is a bit like "pre-training" a full network with 'synthetic' experiences that quickly tune it for later. In that sense, I think its like these "dreams" are a 'distilled dataset' [1]. The question then is: how is this dataset being passed on from mother/father to child?
[1] https://arxiv.org/abs/1811.10959 https://arxiv.org/abs/1811.10959
Abstract:
> Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few gradient descent steps, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.
- UnFleshedOne 5y ago> Also - the baby brain is not really a 'pruned', smaller version of the adult brain, right I think it is the other way around -- adult brain is a physically pruned version of baby brain. We start with lots of connections from everywhere to wherever and then lose a lot of them as we learn, while strengthening useful ones.