8 ms·
The Overfitted Brain: Dreams evolved to assist generalization
- irrational 6y agoTo the best of my knowledge, I have ever dreamed. The abstract speaks of the dangers of not dreaming from lack of sleep and I wonder if the same thing applies to naturally not dreaming at all?
- IAmGraydon 6y agoIt’s far more likely that you just don’t remember your dreams.
- tgv 6y agoWhat’s a dream then?
- irrational 6y agoI’ve been alive for almost 50 years. I would think that if I dreamed, I would have been aware of it at least once. That is approximately 20,000 times I have woken up without any recollection of anything since the moment I fell asleep.
- hyperpallium2 6y ago> To the best of my knowledge, I have ever dreamed. This typo sounds profoundly, of a butterfly; vague and formless, overgeneralizing.
- bil7 6y agoWow. Never before has an abstract felt so mind blowing to read
- belly_joe 6y agoCame here to say the same. At least personally, it's the insight into human brain mechanics and new abilities to test hypotheses in this field that gets me most excited about deep learning developments, rather than the improvements in performance on various tasks.
- deleted 6y ago[deleted]
- choonway 6y agohow about daydreaming? does that work too?
- ancaster 6y agoI recall this being essentially the inspiration for the naming of the Wake-Sleep algorithm of Boltzmann machines.
- isanybodythere 6y agoIn the paper: "It is worth noting that the proposal of a "wake/sleep" specific algorithm for unsupervised learning of generative models based on feedback from stochastic stimulation goes back 25 years (Hinton et al., 1995)"
- sgdpk 6y agoIn the book "Why we sleep?", Matthew Walker suggests something similar. That dream sleep is fundamental in making associations. In this case, generalizing and getting rid of overfitting. He talks a bit about problem solving when sleeping and how this leads to "a-ha" moments when waking up. This means that the idea in this paper is already "out there", contrary to what the abstract states. But it's exciting to have a framework to talk about it quantitatively.
- longtom 6y agoSeems like a testable hypothesis: Prepare a "training" deck of cards with an A side and B side. Each side has a simple symbol on it. Create a second "test" deck of cards which is identical, expect each A symbol is slightly shifted in meaning (e.g. horse -> donkey). The task is to predict side B from side A. If less dreaming leads to overfitting, we would expect REM sleep deprived probands to do better on the "training" set after learning with them and do worse on the "test" set, compared to probands without sleep deprivation.
- darksaints 6y agoThis paper isn't about sleep deprivation, it's about dreams.
- longtom 6y agoRight, REM phase interruption would be sufficient (according to this theory). There are also a bunch of additional variables that one likely cannot (easily) control for. Still, this theory should make predictions of the sort that more overfitting occurs in absence of dreaming.
- seesawtron 6y ago"The goal of this paper is to argue that the brain faces a similar challenge of overfitting, and that nightly dreams evolved to combat the brain's overfitting during its daily learning....Sleep loss, specifically dream loss, leads to an overfitted brain that can still memorize and learn but fails to generalize appropriately." This is a beautiful idea. Will have to read the whole paper to understand how they support this claim.
- deleted 6y ago[deleted]
- jere 6y agoThe abstract is indeed fascinating and I'm reading through the full text, which is so far mostly easy to understand for a layman. The high number of typos throws me a bit though. Amusingly they even misspell their central idea "Overfitted Brian Hypothesis" and I gotta say that the opening reminds me a bit of fluff you see in mindless high school essays. > During the Covid-19 pandemic of 2020, many of those in isolation reported an increase in the vividness and frequency of their dreams (Weaver, 2020), even leading #pandemicdreams to trend on Twitter. Yet dreaming is so little understood there can be only speculative answers to the why behind this widespread change in dream be- havior.
- MaxBarraclough 6y ago> "Overfitted Brian Hypothesis" and I gotta say that the opening reminds me a bit of fluff you see in mindless high school essays My first thought was Monty Python.
- datenhorst 6y agoMaybe presence of typos is the way to signal non-authorship by GPT-3?
- nullc 6y agoGPT-3 can produce typos.
- s_gourichon 6y agoIndeed: GPT-3 is liable to produce the same density of typos as its learning corpus, which is from humans.
- hliyan 6y agoI wonder what psychology as a discipline will look like in twenty years? I feel like what we're learning now about the human mind through our study of neural networks is similar to the early work in cellular biology that eventually replaced metaphor-based models in medicine (e.g. 'humors') with more ontological ones. Freud's Ego, Superego and Id are gone. So are 'complexes'. Our model of the human mind currently seems to be limited to 'conscious and subconscious'. I'm excited at the prospect of something much better.
- TaupeRanger 6y agoI'm foreseeing impending downvotes but I have to rant somewhere. There should be a name for this kind of ridiculous hubris. Unfalsifiable non-insights by people trying to apply arbitrary deep learning algorithms to a brain which is definitely not using any of them. DLcentrism? We don't even know how the brain does almost anything and you think you can use trendy AI topics to explain something as complex and mysterious as dreams? This reminds me of Matt Walker's terrible book on sleep, which, as with almost all neuroscience research recently, tries to explain "why" we have some behavioral pattern or experience, but literally never offers an explanation at all, opting to say "this region lights up in an fMRI machine", as if that answers anything at all. It's like if you asked "why does the heart pump blood?" and a cardiologist answered, "well, the heart is very important for exercise, and people with healthy hearts live longer, and when we attach electrodes to it we see these interesting patterns associated with pulse and breathing...". That's Matt Walker's book applied to the brain. This allows "neuroscience" to get away these ridiculously overextended papers, because you can't disprove anything about something so hard to understand in the first place.
- deleted 6y ago[deleted]
- longtom 6y agoThere are way way worse people out there than the ones who put up curious papers on a pre-print server for others to read for free.
- deleted 6y ago[deleted]
- isseu 6y agoYou can think of DL as a model to explore huge solution spaces, is clearly not the only one but looking at things with some models sometimes makes much more sense and DL has been extremely useful. Is true than real neurons and artificial ones are different but "both are systems that perform complex tasks via the updating of weights within an astronomically large parameter space.". The brain is clearly not using the same mechanisms but it probably have some of the same problems and looking it from a DL perspective (that we understand better than the brain) could help us understand it better.
- cs702 6y ago> ... all DNNs face the issue of overfitting as they learn, which is when performance on one data set increases but the network's performance fails to generalize (often measured by the divergence of performance on training vs testing data sets). This ubiquitous problem in DNNs is often solved by experimenters via "noise injections" in the form of noisy or corrupted inputs. The goal of this paper is to argue that the brain faces a similar challenge of overfitting, and that nightly dreams evolved to combat the brain's overfitting during its daily learning. Actually, there's compelling evidence that overfitting is a necessary step for achieving state-of-the-art performance with DNNs! Many state-of-the-art deep learning models today are trained until they achieve ~100% accuracy on the training data, and then we continue to train them because once they go past this "interpolation threshold" they continue learning to generalize better to unseen data. This is known as the "double descent" phenomenon. See, for example: https://openai.com/blog/deep-double-descent/ https://openai.com/blog/deep-double-descent/ https://arxiv.org/abs/1912.02292 https://arxiv.org/abs/1912.02292 https://arxiv.org/abs/1903.08560 https://arxiv.org/abs/1903.08560 The author makes no mention of double descent and the need for overfitting. In fact, he seems completely unaware of it. -- EDIT: Also, see this comment https://news.ycombinator.com/item?id=23957501 https://news.ycombinator.com/item?id=23957501 elsewhere on this page.
- SoerenMind 6y agoThis paper argues that dreams are a form of data augmentation. Data augmentation is still useful even given double descent. Double descent is theoretically interesting but every DL practitioner will tell you that overfitting is still a problem.
- seesawtron 6y agoMost of the DL models we use are already above the "interpolation threshold" because they use millions of parameters which implies that they are already being trained in the "double descent" regime. Those papers and more [0] argue to explain why using millions of parameters still doesn't overfit our models that are trained on much less data (opposing the traditional Vapnik machine learning argument that your model paramters should not exceed number of data points otherwise you see overfit) because the models still manage to perform well on test data. In this paper, I think the authors are not focusing on overfitting on the data in the traditional Vapnik sense, but overfitting on unimportant information within that data. Noise injection and data augmentation are the techniques we use to de-correlate signal from noise in the data so that the networks can focus on signal and not on noise. Here sleep and dreams are argued to be that source of "noise" for our brain to decorrelate "signal" from the tasks we learn while awake. [0] https://arxiv.org/abs/1903.07571 https://arxiv.org/abs/1903.07571
- amitport 6y ago"Notably, all DNNs face the issue of overfitting as they learn, which is when performance on one data set increases but the network's performance fails to generalize (often measured by the divergence of performance on training vs testing data sets)." Not really. For example, "Gradient Methods Never Overfit On Separable Data" https://arxiv.org/abs/2007.00028 https://arxiv.org/abs/2007.00028
- blackbear_ 6y ago"In this paper, we consider the implicit bias in a well-known and simple setting, namely learning linear predictors (x->x'w) for binary classification with respect to linearly-separable data" Hoping that this applies to deep neural networks is a huge leap of faith to be honest.
- g_airborne 6y agoIf we’re going down this road of theorizing about the human brain based on DNNs, what is the deal with dropout? Could we help human brains with generalization by randomly removing 10% of our newly created connections at the end of each day to improve long term learning? :)
- rtkaratekid 6y agoThat’s called synaptic pruning and, while it most happens as a human matures, there’s evidence indicating that it occurs during sleep in adults to help consolidate the most important connections and remove the unimportant ones. It’s not exactly like dropout, but at a high level it kind of looks like it.
- darksaints 6y agoWhile an interesting hypothesis, we should always be careful with research that tries to derive biological understanding from AI research. AI is, by necessity, a simplification of how our brains work. For example, current neural networks really only have one definition of neuron. But biological neurons can be very different...there are hundreds of types of neurons. Even if we limit the definition of neuron type to just describe variation in switching behavior, mice have been found to have 19 distinct neuron types with distinct switching behavior, and humans likely have dozens more.
- bfirsh 6y agoIf you’re on a phone, here’s an HTML version: https://www.arxiv-vanity.com/papers/2007.09560/ https://www.arxiv-vanity.com/papers/2007.09560/
- deleted 6y ago[deleted]
- metachor 6y agoIs this a form of mechanomorphism, where we try to reason about how human cognition might work by drawing an analogy from how computers work (specifically, overfitting in ANNs) and try to apply it back to humans?
- AndyPatterson 6y agoSkimmed the paper so couldn't possibly give it a fair review but I always feel there's something off when people make comparisons of ANNs to actual biological brains. Even more so when it's the other way about.
- briga 6y agoI love it when two disciplines come together to find new solutions to scientific problems, but something seems off here. An artificial neural network is incredibly simplistic compared the the messy complexity of the brain. Generalization seems to happen in some places and for some dreams, but is that really the only function of dreams? I somehow doubt it.