8 ms·
Artificial brains may need sleep too
- BrissyCoder 6y agoHaven't read but I'm betting that there's an analogy to disk defragging...
- jedberg 6y agoDamn that was fascinating. And it makes a ton of sense too. In case anyone is looking, here is they paper they presented: https://drive.google.com/file/d/13u14zNsi7uthLpwCoMrckwhasauw157I/view https://drive.google.com/file/d/13u14zNsi7uthLpwCoMrckwhasau...
- panpanna 6y agoNot sure what "stabilization" means in this context. But isn't what they call sleeping basically regularization? In particular, mini-batch and drop-out??
- ouid 6y agoI don't know what you think regularization means, but there is no such thing as "just regularization". Not all regularization is equal, and different types of regularization combine in unpredictable ways. The regularization occurring in the context of this research is as follows. The model that they are creating is a sparse model. This means that, for the vast majority of possible inputs, the machine should produce no hypothesis whatsoever about what it is looking at. In fact, a good sparse representation only produces a representation when the input is in the space of valid inputs. If the model is given a valid input, it should come up with a representation of that input, if it is given garbage, it should prefer to not try. Signal is pretty easy to find, you just open your eyes, explore your environment, and update your weights accordingly. We would alternatively like to train the model to recognize when it is looking at nothing, but we have to be pretty careful about what we show it. If any signal is present at all, we definitely don't want to discourage our model from finding it. That was, after all, the point of the model in the first place. Fortunately, because of the size of the space, we can be certain that if we completely shut off the inputs and feed it noise instead, it is very very far from a valid input, and anything that it includes in its representation was included in error.
- deleted 6y ago[deleted]
- jhanschoo 6y agoIt's important to note that what is being investigated are neuromorphic processors. i.e. attempts to model biological brains. A paragraph from the article directly addresses why conventional regularization isn't used. > “The issue of how to keep learning systems from becoming unstable really only arises when attempting to utilize biologically realistic, spiking neuromorphic processors or when trying to understand biology itself,” said Los Alamos computer scientist and study coauthor Garrett Kenyon. “The vast majority of machine learning, deep learning, and AI researchers never encounter this issue because in the very artificial systems they study they have the luxury of performing global mathematical operations that have the effect of regulating the overall dynamical gain of the system.”
- erikerikson 6y agoSee also: https://en.wikipedia.org/wiki/Spiking_neural_network https://en.wikipedia.org/wiki/Spiking_neural_network https://en.m.wikipedia.org/wiki/Hopfield_network https://en.m.wikipedia.org/wiki/Hopfield_network
- tetris11 6y agoIf lack of sleep is linked to dementia and poor long term learning in humans, and that every living organism in the world sleeps, I can believe machines might need it too
- VMG 6y agoThat is flawed reasoning. Every intelligent organism in the world needs blood, and machines do not. Modern airplanes don't flap their wings either.
- sgillen 6y agoThey don’t need blood but they need energy, planes don’t flap their wings but they need to generate lift. Whatever the function of sleep is, I think artificial minds will likely need it too.
- Smaug123 6y agoWe have a handy analogy in the computing world, in that stop-the-world and concurrent garbage collection algorithms both exist. If sleep is pure garbage-collection, then I don't think there's any strong reason to expect it to be required of all possible mind designs. However, it seems increasingly likely that the empirical question may be answered at some point soon!
- Erlich_Bachman 6y agoBut it shouldn't surprise you if it did require sleep. Now the airplanes don't happen to flap their wings, but you shouldn't have been surprised or in disbelief if they did require it. We are still too far off from AGI to know for sure. Time will tell.
- smusamashah 6y agoNo
- op03 6y ago> spiking neuromorphic processor what is this pls?
- cinntaile 6y agoMy (non-expert) take on this... A neuromorphic processor is a processor that tries to function as the brain does. I say tries to because the models are still very simplified versions of how we think that a neuron works. The spiking part comes from one of these models. A neuron will only send an electrical signal to another neuron once a certain threshold is met, this is what is meant by a spike. What this allows us to do is to add a temporal/time component to a neural network. You've probably heard of "neurons that wire together, fire together". If you have a lower threshold it means that the neuron you are sending the signal to is more relevant for whatever thought process is going on right now and vice versa. New input can change these thresholds. The biggest promise of spiking neuromorphic computing seems to be a massive reduction in energy usage while still offering decent accuracy. So for example you could use it to train a neural network to get to 80% accuracy after which you'd let another type take over to get to 95%. This field is still in its infancy though, so expect things to change/improve fast.
- op03 6y agoInteresting. Thanks for that.
- erikerikson 6y ago> neurons that wire together, fire together i.e. Hebbian learning Usually with a Hopfield network or something similar. My personal take is that biologically plausible approaches are attempting to discover (and in lieu of understanding, harness) the algorithms (or mechanisms that cause emergent intelligence) of our brains.
- realtalk_sp 6y agoReally tiring of this clickbaity nonsense infecting science and math. This analogy is so tenuous you could snap it with a feather.
- baxtr 6y ago”AI may need food too” “AI may need friends too” “AI may need training too!”
- shahbaby 6y agoFor ~70 years, the brightest minds on earth have been trying and failing to solve AI. Many of them refusing to consider that something useful can be learned from the human brain; the gold standard of what it means to be intelligent. Yet today, cutting edge deep learning technology is based on a crude and increasingly inaccurate model of neurons. If we're now making discoveries that are revealing artificial processes that are similar to our own, it's a sign we're headed in the right direction.
- nix23 6y ago>the only indisputable example of intelligence in the known universe I dispute that any time any day with you, just because we define what intelligence is, do's not mean that we are intelligent.
- shahbaby 6y agoUnless we've specifically defined intelligence in a way that reflects ourselves. The useful things that we want to get out of an AI system, i.e generalized learning of abstract concepts, are most clearly demonstrated by the human brain. Since we now seem to prefer down votes over discussion, I'll just leave this with my own speculation that the reason for this strange avoidance of the brain is that it's a dead end for both academia and industry. It's much easier and more profitable to expand on already existing machine learning technologies than to try and find some revolutionary breakthrough in neuroscience.
- irrational 6y agoHave you tried turning it off and turning it back on? It works for all other computers, why not artificial brains?
- Smaug123 6y agoThe obvious answer is that if the source of the problems is in state that is persisted, then an active cleanup process may be required. Turning a database off and on again will not help reduce its index bloat.
- bachmeier 6y agoQuestions. Does this explain - Why we dream? - Why we process information while we sleep? - Why lack of sleep is associated with serious health problems? - Why the amount of sleep is so long - about a third of the day in the most common case - but varies a lot from person to person and by age? A model of a brain as a neural network that's constantly learning, but for stability purposes needs a restart every so often, doesn't seem convincing in terms of other things we think we know about sleep.
- Symmetry 6y ago1) REM sleep is associated learning new skills. We can teach experimental subjects a new skill in the laboratory, selectively interrupt their REM sleep, and see that their skill doesn't increase to the same extent as a control group who don't have that phase interrupted. The same isn't true with non-REM sleep. We don't really understand the mechanism though and this potentially does bear directly on the question. 2) Well, it would get in the way of actually living? If you're talking down some stairs and then your brain's motor cortex starts working on consolidating a skill and you fall you'd hurt yourself. 3) I don't think we know that. Possibly we had to sleep for memory related reasons and our bodies just evolved to use that time for other sorts of important housekeeping when it wasn't moving around doing stuff? 4) Learning is very important and takes time. In addition to the procedural memory consolidation in REM sleep we also move long term memory to a more stable form of storage in other phases. But neural nets don't really have anything analogous to a brain's explicit memory so this doesn't really bare on that.
- kthejoker2 6y ago#3: Doesn't hibernation suggest the reverse? Sleep is primarily an energy conserving activity, being able to survive periods of low energy availability (winter) is an evolutionary advantage.
- waheoo 6y agoProbably. The why we sleep book made me think of this hypothesis. He states that dreaming likely came before consciousness too which I found interesting. As in, our early brains form reality using the same concept as dreaming.
- blackboxlogic 6y agoGood idea or bad idea: browser profiles where User enters their level of understanding for a variety of subjects. browser sends the level specific to each web request. Server customizes content, returning the same article, styles ranging from explain-it-like-I'm-five to just-show-the-research-paper.
- truculent 6y agoThat would depend on how it's exposed to the user. Are you empowering the user to run up and down that scale, or just feeding them whatever your model thinks is best?
- blackboxlogic 6y agoA slider at the top of the page gradually adding detail and vocabulary?
- truculent 6y agoThat could be cool. Another idea would be to present an executive summary at the top and more detailed versions as you scroll down. Sort of akin to how pictures load from more -> less pixelated on a slow internet connection.
- penagwin 6y agoWikipedia sorta supports something like this with a lot of English articles if you chance the domain to simple.wikipedia.com for the language. Then it gives you articles that are much more ElI5.
- wcoenen 6y agoThis exists for media types ("Accept" HTTP request header) and for language ("Accept-Language" HTTP request header). You can configure your browser to tell it which languages you understand, and your order of preference. You're proposing something like "Accept-Semantic-Complexity", or maybe extend Accept-Language somehow to include acceptable complexity. It could work in principle but I doubt anyone would use it. Many websites already have a tendency to ignore Accept-Language and instead use some fuzzy region detection when they offer multiple languages.
- MindGods 6y agoA good discussion from May: https://news.ycombinator.com/item?id=23366516 https://news.ycombinator.com/item?id=23366516
- 2OEH8eoCRo0 6y agoWhy do things need to be unconscious for brain repairs to happen? Wouldn't evolution find it advantageous for a mind to stay awake while this happens? Yet no animal has found a way to do so.
- gfody 6y agosome animals alternate sleep between each hemisphere, there's a bunch of examples of exotic sleep patterns in 'why we sleep'
- 2OEH8eoCRo0 6y agoThat's a workaround to ultimately still sleeping.
- Ericson2314 6y agoMaybe instead of thinking of sleep as a crutch, think of it as an optimization enabled by periods of decreased predator effectiveness? Think about how real-time guarantees on GC are bad for throughput. Evolution can get "stuck" when optomizations become obligatory. One theory of "hot bloodedness" (which is not one thing, nor something that evolved once, to be clear) is large species that passively maintained some body temperature then shrinking and being forced to actively maintain it.
- Pick-A-Hill2019 6y ago(previously submitted by laurex) https://news.cornell.edu/stories/2020/03/researchers-sniff-out-ai-breakthroughs-mammal-brains https://news.cornell.edu/stories/2020/03/researchers-sniff-o...
- micimize 6y agoFor a functionalist, this and every other "more general AI might need X also" is expected. Like, humans need sleep, pain, anxiety, social mediation, etc to achieve our high level of generality. IMO, our base case should be that a sufficiently general AI will need them as well. I'm very excited by the budding neuromorphic phase of generality research for these kinds of discoveries
- AnHonestComment 6y agoProduction ML systems already do this: servers used for peak capacity are repurposed during lulls to retrain the model and do tasks like compact data. That retrained model is then shared across the system. So I’d argue that ML systems already “sleep”, it just looks more like dolphin sleep due to their similar need for continuous operation.
- historyremade 6y agoI'm betting that General AI need many more things and some may be expensive to supply and once people realize that the hype will go down in road.
- combatentropy 6y ago> The best results came when they used waves of so-called Gaussian noise, which includes a wide range of frequencies and amplitudes. Yesterday I was listening again to some videos of rain, to help me relax. I know very little about what happens when we sleep or why we dream. But if it is sort of like exposing our brains to soft noise, then that helps me understand why rain relaxes me.