5 ms·

AI frenzy almost convinced me that sleep was the training of our neural network with all the prompts of the day. And now this /o\

by lolive 1y ago

ozgung 1y ago
That's what I still 'believe'. Wake-sleep algorithm [1] is a good start for speculation. I think brain needs to be in a different mode to reorganize its weights and to forget unnecessary things to prevent overfitting. In this mode we happen to be unconscious. I also believe dreams are just hallucinations caused by random noise input to the system. The brain converts noise distribution to a meaningful distribution and samples from that. I have zero evidence btw, but I believe these are related. [1] https://en.wikipedia.org/wiki/Wake-sleep_algorithm https://en.wikipedia.org/wiki/Wake-sleep_algorithm
- dr_dshiv 1y ago
  When we don’t sleep, we can lose sensory and cognitive coherence. Mild visual hallucinations begin and reality can start slipping. Sleep itself is characterized by coherent neural activity— the large number of brain regions with synchronized neural activity. The slow waves where huge numbers are all firing close together in a rhythm. Low frequency and high amplitude delta brainwaves (1-2 hertz). Complex adaptive brain activity requires more complex firing than a simple rhythmic frequency. So, in a way, the complex activity must be stopped in order to support global synchrony. Why would our neurons want to all fire synchronously? Well, it is healthy for neurons to fire together in a causal manner— neurons release growth hormones then. That neural growth during synchronized firing is the basis of “neurons that fire together wire together.” And it doesn’t seem coincidental that a successfully predicted model feels good, as in the case of successfully throw a ball in a basket. Neurons are trying to predict other neuron firing and respond to it. If they are unable to effectively, they may become like the 1/3 of our baby neurons in the cortex — they will be pruned and die. Good feelings is positive reinforcement—behaviors leading to good feelings get reinforcement. The feeling of harmony or harmonization, where we have to balance a broad set of internal neural impulses, feels good when we do it well. We feel harmony in music — and in our own internal sensory resonance to the world. Hypothesis 1: the harmonization of neural activity might cause conscious feelings due to the convergence of the activity to platonic forms (see Platonic Representation Hypothesis in LLM research). Returning to sleep — this is a proposal for why sleep feels good. Synchronization might intrinsically feel good. But because the sleep also disrupts your working memory contextual attunements (ie, whatever your day was about) - taking your brain into deep synchrony — it strengthens the overall dendritic connections between the synchronizing neurons. And, because it wears off the edges of your previous experiences — you can return refreshed. In this way, sleep seems to contribute to the overall integrity of the operation of our intelligence. Without it, we lose integrity and internal harmony. And yet, not sleeping is one of my favorite drugs. Can be a major performance enhancer, even if it is variable. Hypothesis 2: Not sleeping increases the (statistical) temperature of the brain.
  - skirmish 1y ago
    > not sleeping is one of my favorite drugs. Can be a major performance enhancer, even if it is variable. Sleep deprivation is a well known treatment for depression [1]. Maybe you lean to the depressive side, that would explain positive effects. [1] https://en.wikipedia.org/wiki/Sleep_deprivation#Treating_depression https://en.wikipedia.org/wiki/Sleep_deprivation#Treating_dep...
incognito124 1y ago
It's not training as much as it's discarding bad examples. Sort of.
andrepd 1y ago
Jesus christ, not even a biology thread is safe in the orange website.
- gitremote 1y ago
  Indeed. Animals without linguistic ability (like fruit flies) need sleep, but after ChatGPT's release in 2022, now tech bros think LLMs specifically might model the animal brain in general because of anthropocentrism and anthropomorphism. It's also a fundamental misunderstanding of how LLMs work, mixing up inference with training.
  - immibis 1y ago
    It was applicable to all neural networks, not just LLMs. Can we say that after ChatGPT's release in 2022, now antitech bros think everything is about LLMs specifically?
    - gitremote 1y ago
      The statement was "AI frenzy almost convinced me that sleep was the training of our neural network with all the prompts of the day." Prompts are specific to LLMs. Most neural networks don't have prompts. Additionally, prompts happen during LLM inference, not LLM training. There are many non-technical people who claim they have experience "training" LLMs, when they are just an end user who added a lot of tokens to the context window during inference.
      - sva_ 1y ago
        > Additionally, prompts happen during LLM inference, not LLM training. It is pretty common during the fine-tuning phase.
        gitremote 1y ago
        Sure. Foundation models aren't fine-tuned, and companies fine-tune foundation models to optimize user experience. So they are modeling the animal brain on an even more specific type of LLM that happens to be related to being a consumer of AI products.
dspillett 1y ago
> sleep was the training of our neural network with all the prompts of the day Periods of sleep certainly seem to be used in that sort of way, but that is an extra use evolution found for the sleep cycle once it existed rather than the reason sleep developed in the first place. There are a number of things that seem tied to, or at least aligned with, our wake/sleep cycle that likely didn't exist when sleep first came about.
nahuel0x 1y ago
Curious how the zeitgeist changes, on a previous AI cycle we could thought sleep was required/generated by a semi-space garbage collection brain-LISP process :)
AIPedant 1y ago
You didn’t need this study to realize that this was wrong: jellyfish and hydras also sleep despite not having a central nervous system. There are indications that sponges sleep too, despite not having any neurons (though obviously it’s somewhat ambiguous): https://www.science.org/content/article/if-alive-sleeps-brainless-creatures-shed-light-why-we-slumber https://www.science.org/content/article/if-alive-sleeps-brai...
xgkickt 1y ago
Rebalancing the weights.

AI frenzy almost convinced me that sleep was the training of our neural network with all the prompts of the day. And now this /o\

by lolive 1y ago

AI frenzy almost convinced me that sleep was the training of our neural network with all the prompts of the day. And now this /o\

ozgung 1y ago

That's what I still 'believe'. Wake-sleep algorithm [1] is a good start for speculation. I think brain needs to be in a different mode to reorganize its weights and to forget unnecessary things to prevent overfitting. In this mode we happen to be unconscious. I also believe dreams are just hallucinations caused by random noise input to the system. The brain converts noise distribution to a meaningful distribution and samples from that. I have zero evidence btw, but I believe these are related. [1] https://en.wikipedia.org/wiki/Wake-sleep_algorithm https://en.wikipedia.org/wiki/Wake-sleep_algorithm

dr_dshiv 1y ago
When we don’t sleep, we can lose sensory and cognitive coherence. Mild visual hallucinations begin and reality can start slipping. Sleep itself is characterized by coherent neural activity— the large number of brain regions with synchronized neural activity. The slow waves where huge numbers are all firing close together in a rhythm. Low frequency and high amplitude delta brainwaves (1-2 hertz). Complex adaptive brain activity requires more complex firing than a simple rhythmic frequency. So, in a way, the complex activity must be stopped in order to support global synchrony. Why would our neurons want to all fire synchronously? Well, it is healthy for neurons to fire together in a causal manner— neurons release growth hormones then. That neural growth during synchronized firing is the basis of “neurons that fire together wire together.” And it doesn’t seem coincidental that a successfully predicted model feels good, as in the case of successfully throw a ball in a basket. Neurons are trying to predict other neuron firing and respond to it. If they are unable to effectively, they may become like the 1/3 of our baby neurons in the cortex — they will be pruned and die. Good feelings is positive reinforcement—behaviors leading to good feelings get reinforcement. The feeling of harmony or harmonization, where we have to balance a broad set of internal neural impulses, feels good when we do it well. We feel harmony in music — and in our own internal sensory resonance to the world. Hypothesis 1: the harmonization of neural activity might cause conscious feelings due to the convergence of the activity to platonic forms (see Platonic Representation Hypothesis in LLM research). Returning to sleep — this is a proposal for why sleep feels good. Synchronization might intrinsically feel good. But because the sleep also disrupts your working memory contextual attunements (ie, whatever your day was about) - taking your brain into deep synchrony — it strengthens the overall dendritic connections between the synchronizing neurons. And, because it wears off the edges of your previous experiences — you can return refreshed. In this way, sleep seems to contribute to the overall integrity of the operation of our intelligence. Without it, we lose integrity and internal harmony. And yet, not sleeping is one of my favorite drugs. Can be a major performance enhancer, even if it is variable. Hypothesis 2: Not sleeping increases the (statistical) temperature of the brain.
- skirmish 1y ago
  > not sleeping is one of my favorite drugs. Can be a major performance enhancer, even if it is variable. Sleep deprivation is a well known treatment for depression [1]. Maybe you lean to the depressive side, that would explain positive effects. [1] https://en.wikipedia.org/wiki/Sleep_deprivation#Treating_depression https://en.wikipedia.org/wiki/Sleep_deprivation#Treating_dep...

incognito124 1y ago

It's not training as much as it's discarding bad examples. Sort of.

andrepd 1y ago

Jesus christ, not even a biology thread is safe in the orange website.

gitremote 1y ago
Indeed. Animals without linguistic ability (like fruit flies) need sleep, but after ChatGPT's release in 2022, now tech bros think LLMs specifically might model the animal brain in general because of anthropocentrism and anthropomorphism. It's also a fundamental misunderstanding of how LLMs work, mixing up inference with training.
- immibis 1y ago
  It was applicable to all neural networks, not just LLMs. Can we say that after ChatGPT's release in 2022, now antitech bros think everything is about LLMs specifically?
  - gitremote 1y ago
    The statement was "AI frenzy almost convinced me that sleep was the training of our neural network with all the prompts of the day." Prompts are specific to LLMs. Most neural networks don't have prompts. Additionally, prompts happen during LLM inference, not LLM training. There are many non-technical people who claim they have experience "training" LLMs, when they are just an end user who added a lot of tokens to the context window during inference.
    - sva_ 1y ago
      > Additionally, prompts happen during LLM inference, not LLM training. It is pretty common during the fine-tuning phase.
      - gitremote 1y ago
        Sure. Foundation models aren't fine-tuned, and companies fine-tune foundation models to optimize user experience. So they are modeling the animal brain on an even more specific type of LLM that happens to be related to being a consumer of AI products.

dspillett 1y ago

> sleep was the training of our neural network with all the prompts of the day Periods of sleep certainly seem to be used in that sort of way, but that is an extra use evolution found for the sleep cycle once it existed rather than the reason sleep developed in the first place. There are a number of things that seem tied to, or at least aligned with, our wake/sleep cycle that likely didn't exist when sleep first came about.

nahuel0x 1y ago

Curious how the zeitgeist changes, on a previous AI cycle we could thought sleep was required/generated by a semi-space garbage collection brain-LISP process :)

AIPedant 1y ago

You didn’t need this study to realize that this was wrong: jellyfish and hydras also sleep despite not having a central nervous system. There are indications that sponges sleep too, despite not having any neurons (though obviously it’s somewhat ambiguous): https://www.science.org/content/article/if-alive-sleeps-brainless-creatures-shed-light-why-we-slumber https://www.science.org/content/article/if-alive-sleeps-brai...

xgkickt 1y ago

Rebalancing the weights.