6 ms·

Once you prune your model, can you get even better performance by re-training it? I've heard theories that this is the function of sleep in brains.

napo 3y ago
It sounds nice (maybe too nice?). I always wanted to see that it would be necessary to have a "sleeping" phase in AI. It always felt weird that we have to sleep, it doesn't seem to give any evolutionary advantages.
- roomey 3y ago
  It must give an evolutionary advantage, or we wouldn't sleep. It may be hard to pin point exactly what advantage, but as we do it, it must have given us an advantage!
  - dougmwne 3y ago
    Especially considering that it is so widespread in nearly every creature with a brain. And it’s not simply a period of motionless energy conservation but has very specific neural patterns. The science is definitely zeroing in on a connection to learning.
- richardw 3y ago
  I have an unbaked theory, but the very short version is: - Animals that have peaks of energy use outcompete animals that have a steady-state energy use. Catch the animal, then rest and recover. For any given amount of energy, this means we can recruit more in a smaller window compared to an animal that plods along with no recuperative phase. - Many things happen when you're sleeping. Rather than having everything running 24/7, having different phases means we can specialise action and recovery. Since the time is already driven by energy demands, many parts of our body and mind leverage it for different purposes.
  - LesZedCB 3y ago
    1 day of in-context learning and 1 night of fine-tuning on context. that's my pet theory, just shooting from the hip as a total layperson.
- visarga 3y ago
  Continual learning. When models will do that, they will have to sleep as well to avoid catastrophic forgetting.
  - feuerwehrnrw 3y ago
    [dead]
  - klobuerste 3y ago
    [dead]
- nomel 3y ago
  You've, intentionally or not, ignored a whole body of research. An extremely cursory dive into sleep will show all sorts of functional reasons, related to memory formation.
valine 3y ago
Retraining a large GPT is very expensive. The goal of this paper is to help limit the need for retraining after pruning.
- dougmwne 3y ago
  Extremely expensive, though as I understand it, the goal is to get maximum performance out of a given model size so that you can actually inference at product scale. A few extra million for training is expensive, but then consider what it costs to run inference for something like Bing for 100 million daily active users. If they can develop new methods to “overtrain” these models they will get more bang out of the smaller parameter model buck.

napo 3y ago

It sounds nice (maybe too nice?). I always wanted to see that it would be necessary to have a "sleeping" phase in AI. It always felt weird that we have to sleep, it doesn't seem to give any evolutionary advantages.

roomey 3y ago
It must give an evolutionary advantage, or we wouldn't sleep. It may be hard to pin point exactly what advantage, but as we do it, it must have given us an advantage!
- dougmwne 3y ago
  Especially considering that it is so widespread in nearly every creature with a brain. And it’s not simply a period of motionless energy conservation but has very specific neural patterns. The science is definitely zeroing in on a connection to learning.
richardw 3y ago
I have an unbaked theory, but the very short version is: - Animals that have peaks of energy use outcompete animals that have a steady-state energy use. Catch the animal, then rest and recover. For any given amount of energy, this means we can recruit more in a smaller window compared to an animal that plods along with no recuperative phase. - Many things happen when you're sleeping. Rather than having everything running 24/7, having different phases means we can specialise action and recovery. Since the time is already driven by energy demands, many parts of our body and mind leverage it for different purposes.
- LesZedCB 3y ago
  1 day of in-context learning and 1 night of fine-tuning on context. that's my pet theory, just shooting from the hip as a total layperson.
visarga 3y ago
Continual learning. When models will do that, they will have to sleep as well to avoid catastrophic forgetting.
- feuerwehrnrw 3y ago
  [dead]
- klobuerste 3y ago
  [dead]
nomel 3y ago
You've, intentionally or not, ignored a whole body of research. An extremely cursory dive into sleep will show all sorts of functional reasons, related to memory formation.

valine 3y ago

Retraining a large GPT is very expensive. The goal of this paper is to help limit the need for retraining after pruning.

dougmwne 3y ago
Extremely expensive, though as I understand it, the goal is to get maximum performance out of a given model size so that you can actually inference at product scale. A few extra million for training is expensive, but then consider what it costs to run inference for something like Bing for 100 million daily active users. If they can develop new methods to “overtrain” these models they will get more bang out of the smaller parameter model buck.