4 ms·
Interesting read. Was surprised to learn how much damage can be done to a model's parameters without making any discernible difference in its quality of output.
by rglover 1y ago
Interesting read. Was surprised to learn how much damage can be done to a model's parameters without making any discernible difference in its quality of output.
- cma 1y agoI didn't see any mention of dropout in the article, during training parameters or whole layers are removed in different places which helps force it into a distributed representation.
- danielmarkbruce 1y agodropout has sort of dropped out... not used much in LLM training anymore
- cma 1y agoI didn't realize that, looks like it isn't used much at all anymore except in finetuning
- rglover 1y ago"Lessons from the Real World" and "The Limits of Resilience" discussed this.
- cma 1y agoNot quite, that seems to be about quantizing and dropping out after training, not random dropout throughout training. But apparently that isn't done much any more and has been partly superseded with things like weight decay.