3 ms·
I didn't see any mention of dropout in the article, during training parameters or whole layers are removed in different places which helps force it into a distr
by cma 1y ago
I didn't see any mention of dropout in the article, during training parameters or whole layers are removed in different places which helps force it into a distributed representation.
- danielmarkbruce 1y agodropout has sort of dropped out... not used much in LLM training anymore
- cma 1y agoI didn't realize that, looks like it isn't used much at all anymore except in finetuning
- rglover 1y ago"Lessons from the Real World" and "The Limits of Resilience" discussed this.
- cma 1y agoNot quite, that seems to be about quantizing and dropping out after training, not random dropout throughout training. But apparently that isn't done much any more and has been partly superseded with things like weight decay.