4 ms·
> This is likely because LLMs are typically trained for only a single epoch over massive datasets, which is in contrast to the multi-hundred-epoch training regi
by chaos_emergent 1y ago
> This is likely because LLMs are typically trained for only a single epoch over massive datasets, which is in contrast to the multi-hundred-epoch training regimes for which dropout was first introduced.
Wait, is this true? That seems like a wild statement to make, relatively unsubstantiated?
- typon 1y agoNo this is well known. Look for Table 2.2 in GPT3 paper.
- chaos_emergent 1y agoThank you, that was a wild thing to learn!