2 ms·
GPT-2 X 10 is misleading; this model size is 10x, sure, but that doesn't necessarily mean the output will be 10x better. For r/SubSimulatorGPT2, it did go from
by minimaxir 7y ago
GPT-2 X 10 is misleading; this model size is 10x, sure, but that doesn't necessarily mean the output will be 10x better. For r/SubSimulatorGPT2, it did go from 355M to 1.5B recently, but the quality isn't necessarily 4x (although it did improve).
I'm more interested in shrinking models that maintain the same level of generative robustness (e.g. distillation, with distilGPT2)
- lowdose 7y agoThe model did go from a 100 different trained models to 1. So it seems to hold at least 4x as much in knowledge but maybe we should ask the people that actually trained it. Btw thank you for your GPT-2 simple, played around with it last weekend and it made building a toy surprisingly simple!