3 ms·
What would the size be of the equivalent dictionary to GPT-2?
by rijoja 6y ago
What would the size be of the equivalent dictionary to GPT-2?
- liuliu 6y agoThe training data is around 40GiB: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l... Consider the compression ratio for such text, and you need 2-word / 3-word to 1000-word dictionary, and at 1000-word length, it will be very sparse. I would guess an accelerated structure for such lookup would take around 60GiB. It would require quite a bit of engineering to get it right otherwise can easily blow-up pass that.