The training data is around 40GiB: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4

The training data is around 40GiB: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l... Consider the compression ratio for such text, and you need 2-word / 3-word to 1000-word dictionary, and at 1000-word length, it will be very sparse. I would guess an accelerated structure for such lookup would take around 60GiB. It would require quite a bit of engineering to get it right otherwise can easily blow-up pass that.