I would recommend reading the paper: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://

I would recommend reading the paper: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l... and the previous paper https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf https://s3-us-west-2.amazonaws.com/openai-assets/research-co... It's a transformer, not LSTM, and it's very large but not structured in a particularly unusual way.