Fine tuning BERT (what most people in practice do) is so much more efficient. You can do it in 8 hours on a 2080ti. They need to mention that the total number

Fine tuning BERT (what most people in practice do) is so much more efficient. You can do it in 8 hours on a 2080ti. They need to mention that the total number of people training large transformers from scratch is very very small. If wager that the total number of different, uniquely trained (not using previous weights - which reduces compute necessity by massive amounts) language models in existence is in the low hundreds I'd claim that these mass language models serving as the underlying encoding backbone behind more specific systems actually save energy and compute compared to the previous methods (needing far more data and thus more energy spent on getting it combined with less efficient representations like tf-idf causing many classifers to perform very slowly and thus burn lots of energy) Also, much of the recent research in this field is about model pruning, quantization, and any technique you can imagine to reduce training and inference time or memory requirements. All in all, big language models are a net positive for the environment. The effeciency gains in any number of fields from increasingly sophisticated NLP systems far, far outweighs the costs or of training them. Foundational research in environmental conservation will be accelerated by effective NLP semantic search and question answering systems. That's a single, tiny example of the potential for benefits from large language models. Pick a better target.