Of course BERT is an LM, I never claimed it wasn't. It is just much smaller than what is now termed "LLM" which is typically an extremely large generative decod

Of course BERT is an LM, I never claimed it wasn't. It is just much smaller than what is now termed "LLM" which is typically an extremely large generative decoder-only transformer for general multi-language (often instruction and chat tuned). I was training and finetuned dozens of LMs back when BERT-large was considered a GPU memory issue and I have finetuned hundreds LMs over my NLP research and engineering career. I have even implemented Hidden Markov Model LMs from scratch (as an exercise mostly, performance was mostly bad even in 2015). When I used "specific task", I meant specialized, domain specific tasks like financial sentiment and event extraction in which I hold a PhD. As a matter of fact, for Fiqa SA finetuned Roberta scores 88% F1 while BloombergGPT scores 75% F1. [1] Still very impressive for zero/few shot learner, but depending on data availability, performance targets and inference cost tradeoffs, it might not need your meds. My point was "small" masked encoder transformer LMs like BERT can still hold their own on narrow domain tasks. And what OP claims that all NLP is solved by prompting a general purpose LLM service is simply inaccurate. 1. https://arxiv.org/pdf/2310.12664.pdf https://arxiv.org/pdf/2310.12664.pdf