3 ms·
> There's transparent evidence that transformer-based LLMs existed before ChatGPT or Llama. Transformers were first published about in "Attention is all you ne
by filterfiber 3y ago
> There's transparent evidence that transformer-based LLMs existed before ChatGPT or Llama.
Transformers were first published about in "Attention is all you need" in 2017 by google https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762
> There was originally GPT-2, which was not a public release but did get reverse engineered into GPT-Neo and GPT-J (among others).
GPT 2 had a proper release in 2019 with the code, structure, checkpoints, etc. - https://openai.com/research/gpt-2-1-5b-release https://openai.com/research/gpt-2-1-5b-release
> Besides that, there was BERT and T5 from Google, Longformers, XLNet, and dozens of other LLMs that predate the modern ones.
Yes all the big players had various research models. We simply crossed the line into large-scale usefulness just recently (outside of classification/summerization), and like you said OpenAI was the first to really make it accessible to the public.