4 ms·
While there is no real definition of LLM I’m not sure I would say both involve LLMs. There is a trend towards using the hidden state of an LLM as an embedding b
by qeternity 2y ago
While there is no real definition of LLM I’m not sure I would say both involve LLMs. There is a trend towards using the hidden state of an LLM as an embedding but this is relatively recent, and overkill for most use-cases. Plenty of embedding models are not large, and it’s fairly trivial to train a small domain-specific embedding model that has incredible utility.
- refulgentis 2y agoTo some approximation, if you understood what BERT was at the time it was released, you'd consider it the first modern LLM. GPT-1 was OpenAI's BERT. Timeline would be viewed as: 2017: transformers 2018: bert 2018: gpt-1 2019: gpt-2 2020: gpt-3 2022: gpt-3.5 (chatgpt)
- rolisz 2y agoBERT was not large (it was under a billion parameters). And it wasn't an autoregressive language model like GPT
- refulgentis 2y agoRelative to today's models, it is small. For the time, it was large. Re: that it's not auto regressive, that's correct. Things built on eachother smoothly. Transformers to BERT to GPT.