4 ms·
> I disagree with this. Decoders were absolutely dominant in 2020 for chat. GPT2 was considered too dangerous to release, and I remember scrambling to get on th
by matusp 1mo ago
> I disagree with this. Decoders were absolutely dominant in 2020 for chat. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist. It worked.
He's not talking about decoders, he's talking about auto-regression. Before ChatGPT, the dominant paradigm was fine-tuning BERT-like models.
> Before ChatGPT there really wasn’t much of a concept of pre-training and post-training.
Again, people spend years just post-training BERTs in various ways.
- janalsncm 1mo agoGPT2 and 3 work via autoregression. In chat bots, decoders work via autoregression. > people spend years just post-training BERTs in various ways Yes, I was one of them. That’s not called “post-training” it’s called fine-tuning.
- Yoric 1mo agoAsking out of curiosity, because I have limited experience in that domain. I thought that fine-tuning was changing the weights in the model, not the embedding? Or did I misunderstand?
- janalsncm 1mo agoIt’s both. Fine-tuning a BERT model changes its weights, which causes the embeddings to change. For example you might have one model which embeds a text query and another model which embeds an image. You also have a dataset of image + text captions. Training means updating the weights of those models so that the embedding of the image is close to the embedding of its corresponding caption.