3 ms·

I've read many articles explains how gpt-3 work, and did a great job. What buffles me is the context consistency. ChatGPT was a huge leap compared to previous

by est 4y ago

I've read many articles explains how gpt-3 work, and did a great job. What buffles me is the context consistency. ChatGPT was a huge leap compared to previous models. I have never seen it failed once. I often use "this" or "that" in my conversation with ChatGPT and it would guess 100% correct what I am refering to. Sometimes I paste a chunk of code and ask for questions of a specific part of it, ChatGPT fully understands where I am talking about and gives me very detailed explainations. It's astonishing and I never knew how it worked so well. Also the title suggests "and why does it work" but I failed to find the reason why ChatGPT worked as in contrast that gpt-3/2/1 never really worked (well)

kqr2 4y ago
What other articles do you recommend?
- est 4y ago
  a recent one: https://news.ycombinator.com/item?id=34726115 https://news.ycombinator.com/item?id=34726115
blackbear_ 4y ago
> I failed to find the reason why ChatGPT worked as in contrast that gpt-3/2/1 never really worked (well) The answer may be as simple as "it has a lot more parameters", plus the additional fine-tuning from human conversation data.
- GaggiX 4y ago
  GPT-3 has the same number of parameters as ChatGPT, it's just the finetuning (or alignment).
  - est 4y ago
    I kind of doubt it? https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155 it says > outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters
    - GaggiX 4y ago
      https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ >ChatGPT is fine-tuned from a model in the GPT-3.5 series https://platform.openai.com/docs/model-index-for-researchers/models-referred-to-as-gpt-3-5 https://platform.openai.com/docs/model-index-for-researchers... >GPT-3.5 series is a series of models that was trained on a blend of text and code from before Q4 2021. The following models are in the GPT-3.5 series: code-davinci-002 is a base model, so good for pure code-completion tasks text-davinci-002 is an InstructGPT model based on code-davinci-002 text-davinci-003 is an improvement on text-davinci-002 the davinci models are the biggest models by OpenAI, they have 175b parameters.

by est 4y ago

kqr2 4y ago

What other articles do you recommend?

est 4y ago
a recent one: https://news.ycombinator.com/item?id=34726115 https://news.ycombinator.com/item?id=34726115

blackbear_ 4y ago

> I failed to find the reason why ChatGPT worked as in contrast that gpt-3/2/1 never really worked (well) The answer may be as simple as "it has a lot more parameters", plus the additional fine-tuning from human conversation data.

GaggiX 4y ago
GPT-3 has the same number of parameters as ChatGPT, it's just the finetuning (or alignment).
- est 4y ago
  I kind of doubt it? https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155 it says > outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters
  - GaggiX 4y ago
    https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ >ChatGPT is fine-tuned from a model in the GPT-3.5 series https://platform.openai.com/docs/model-index-for-researchers/models-referred-to-as-gpt-3-5 https://platform.openai.com/docs/model-index-for-researchers... >GPT-3.5 series is a series of models that was trained on a blend of text and code from before Q4 2021. The following models are in the GPT-3.5 series: code-davinci-002 is a base model, so good for pure code-completion tasks text-davinci-002 is an InstructGPT model based on code-davinci-002 text-davinci-003 is an improvement on text-davinci-002 the davinci models are the biggest models by OpenAI, they have 175b parameters.