5 ms·

7B parameters is next to nothing when compared to gpt3. If 7B works as well as it does here, A fine tuned 65B model could very easily achieve chatGPT level perf

by valine 4y ago

7B parameters is next to nothing when compared to gpt3. If 7B works as well as it does here, A fine tuned 65B model could very easily achieve chatGPT level performance.

SteveDR 4y ago
I thought ChatGPT is only 20B parameters to begin with? (Source https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is-bigger-better-why-the-chatgpt-vs-gpt-3-vs-gpt-4-battle-is-just-a-family-chat/amp/ https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is...)
- sebzim4500 4y ago
  I haven't seen anything official from OpenAI confirming that ChatGPT has fewer than 175B parameters, although it is a reasonable guess if you read between the lines of their statements. Given the author of that article is a CEO of an 'AI Ad Optimization Platform' I think that number is speculative at best.
  - stavros 4y ago
    ChatGPT is fine-tuned InstructGPT, which has 1.3B parameters, if I'm not mistaken. Reference for the former: https://www.technologyreview.com/2023/03/03/1069311/inside-story-oral-history-how-chatgpt-built-openai/ https://www.technologyreview.com/2023/03/03/1069311/inside-s...
    - sebzim4500 4y ago
      InstructGPT isn't a single model, it's a set of techniques for finetuning a foundation model
      - est 4y ago
        what does "1.3B parameters" mean in this context? Does it mean we load 175B gpt-3 model first, then overwrite 1.3B parameters with InstructGPT? I find this sentence difficult to understand > Our labelers prefer outputs from our 1.3B InstructGPT model over outputs from a 175B GPT-3 model https://openai.com/research/instruction-following https://openai.com/research/instruction-following I am a newbie, plz correct me if I am wrong.
        sebzim4500 4y ago
        They mean that they took a 1.3B parameter model, applied the InstructGPT finetuning model and found that it worked better for their usecase than a 175B parameter model which had not gone through that process.
        est 4y ago
        Ah I got it now. Thanks. From the gpt-3 paper it looks like they have many variants like - GPT-3-350M - GPT-3-1.3B - GPT-3-2.7B - GPT-3-6.7B - GPT-3-13B - GPT-3-175B Ada, Babbage, Curie and Davinci line up closely with 350M, 1.3B, 6.7B, and 175B respectively. The names are pretty suggestive.
- deleted 4y ago
  [deleted]
thot_experiment 4y ago
I mean, I'd push back a little bit on that statement but not much, 13B is crazy nice even a 4bit on a 5 year old graphics card. I was able to get it to pretend to be a VM, cat some code into a file, and then pretend it executed the code and it returned correct answers on the first try. I can't wait to try 30B and 65B when I can get access to hardware large enough.

by valine 4y ago

7B parameters is next to nothing when compared to gpt3. If 7B works as well as it does here, A fine tuned 65B model could very easily achieve chatGPT level performance.

SteveDR 4y ago

I thought ChatGPT is only 20B parameters to begin with? (Source https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is-bigger-better-why-the-chatgpt-vs-gpt-3-vs-gpt-4-battle-is-just-a-family-chat/amp/ https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is...)

sebzim4500 4y ago
I haven't seen anything official from OpenAI confirming that ChatGPT has fewer than 175B parameters, although it is a reasonable guess if you read between the lines of their statements. Given the author of that article is a CEO of an 'AI Ad Optimization Platform' I think that number is speculative at best.
- stavros 4y ago
  ChatGPT is fine-tuned InstructGPT, which has 1.3B parameters, if I'm not mistaken. Reference for the former: https://www.technologyreview.com/2023/03/03/1069311/inside-story-oral-history-how-chatgpt-built-openai/ https://www.technologyreview.com/2023/03/03/1069311/inside-s...
  - sebzim4500 4y ago
    InstructGPT isn't a single model, it's a set of techniques for finetuning a foundation model
    - est 4y ago
      what does "1.3B parameters" mean in this context? Does it mean we load 175B gpt-3 model first, then overwrite 1.3B parameters with InstructGPT? I find this sentence difficult to understand > Our labelers prefer outputs from our 1.3B InstructGPT model over outputs from a 175B GPT-3 model https://openai.com/research/instruction-following https://openai.com/research/instruction-following I am a newbie, plz correct me if I am wrong.
      - sebzim4500 4y ago
        They mean that they took a 1.3B parameter model, applied the InstructGPT finetuning model and found that it worked better for their usecase than a 175B parameter model which had not gone through that process.
        est 4y ago
        Ah I got it now. Thanks. From the gpt-3 paper it looks like they have many variants like - GPT-3-350M - GPT-3-1.3B - GPT-3-2.7B - GPT-3-6.7B - GPT-3-13B - GPT-3-175B Ada, Babbage, Curie and Davinci line up closely with 350M, 1.3B, 6.7B, and 175B respectively. The names are pretty suggestive.
deleted 4y ago
[deleted]

thot_experiment 4y ago

I mean, I'd push back a little bit on that statement but not much, 13B is crazy nice even a 4bit on a 5 year old graphics card. I was able to get it to pretend to be a VM, cat some code into a file, and then pretend it executed the code and it returned correct answers on the first try. I can't wait to try 30B and 65B when I can get access to hardware large enough.