4 ms·

LLamA 2 at 70B is, let’s say pessimistically 70% as good as GPT3.5. This makes me think that OpenAI is lying about their parameter count, are vastly less effici

LLamA 2 at 70B is, let’s say pessimistically 70% as good as GPT3.5. This makes me think that OpenAI is lying about their parameter count, are vastly less efficient than LLaMA, or, the lager model sizes have diminishing returns. Either way, your point is a good one. Something doesn’t add up.

dr_dshiv 3y ago
IMO Llama2 really isn’t close to 3.5. It still has regular mode collapse (or whatever you call getting repetitive and nonsensical responses after a while), it has very poor mathematical/logical reasoning and is not good at following multi-part instructions. It just sounds like 3.5/4 because it was trained on it.
- geysersam 3y ago
  You're mixing up the language model with the chat bot. The llama2 is a language model. I imagine the language model behind chatgpt is not much different (perhaps it's better, but not by many months AI research time). It likely also suffers from "mode collapse" issues etc. But 3.5 also has a lot of systems around it that detects mode collapse and applies some kind of mitigation, forcing the model to give a more reasonable output. Mathematical / logical reasoning questions are likely also detected hand passed on in some form to a separate system.
  - dr_dshiv 3y ago
    So this would be testable by showing that chatGPT makes more mistakes than prompting via API? Or would you consider the API a chatbot, too?
    - geysersam 3y ago
      I don't think there's any public interface to the LLM underlying ChatGPT, so the only ones able to test this are openAI engineers.
- vorticalbox 3y ago
  This is what presence_penalty and frequency_penalty are for.
- nl 3y ago
  Llama 2 wasn't trained on ChatGPT/GPT4. I think maybe you are thinking of the Vicuna models? https://lmsys.org/blog/2023-03-30-vicuna/ https://lmsys.org/blog/2023-03-30-vicuna/
  - dr_dshiv 3y ago
    So it’s true that it would violate the OpenAI terms for Llama to be trained with ChatGPT completions, but how do we know? We don’t know the training data for Llama, we just get weights.
    - nl 3y ago
      The Llama2 paper describes the training data in some detail.
mattlutze 3y ago
We just don't have the information to make judgements, much less leaping to "they must be lying." There's a few public numbers from a handful of foundation models as to performance vs parameter count vs architecture generation. Not being able to compare in detail the architecture of the various closed models nor being more rigorous on training with progressively sized parameter sets, the conclusion at the moment is a general feeling or conjecture.
qrios 3y ago
Without questioning the statement '70% as good as GPT3.5', but wouldn't that be quantifying a quality, and a Turing test? Also: maybe these missing 30% are the hard part.

dr_dshiv 3y ago

IMO Llama2 really isn’t close to 3.5. It still has regular mode collapse (or whatever you call getting repetitive and nonsensical responses after a while), it has very poor mathematical/logical reasoning and is not good at following multi-part instructions. It just sounds like 3.5/4 because it was trained on it.

geysersam 3y ago
You're mixing up the language model with the chat bot. The llama2 is a language model. I imagine the language model behind chatgpt is not much different (perhaps it's better, but not by many months AI research time). It likely also suffers from "mode collapse" issues etc. But 3.5 also has a lot of systems around it that detects mode collapse and applies some kind of mitigation, forcing the model to give a more reasonable output. Mathematical / logical reasoning questions are likely also detected hand passed on in some form to a separate system.
- dr_dshiv 3y ago
  So this would be testable by showing that chatGPT makes more mistakes than prompting via API? Or would you consider the API a chatbot, too?
  - geysersam 3y ago
    I don't think there's any public interface to the LLM underlying ChatGPT, so the only ones able to test this are openAI engineers.
vorticalbox 3y ago
This is what presence_penalty and frequency_penalty are for.
nl 3y ago
Llama 2 wasn't trained on ChatGPT/GPT4. I think maybe you are thinking of the Vicuna models? https://lmsys.org/blog/2023-03-30-vicuna/ https://lmsys.org/blog/2023-03-30-vicuna/
- dr_dshiv 3y ago
  So it’s true that it would violate the OpenAI terms for Llama to be trained with ChatGPT completions, but how do we know? We don’t know the training data for Llama, we just get weights.
  - nl 3y ago
    The Llama2 paper describes the training data in some detail.

mattlutze 3y ago

We just don't have the information to make judgements, much less leaping to "they must be lying." There's a few public numbers from a handful of foundation models as to performance vs parameter count vs architecture generation. Not being able to compare in detail the architecture of the various closed models nor being more rigorous on training with progressively sized parameter sets, the conclusion at the moment is a general feeling or conjecture.

qrios 3y ago

Without questioning the statement '70% as good as GPT3.5', but wouldn't that be quantifying a quality, and a Turing test? Also: maybe these missing 30% are the hard part.