5 ms·
> The closed source players have the best researchers Is that definitely why? GPT 3.5 and GPT 4 are far larger than 70B, right? So if a 70B, local model like L
by xcdzvyn 3y ago
> The closed source players have the best researchers
Is that definitely why? GPT 3.5 and GPT 4 are far larger than 70B, right? So if a 70B, local model like LLaMA can even remotely rival them, would that not suggest that LLaMA is fundamentally a better model?
For example, would a LLaMA model with even half of GPT 4's parameters be projected to outperform it? Is that how it works?
[I'm not super familiar with LLM tech]
- hooande 3y agoThere is no clear answer. It's debatable among experts. The grandparent post seems to believe that the issue is algorithmic complexity and programming aptitude. Personally, I think that all the major LLMs are using the same basic transformer architecture with relatively minor differences in code. GPT is trained on more data with more parameters than any open source model. The size does matter, far more than the software does. In my experience with data science, the best programmers in the world can only do so much if they are operating with 1/10th the scale of data. That applies to any problem.
- oceanplexian 3y agoLLamA 2 at 70B is, let’s say pessimistically 70% as good as GPT3.5. This makes me think that OpenAI is lying about their parameter count, are vastly less efficient than LLaMA, or, the lager model sizes have diminishing returns. Either way, your point is a good one. Something doesn’t add up.
- dr_dshiv 3y agoIMO Llama2 really isn’t close to 3.5. It still has regular mode collapse (or whatever you call getting repetitive and nonsensical responses after a while), it has very poor mathematical/logical reasoning and is not good at following multi-part instructions. It just sounds like 3.5/4 because it was trained on it.
- geysersam 3y agoYou're mixing up the language model with the chat bot. The llama2 is a language model. I imagine the language model behind chatgpt is not much different (perhaps it's better, but not by many months AI research time). It likely also suffers from "mode collapse" issues etc. But 3.5 also has a lot of systems around it that detects mode collapse and applies some kind of mitigation, forcing the model to give a more reasonable output. Mathematical / logical reasoning questions are likely also detected hand passed on in some form to a separate system.
- dr_dshiv 3y agoSo this would be testable by showing that chatGPT makes more mistakes than prompting via API? Or would you consider the API a chatbot, too?
- geysersam 3y agoI don't think there's any public interface to the LLM underlying ChatGPT, so the only ones able to test this are openAI engineers.
- vorticalbox 3y agoThis is what presence_penalty and frequency_penalty are for.
- nl 3y agoLlama 2 wasn't trained on ChatGPT/GPT4. I think maybe you are thinking of the Vicuna models? https://lmsys.org/blog/2023-03-30-vicuna/ https://lmsys.org/blog/2023-03-30-vicuna/
- dr_dshiv 3y agoSo it’s true that it would violate the OpenAI terms for Llama to be trained with ChatGPT completions, but how do we know? We don’t know the training data for Llama, we just get weights.
- mattlutze 3y agoWe just don't have the information to make judgements, much less leaping to "they must be lying." There's a few public numbers from a handful of foundation models as to performance vs parameter count vs architecture generation. Not being able to compare in detail the architecture of the various closed models nor being more rigorous on training with progressively sized parameter sets, the conclusion at the moment is a general feeling or conjecture.
- qrios 3y agoWithout questioning the statement '70% as good as GPT3.5', but wouldn't that be quantifying a quality, and a Turing test? Also: maybe these missing 30% are the hard part.
- nl 3y agoIf you read the Llama2 paper it is very clear that small amounts of data (thousands of records) make vast difference at the instruction turning stage. From the Llama2 paper: > Quality Is All You Need. > Third-party SFT data is available from many different sources, but we found that many of these have insufficient diversity and quality — in particular for aligning LLMs towards dialogue-style instructions. As a result, we focused first on collecting several thousand examples of high-quality SFT data, as illustrated in Table 5. By setting aside millions of examples from third-party datasets and using fewer but higher-quality examples from our own vendor-based annotation efforts, our results notably improved. These findings are similar in spirit to Zhou et al. (2023), which also finds that a limited set of clean instruction-tuning data can be sufficient to reach a high level of quality. We found that SFT annotations in the order of tens of thousands was enough to achieve a high-quality result. We stopped annotating SFT after collecting a total of 27,540 annotations. Note that we do not include any Meta user data. It's likely OpenAI has invested in this and has good coverage in a larger range of domains. That alone probably explains a large amount of the gap.
- wkat4242 3y agoYeah I've been wondering about this too. Word on the street is that GPT4 is several times the size of GPT3.5. Yet I don't feel it's several times as good for sure. Apparently there's a diminishing returns effect on ever enlarging the model.
- kordlessagain 3y agoI believe what they discovered was that 4 is an ensemble model, comprised of (8) GPT3.5s. Things may have changed or been found to not be true on this though.