4 ms·

In other words: "our 70B finetune is as good as a 8x200B model" Yeah, right.

google234123 3y ago
I'm not sure GPT 4 is still 8x200B
minimaxir 3y ago
The one thing we've learnt from the past few months of LLM optimization is that model size is no longer the most important thing in determining LLM quality. A better training regimen and better architecture optimizations have allowed smaller models to push above their weight. The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- ein0p 3y ago
  It kinda is, if you want not just performance on synthetic benchmarks but a good coverage of the long tail. This is where GPT4 excels, and also why I pay for it. Transformers are basically fancy associative memories. A smaller model, much like a smaller search index, will not be able to contain as much nuanced information for some hard, immutable, information theoretic reasons.
- behnamoh 3y ago
  > The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... I follow your posts and comments here so I'm surprised you say that. The leaderboard at this point is pretty pointless. Lots of ways to "cheat" and get higher ranking there. I do agree that smaller models have made significant progress, but somethings you can't just solve without adding #parameters and FLOPs. Not to mention, ctx_window is an important factor in code quality, but most OSS models (including llama 2) have pretty limited ctx, despite methods like grp and yarn.
  - minimaxir 3y ago
    It's more a comment on the capabilities of smaller models, the quality of output outside of benchmarks is always subjective and you'd need something like Chatbot Arena (https://chat.lmsys.org/ https://chat.lmsys.org/) to evaluate it more quantitatively. Even after filtering out the common cheat techniques like merges, there are still 7B and 13B near the top, but yes it's still possible to train models on the evaluation datasets without decontamination. If you look at the Chatbot Arena leaderboards there are still decently-high ELOs for 7B models.
    - visarga 3y ago
      I evaluated many Mistrals for an information extraction task and the merged models were much better than direct fine-tunes. About 5% better.
- brucethemoose2 3y ago
  I agree... Except for the leaderboard. Its all but useless, not just because of the data contamination/cheating but because the benchmarks themselves are flawed. They are full of ambiguity/errors, and they dont even use instruct formatting.
- ignoramous 3y ago
  I've found that GPT4 (via GitHub Copilot) and Gemini models are better at code tasks like reviewing for logical and functional errors, reasoning about structure and test/edge cases, and refactoring. Gemini is capable of devouring some very large files I've thrown at it. Phind at times is hampered by whatever it is they're doing in addition (RAG?). It is still phenomenal, though. I regularly find myself using Phind to grok assembly code or learn Typescript.
  - sroussey 3y ago
    How do you know that copilot is using gpt4? I pay for it and for chatGPT and I find copilot much worse.
    - ignoramous 3y ago
      Looks like Copilot may use GPT4 or GPT3.5 depending on as of yet unpublished criteria: https://github.com/microsoft/vscode-copilot-release/issues/6#issuecomment-1791319670 https://github.com/microsoft/vscode-copilot-release/issues/6... For code review, I tend to engage Copilot Chat which probably uses GPT4 more often? https://github.com/orgs/community/discussions/58059#discussioncomment-8497038 https://github.com/orgs/community/discussions/58059#discussi...
- SirMaster 3y ago
  But what if you apply the same level of optimization, same training regimen to the larger models?
rushingcreek 3y ago
Phind-70B is a specialist model, unlike GPT-4. It optimizes for a different function than GPT-4 and therefore needs fewer parameters to learn it. It's also true that specialist models still need to be sufficiently large to be able to reason well, but we've observed diminishing returns as models get larger.
CuriouslyC 3y ago
I mean, it could be as good or better at a lot of reasoning related tasks and just have less baked in general knowledge, in which case it'd make an amazing RAG model if the context length is reasonable.

I'm not sure GPT 4 is still 8x200B

The one thing we've learnt from the past few months of LLM optimization is that model size is no longer the most important thing in determining LLM quality. A better training regimen and better architecture optimizations have allowed smaller models to push above their weight. The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

ein0p 3y ago
It kinda is, if you want not just performance on synthetic benchmarks but a good coverage of the long tail. This is where GPT4 excels, and also why I pay for it. Transformers are basically fancy associative memories. A smaller model, much like a smaller search index, will not be able to contain as much nuanced information for some hard, immutable, information theoretic reasons.
behnamoh 3y ago
> The leaderboard has many open 7B and 13B models that are comparable with 72B models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... I follow your posts and comments here so I'm surprised you say that. The leaderboard at this point is pretty pointless. Lots of ways to "cheat" and get higher ranking there. I do agree that smaller models have made significant progress, but somethings you can't just solve without adding #parameters and FLOPs. Not to mention, ctx_window is an important factor in code quality, but most OSS models (including llama 2) have pretty limited ctx, despite methods like grp and yarn.
- minimaxir 3y ago
  It's more a comment on the capabilities of smaller models, the quality of output outside of benchmarks is always subjective and you'd need something like Chatbot Arena (https://chat.lmsys.org/ https://chat.lmsys.org/) to evaluate it more quantitatively. Even after filtering out the common cheat techniques like merges, there are still 7B and 13B near the top, but yes it's still possible to train models on the evaluation datasets without decontamination. If you look at the Chatbot Arena leaderboards there are still decently-high ELOs for 7B models.
  - visarga 3y ago
    I evaluated many Mistrals for an information extraction task and the merged models were much better than direct fine-tunes. About 5% better.
brucethemoose2 3y ago
I agree... Except for the leaderboard. Its all but useless, not just because of the data contamination/cheating but because the benchmarks themselves are flawed. They are full of ambiguity/errors, and they dont even use instruct formatting.
ignoramous 3y ago
I've found that GPT4 (via GitHub Copilot) and Gemini models are better at code tasks like reviewing for logical and functional errors, reasoning about structure and test/edge cases, and refactoring. Gemini is capable of devouring some very large files I've thrown at it. Phind at times is hampered by whatever it is they're doing in addition (RAG?). It is still phenomenal, though. I regularly find myself using Phind to grok assembly code or learn Typescript.
- sroussey 3y ago
  How do you know that copilot is using gpt4? I pay for it and for chatGPT and I find copilot much worse.
  - ignoramous 3y ago
    Looks like Copilot may use GPT4 or GPT3.5 depending on as of yet unpublished criteria: https://github.com/microsoft/vscode-copilot-release/issues/6#issuecomment-1791319670 https://github.com/microsoft/vscode-copilot-release/issues/6... For code review, I tend to engage Copilot Chat which probably uses GPT4 more often? https://github.com/orgs/community/discussions/58059#discussioncomment-8497038 https://github.com/orgs/community/discussions/58059#discussi...
SirMaster 3y ago
But what if you apply the same level of optimization, same training regimen to the larger models?

rushingcreek 3y ago

Phind-70B is a specialist model, unlike GPT-4. It optimizes for a different function than GPT-4 and therefore needs fewer parameters to learn it. It's also true that specialist models still need to be sufficiently large to be able to reason well, but we've observed diminishing returns as models get larger.

CuriouslyC 3y ago

I mean, it could be as good or better at a lot of reasoning related tasks and just have less baked in general knowledge, in which case it'd make an amazing RAG model if the context length is reasonable.