4 ms·

The Github repository, gives a better introduction/howto: https://github.com/01-ai/yi https://github.com/01-ai/yi > Yi-34B-Chat model landed in second place (

The Github repository, gives a better introduction/howto: https://github.com/01-ai/yi https://github.com/01-ai/yi > Yi-34B-Chat model landed in second place (following GPT-4 Turbo), outperforming other LLMs (such as GPT-4, Mixtral, Claude) on the AlpacaEval Leaderboard (based on data available up to January 2024). > Yi-34B model ranked first among all existing open-source models (such as Falcon-180B, Llama-70B, Claude) in both English and Chinese on various benchmarks, including Hugging Face Open LLM Leaderboard (pre-trained) and C-Eval (based on data available up to November 2023).

WhitneyLand 3y ago
It’s been ~1 year since gpt-4 was released. It’s hard to guess how long before any flavor of an “open” model will consensus match what was released in 2023 let alone potentially exceed it. A big part of the race seems like it will depend on how high gpt-5 can raise the bar. If it’s only incrementally things may converge quickly.
- c0n5pir4cy 3y ago
  The Yi models were actually released back in early November 2023. So there isn't as big a gap in time as it seems. I'm not sure why there is such a big gap between the release of the models and the publication of the paper. EDIT: Okay, this appears to be a new set of models with the same name, based on the same models from November but now with multimodal capabilities.
oersted 3y ago
Looking at the leaderboard shows a clearer picture: https://tatsu-lab.github.io/alpaca_eval/ https://tatsu-lab.github.io/alpaca_eval/ - GPT-4-Turbo: 50.00% - Snorkel (current 2nd, Mistral 7B fine-tune): 34.86% - Yi 34B Chat (current 6th): 29.66% - GPT-4: 23.58% Thoughts: - Just saying that it came 2nd is quite misleading, the difference in score is significant. - Not sure what's up with this benchmark, I've never seen GPT-4-Turbo vs GPT-4 performing so differently. - The Snorkel model is impressive with just 7B parameters. The Yi authors claim that their success is based on good training data cleaning. This seems to be key at least for this benchmark. Snorkel has also always been all about that, using programmatic methods to generate lots of quality training data.
- doctorpangloss 3y ago
  > Not sure what's up with this benchmark, I've never seen GPT-4-Turbo vs GPT-4 performing so differently. The benchmark is bad.
- lossolo 3y ago
  The only reliable benchmark is found at https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Model creators train models including open-source benchmarks in the data, either intentionally to achieve better scores or inadvertently through leaks from various sources.
nickpsecurity 3y ago
Anytime you see that, we should assume the newer models might have been trained on either the benchmarks themselves or something similar to them. If I was an evaluator, I’d keep a secret pile of tests that I know aren’t in any LLM’s, do the evaluations privately, and not publish scores either. Just rank plus how far apart they are. The best tests of these models are people who want to use AI to solve real problems attempting to do that with various models. If they work, report that they worked. Also, publish the work and result pairs permissively when possible to evaluate that and use it for fine-tuning, too.

WhitneyLand 3y ago

It’s been ~1 year since gpt-4 was released. It’s hard to guess how long before any flavor of an “open” model will consensus match what was released in 2023 let alone potentially exceed it. A big part of the race seems like it will depend on how high gpt-5 can raise the bar. If it’s only incrementally things may converge quickly.

c0n5pir4cy 3y ago
The Yi models were actually released back in early November 2023. So there isn't as big a gap in time as it seems. I'm not sure why there is such a big gap between the release of the models and the publication of the paper. EDIT: Okay, this appears to be a new set of models with the same name, based on the same models from November but now with multimodal capabilities.

oersted 3y ago

Looking at the leaderboard shows a clearer picture: https://tatsu-lab.github.io/alpaca_eval/ https://tatsu-lab.github.io/alpaca_eval/ - GPT-4-Turbo: 50.00% - Snorkel (current 2nd, Mistral 7B fine-tune): 34.86% - Yi 34B Chat (current 6th): 29.66% - GPT-4: 23.58% Thoughts: - Just saying that it came 2nd is quite misleading, the difference in score is significant. - Not sure what's up with this benchmark, I've never seen GPT-4-Turbo vs GPT-4 performing so differently. - The Snorkel model is impressive with just 7B parameters. The Yi authors claim that their success is based on good training data cleaning. This seems to be key at least for this benchmark. Snorkel has also always been all about that, using programmatic methods to generate lots of quality training data.

doctorpangloss 3y ago
> Not sure what's up with this benchmark, I've never seen GPT-4-Turbo vs GPT-4 performing so differently. The benchmark is bad.
lossolo 3y ago
The only reliable benchmark is found at https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Model creators train models including open-source benchmarks in the data, either intentionally to achieve better scores or inadvertently through leaks from various sources.

nickpsecurity 3y ago

Anytime you see that, we should assume the newer models might have been trained on either the benchmarks themselves or something similar to them. If I was an evaluator, I’d keep a secret pile of tests that I know aren’t in any LLM’s, do the evaluations privately, and not publish scores either. Just rank plus how far apart they are. The best tests of these models are people who want to use AI to solve real problems attempting to do that with various models. If they work, report that they worked. Also, publish the work and result pairs permissively when possible to evaluate that and use it for fine-tuning, too.