4 ms·

Doesn't do so great on the leaderboards: https://tatsu-lab.github.io/alpaca_eval/ https://tatsu-lab.github.io/alpaca_eval/

binarymax 3y ago
Depends on the benchmark. Does well on other metrics when compared to open models. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- cosmojg 3y ago
  Given that HellaSwag performance seems to correlate with reasoning ability more than other benchmarks, Falcon certainly look promising! Hopefully this is a clean result and not the product of dataset contamination.
  - avereveard 3y ago
    I've given it a try, to having a chat is good, to follow langchain prompts it's not. I guess it depends on the type of work you want to extract from it.
- redox99 3y ago
  That leaderboard is incorrect https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/63 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
  - lhl 3y ago
    Actually it's HF's leaderboard that's bugged. Falcon is only on top since their MMLU scores are bugged across all LLaMA models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/63 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
mikeravkine 3y ago
Its a pretty terrible model, I wouldn't use this for anything at all. Vicuna 1.1 outperforms it in all my tests.
- reasonabl_human 3y ago
  What tests are you running? Anything outside of the alpaca eval test suite?
londons_explore 3y ago
I really like the fact that the leaderboards are almost identical when using claude or GPT-4 as evaluators. If a less powerful model can be a good decider of the better answer between two more powerful models, it opens up a lot of research opportunities into perhaps using these evaluations as part of an automated reinforcement learning process.
brucethemoose2 3y ago
That leaderboard does not line up with my personal experience with those models at all...

binarymax 3y ago

Depends on the benchmark. Does well on other metrics when compared to open models. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

cosmojg 3y ago
Given that HellaSwag performance seems to correlate with reasoning ability more than other benchmarks, Falcon certainly look promising! Hopefully this is a clean result and not the product of dataset contamination.
- avereveard 3y ago
  I've given it a try, to having a chat is good, to follow langchain prompts it's not. I guess it depends on the type of work you want to extract from it.
redox99 3y ago
That leaderboard is incorrect https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/63 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- lhl 3y ago
  Actually it's HF's leaderboard that's bugged. Falcon is only on top since their MMLU scores are bugged across all LLaMA models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/63 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

mikeravkine 3y ago

Its a pretty terrible model, I wouldn't use this for anything at all. Vicuna 1.1 outperforms it in all my tests.

reasonabl_human 3y ago
What tests are you running? Anything outside of the alpaca eval test suite?

londons_explore 3y ago

I really like the fact that the leaderboards are almost identical when using claude or GPT-4 as evaluators. If a less powerful model can be a good decider of the better answer between two more powerful models, it opens up a lot of research opportunities into perhaps using these evaluations as part of an automated reinforcement learning process.

brucethemoose2 3y ago

That leaderboard does not line up with my personal experience with those models at all...