3 ms·

It's easy to miss: select English in the dropdown. The scores are quite different in Overall and in English for LMSYS. As I've stated in other comments, yeah..

It's easy to miss: select English in the dropdown. The scores are quite different in Overall and in English for LMSYS. As I've stated in other comments, yeah... Agreed, I'm stretching it a bit. It's just that any indication of a 3.8B model being in the vicinity of GPT-4 is huge. I'm sure that when things are properly measured by third-parties it will show a more sober picture. But still, with good fine-tunes, we'll probably get close. It's a very significant demonstration of what could be possible soon.

saretup 3y ago
Firstly, English is a highly subjective category. Secondly, Llama 3 usually adds first sentences like ‘What a unique question!’ or ‘What an insightful thought’, which might make people like it more than the competition because of the pandering. While Llama 3 is singular in terms of size to quality ratio, calling the 8B model close to GPT4 would be an overstretch.
- YetAnotherNick 3y ago
  Yes, I don't know how people don't realize how much cheap tricks works in Chatbot Arena. A single base model produces 100s of ELO difference depending on the way it is tuned. And on most cases, instruction tuning heavily slightly even decreases reasoning ability on standard benchmark. You can see base model scores better in MMLU/ARC most of the times in huggingface leaderboard. Even GPT-4-1106 seems to only sounds better than GPT-4-0613 and works for wider range of prompt. But in a well defined prompt and follow up questions I don't think there is an improvement in reasoning.
  - imtringued 3y ago
    When I tried Phi2 it was just bad. I don't know where you got this fantasy from that people accept obviously wrong answers, because of "pandering".
    - YetAnotherNick 3y ago
      Obviously correct answer matters more but ~100-200 elo points could be gained just for better writing. Answer would be range of 500 elo in comparison.
      - rgbrgb 3y ago
        > just for better writing in my use cases, better writing makes a better answer

saretup 3y ago

Firstly, English is a highly subjective category. Secondly, Llama 3 usually adds first sentences like ‘What a unique question!’ or ‘What an insightful thought’, which might make people like it more than the competition because of the pandering. While Llama 3 is singular in terms of size to quality ratio, calling the 8B model close to GPT4 would be an overstretch.

YetAnotherNick 3y ago
Yes, I don't know how people don't realize how much cheap tricks works in Chatbot Arena. A single base model produces 100s of ELO difference depending on the way it is tuned. And on most cases, instruction tuning heavily slightly even decreases reasoning ability on standard benchmark. You can see base model scores better in MMLU/ARC most of the times in huggingface leaderboard. Even GPT-4-1106 seems to only sounds better than GPT-4-0613 and works for wider range of prompt. But in a well defined prompt and follow up questions I don't think there is an improvement in reasoning.
- imtringued 3y ago
  When I tried Phi2 it was just bad. I don't know where you got this fantasy from that people accept obviously wrong answers, because of "pandering".
  - YetAnotherNick 3y ago
    Obviously correct answer matters more but ~100-200 elo points could be gained just for better writing. Answer would be range of 500 elo in comparison.
    - rgbrgb 3y ago
      > just for better writing in my use cases, better writing makes a better answer