3 ms·

It’s based on users blind rating the output of two LLMs given the same prompt.

• kozikow 3y ago
  Where GPT really improved for me subjectively is long context - at least since launch. Such ranking can't compare it. Also I felt I keep getting more personalized results, i.e. models are somehow biased towards user. I heard they plan it, I don't know it's launched, but I feel it. And there's also fine-tuning in the other direction - my brain got used to ways of interacting with GPT. Same as with Google, I just somehow subconsciously know how to write prompts that get me what I want.
  • nemonemo 3y ago
    Typically benchmarks have limited aspects they are measuring. I can imagine another suite of benchmarks with longer contexts, but in that case, it might be more difficult to do it in a blind comparison form. At the least, it would be quite costly to run such benchmarks.