3 ms·

I set it up as a joke, to make fun of all of the other benchmarks. To my surprise it ended up being a surprisingly good measure of the quality of the model for

by simonw 7mo ago

I set it up as a joke, to make fun of all of the other benchmarks. To my surprise it ended up being a surprisingly good measure of the quality of the model for other tasks (up to a certain point at least), though I've never seen a convincing argument as to why. I gave a talk about it last year: https://simonwillison.net/2025/Jun/6/six-months-in-llms/ https://simonwillison.net/2025/Jun/6/six-months-in-llms/ It should not be treated as a serious benchmark.

jimbokun 7mo ago
What it has going for it is human interpretability. Anyone can look and decide if it’s a good picture or not. But the numeric benchmarks don’t tell you much if you aren’t already familiar with that benchmark and how it’s constructed.
billywhizz 7mo ago
how can you say "it ended up being a surprisingly good measure of the quality of the model for other tasks" and also "It should not be treated as a serious benchmark" in the same comment? if it is indeed a good measure of the quality of the model (hint: it's not) then, logically, it should be taken seriously. this is, sadly, a great example of the kind of doublethink the "AI" hypesters (yes - whether you like it or not simon - that is what you are now) are all too capable of.
- simonw 7mo ago
  I genuinely don't see how those two statements conflict with each other. Despite not being a serious benchmark (how could it be serious? It's a pelican riding a bicycle!) it still turned out to have some value. You can see that just by scrolling through the archives and watching it improve as the models improved. If your definition of doublethink is "holding two conflicting ideas in your head at once" then I would say doublethink is a necessary skill for navigating the weird AI era we find ourselves inhabiting.
  - billywhizz 7mo ago
    "some value" is not the same as "a surprisingly good measure of the quality of the model for other tasks". doublethink does not mean holding two conflicting ideas in your head at once. it means holding two logically inconsistent positions/beliefs at the same time.

by simonw 7mo ago

jimbokun 7mo ago

What it has going for it is human interpretability. Anyone can look and decide if it’s a good picture or not. But the numeric benchmarks don’t tell you much if you aren’t already familiar with that benchmark and how it’s constructed.

billywhizz 7mo ago

how can you say "it ended up being a surprisingly good measure of the quality of the model for other tasks" and also "It should not be treated as a serious benchmark" in the same comment? if it is indeed a good measure of the quality of the model (hint: it's not) then, logically, it should be taken seriously. this is, sadly, a great example of the kind of doublethink the "AI" hypesters (yes - whether you like it or not simon - that is what you are now) are all too capable of.

simonw 7mo ago
I genuinely don't see how those two statements conflict with each other. Despite not being a serious benchmark (how could it be serious? It's a pelican riding a bicycle!) it still turned out to have some value. You can see that just by scrolling through the archives and watching it improve as the models improved. If your definition of doublethink is "holding two conflicting ideas in your head at once" then I would say doublethink is a necessary skill for navigating the weird AI era we find ourselves inhabiting.
- billywhizz 7mo ago
  "some value" is not the same as "a surprisingly good measure of the quality of the model for other tasks". doublethink does not mean holding two conflicting ideas in your head at once. it means holding two logically inconsistent positions/beliefs at the same time.