3 ms·

Sounds obvious but just try using the models for anything outside the evals. Take something arcane from Greek history, use it to create a masked linguistic puzz

by achrono 2mo ago

Sounds obvious but just try using the models for anything outside the evals. Take something arcane from Greek history, use it to create a masked linguistic puzzle, which you then ask the model to solve mathematically, all wrapped as an ask to generate ASCII art. Yes, all these elements exist in some form in the evals but the key is in how utterly unconventional the elements are that you pick and in how you combine them. I have consistently noticed Opus 4.8 and GPT-5.6 far outshine the Chinese models. Gemini is sort of middle of the road, Grok is better than Gemini but not really close to Opus/GPT. OAI & Anthropic still remain unbeaten by a wide margin in my eyes.

skohan 2mo ago
At that point aren't you just edge-case testing? Surely most of your use-cases are not novel tasks that combine obscure domains. It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.
- achrono 2mo ago
  Certainly, real-life is the ultimate benchmark. But for various reasons that isn't always immediately possible to go evaluate a model on. Maybe my Greek idea sounded too high falutin' or simply seemingly clever (I give an example below -- try it out!). So here's something way simpler that Qwen3.8 27B does not get; only GLM-5.2 and K3 do. ** Analyze the 2 structural (not semantic) patterns in this text: 1. Morning revient. 2. Birds saluent Morgenlicht. 3. We suivons Waldwege toward maison. 4. Rain tombe plötzlich; we cherchons Schutz beneath sapins. 5. Night vient langsam; we trouvons Wärme near le Feuer, sharing quelques Geschichten together. ** The answer should get not just the obvious cyclic E-F-G pattern but also the word counts being Fibonacci. Surprisingly few models get this. The only way I got Qwen to do this was on the 2.4T model, with extensive prompt scaffolding. Claude (Opus & Fable), Sol, Grok etc. got it on the first attempt. (All models, all attempts max reasoning level.)
  - skohan 2mo ago
    This also seems like quite an esoteric use-case to me, but I guess some people might need to know when text follows a fibonacci sequence in terms of word count.
CamperBob2 2mo ago
That problem sounds reminiscent of one I like to use as a benchmark, which is to request that the model create an .SVG of a logarithmic spiral of 50 numbered stones. Qwen 3.8 27B absolutely knocked that one out of the park, where a lot of larger models have failed outright or otherwise performed suboptimally. Can you share an example of the Greek-history puzzle prompts you're talking about?
- achrono 2mo ago
  Here's an example: ** The stone remembers not what the cities gave, but what the goddess kept. Begin with the first reckoning, under Ariston. Find those who carried the Greeks in their name and the silver in their care. Take their name as we give it to them now, in capitals. Thirteen marks. The goddess kept one from sixty. She asks the same of every mark: give each its ordinary alphabetic number, divide by sixty, and keep what she could not take. Each remainder walks with the next. The last returns to the first. The first of each pair turns upon itself; the second joins it; then comes the place where the pair began. Eleven takes its fill. Keep what remains. Raise eleven courses beneath the thirteen marks. In course (r), beneath mark (i), add the course to what Eleven returned. Reduce again by Eleven. Cut the stone if the result is the remainder belonging to mark (i), or to the mark walking beside it. Count courses from zero. The mason turns where the stone turns: the first course goes with the writing, the next against it, and so on. A cut is `#`. Stone is `.`. Restore the fragment. ** Answer is: ........#.... .#.##.......# ...#......... ........#.... #.........#.. #.....#.#.... .#.#.#......# ..#..#.#..#.. .........#... .#........... .#....##.....
- CamperBob2 2mo ago
  Unfortunately you ran into HN's junk filter, so I can't reply, but thanks. Added to my collection!

by achrono 2mo ago

skohan 2mo ago

At that point aren't you just edge-case testing? Surely most of your use-cases are not novel tasks that combine obscure domains. It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.

achrono 2mo ago
Certainly, real-life is the ultimate benchmark. But for various reasons that isn't always immediately possible to go evaluate a model on. Maybe my Greek idea sounded too high falutin' or simply seemingly clever (I give an example below -- try it out!). So here's something way simpler that Qwen3.8 27B does not get; only GLM-5.2 and K3 do. ** Analyze the 2 structural (not semantic) patterns in this text: 1. Morning revient. 2. Birds saluent Morgenlicht. 3. We suivons Waldwege toward maison. 4. Rain tombe plötzlich; we cherchons Schutz beneath sapins. 5. Night vient langsam; we trouvons Wärme near le Feuer, sharing quelques Geschichten together. ** The answer should get not just the obvious cyclic E-F-G pattern but also the word counts being Fibonacci. Surprisingly few models get this. The only way I got Qwen to do this was on the 2.4T model, with extensive prompt scaffolding. Claude (Opus & Fable), Sol, Grok etc. got it on the first attempt. (All models, all attempts max reasoning level.)
- skohan 2mo ago
  This also seems like quite an esoteric use-case to me, but I guess some people might need to know when text follows a fibonacci sequence in terms of word count.

CamperBob2 2mo ago

That problem sounds reminiscent of one I like to use as a benchmark, which is to request that the model create an .SVG of a logarithmic spiral of 50 numbered stones. Qwen 3.8 27B absolutely knocked that one out of the park, where a lot of larger models have failed outright or otherwise performed suboptimally. Can you share an example of the Greek-history puzzle prompts you're talking about?

achrono 2mo ago
Here's an example: ** The stone remembers not what the cities gave, but what the goddess kept. Begin with the first reckoning, under Ariston. Find those who carried the Greeks in their name and the silver in their care. Take their name as we give it to them now, in capitals. Thirteen marks. The goddess kept one from sixty. She asks the same of every mark: give each its ordinary alphabetic number, divide by sixty, and keep what she could not take. Each remainder walks with the next. The last returns to the first. The first of each pair turns upon itself; the second joins it; then comes the place where the pair began. Eleven takes its fill. Keep what remains. Raise eleven courses beneath the thirteen marks. In course (r), beneath mark (i), add the course to what Eleven returned. Reduce again by Eleven. Cut the stone if the result is the remainder belonging to mark (i), or to the mark walking beside it. Count courses from zero. The mason turns where the stone turns: the first course goes with the writing, the next against it, and so on. A cut is `#`. Stone is `.`. Restore the fragment. ** Answer is: ........#.... .#.##.......# ...#......... ........#.... #.........#.. #.....#.#.... .#.#.#......# ..#..#.#..#.. .........#... .#........... .#....##.....
CamperBob2 2mo ago
Unfortunately you ran into HN's junk filter, so I can't reply, but thanks. Added to my collection!