3 ms·

Now that it's a solved benchmark, can we get the animated version?

by swalsh 25d ago

simonw 25d ago
I didn't want to shell out for Max again, so I piped the SVG created by Max back into Fable 5.1 at its default thinking level (of high): llm logs -cx | llm -m claude-fable-5.1 -s 'animate this' Here's the result, which cost $1.37: https://tools.simonwillison.net/markdown-svg-renderer?url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F87282467acb3652e0f99c85155554a32#response https://tools.simonwillison.net/markdown-svg-renderer?url=ht... It's excellent!
- swingboy 25d ago
  No feet lol (also it's not animated for me)
  - LtdJorge 25d ago
    Same happened to me (the feet are on the top left corner, brw). But the generated MP4 works.
  - newswasboring 25d ago
    I see both feet and animation.
  - guelo 25d ago
    For me it worked on Chrome but not on Firefox
  - consumer451 25d ago
    For me: Firefox: No feet, no animation Chrome: Feet included, animated very nicely (uses significant CPU)
  - drusepth 25d ago
    What browser are you using that doesn't have feet (or animations)? Seems to all be there in Chrome.
    - swingboy 25d ago
      Firefox
  - krzyk 24d ago
    And no pedals, also no animation on Firefox.
- peri-cl 25d ago
  Well, the wheels are rotating the wrong way, but other than that! (Maybe it's a strobing artifact?)
  - cainxinth 25d ago
    I looked closely. They are going the wrong way. Still the best one I've seen by a lot.
    - IshKebab 25d ago
      Probably some temporal aliasing on your device. The SVG elements are 100% rotating clockwise.
      - scarmig 25d ago
        Still a flaw--the SVG should account for how different client devices will render it! Existential doom averted, for now.
        jdiff 24d ago
        Let me know when AI patches this bug in the wheels on all the vehicles on the highway, AND my box fan when it's next to the light with the bad PWM dimming. Then we can resume our existential doom.
      - jamiek88 25d ago
        Looks fine on safari iPadOS 26.6.1 wheels rotating forwards. I’m a skeptic but man, this actually has me revising my opinion a bit.
  - saretup 25d ago
    Are they? Looked correct to me. At high enough speeds even real ones can look rotating either way in videos depending on fps.
  - toddmerrill 24d ago
    This is an optical illusion. If I stare at it and mentally expect a direction, I can make the wheels appear to turn either way.
- rafaelmn 25d ago
  First time this benchmark impressed me - 10/10 I would believe a human spent hours making this. There are details a human would have done better (beak attaching to the head) but still impressive.
  - psygn89 24d ago
    I like how precise it got about the math behind the chain/gear/wheel/leg movement... these are the details that ain't no one really got time for.
- throw03172019 25d ago
  He looks very happy riding along the beach. I love the basket fish animation as well. Impressive!
- saretup 25d ago
  Now that it's a solved benchmark, can we get the 3d animated version?
  - consumer451 24d ago
    I can't waste Fable for play time, but I was curius, so I used plain claude.ai with Opus 5 High. It's pretty cool. I asked "create a 3d animation", and pasted the Simon’s animation code. I must admit I used a second prompt to adjust the opening camera angle, but that is all. Originally the camera was above, about 30 degrees. https://claude.ai/public/artifacts/b37a9ee2-f5bc-4ff9-ae90-a1fb8917c737 https://claude.ai/public/artifacts/b37a9ee2-f5bc-4ff9-ae90-a...
    - spiderfarmer 24d ago
      That's seriously impressive.
      - consumer451 24d ago
        I have to admit, these are not directly comparable to Simon's tests though, as I was using claude.ai, and he is using raw API correct? I suppose I could use Claude Code, and disable system prompts. That should be the same, right? If I have spare tokens at the end of the week, I will try real tests.
    - consumer451 24d ago
      I couldn't help myself. Here is the game, one-shot on claude.ai, Opus 5 Extra. There could certainly be play-ability improvements made with 1 or 2 prompts, but neato for one-shot. "Create a fun and cool game from this" - and pasted the code from above. https://claude.ai/public/artifacts/e723244b-fa52-4e63-9819-4f84e5ac0c4f https://claude.ai/public/artifacts/e723244b-fa52-4e63-9819-4... If anyone has the spare Fable tokens, would love to see the difference in the 3D animation and maybe even game.
      - stevage 24d ago
- Wowfunhappy 24d ago
  I find it interesting that you put the instruction ("animate this") in the system prompt, while the prompt just had the existing SVG and no actual instructions on what to do with it. Anthropic's consumer-facing tools generally don't let you do this, unless maybe you're particularly clever with Claude Code command line flags (you can start Claude Code with a custom system prompt). Do you think this had a substantial effect on the output?
  - bakies 24d ago
    You can do it with the Claude cli. You can pipe into it like that and set system prompt
    - Wowfunhappy 24d ago
      Sorry, yes, that's what I meant by being "clever with Claude Code command line flags", I guess it's not really that clever. But in general if you just launch Claude Code normally and use it via the interactive CLI, you can't change the system prompt, so most users probably never do it.
- stevage 24d ago
  Not sure if it's a rendering artefact, but the wheels are spinning backwards for me? Also, 10 tooth sprocket with 22 tooth chainring? Not impossible, but so small!
  - yreg 24d ago
    How can you tell which way are they spinning? The constant fps acts as a shutter.
- chamomeal 24d ago
  Dang the little fish jiggling around the basket is a nice touch
leumon 25d ago
how about trying to draw an airbus a320 in 3d space using only one brush tool that can be moved to specific x,y,z coordinates (and its color, size & hardness can be changed). i think fable 5.1 did quite a good job (reasoning high, cost $0,261): https://files.catbox.moe/umx102.png https://files.catbox.moe/umx102.png for comparision, this is fable 5: https://files.catbox.moe/ihl4m1.png https://files.catbox.moe/ihl4m1.png
pampas 24d ago
I made a 3D browser game a few months ago where you play as a pelican riding a bike via a bluetooth trainer.
alexandra_au 24d ago
Not to be "that" person but it's not solved. The feet are reversed and isn't accurate bird anatomy. In real life, what people think of as bird's feet is actually their toes, and their "knee" is actually their tarsal (ankle bone), and their actual knee is almost hidden in their feathers.
- frumiousirc 24d ago
  > not solved I agree. Other reasoning traces simonw quoted in his blog post showed that the model made changes to consider realistic fork rake. I think this may also be the first case where the chain went inside the seat stays. The overall bike geometry is still comical but these bits show improvement in this model over prior ones. On the other hand, given the absurdity of the original prompt, I should not necessarily expect realistic bike geometry.
- stevage 24d ago
  The problem is it's kind of a question without a well defined answer. If it was just drawing a picture of a pelican, we can compare against real pelicans for accuracy. But what exactly is a pelican riding a bike supposed to look like? It's impossible, so liberties have to be taken somewhere.

Now that it's a solved benchmark, can we get the animated version?

by swalsh 25d ago

Now that it's a solved benchmark, can we get the animated version?

I didn't want to shell out for Max again, so I piped the SVG created by Max back into Fable 5.1 at its default thinking level (of high): llm logs -cx | llm -m claude-fable-5.1 -s 'animate this' Here's the result, which cost $1.37: https://tools.simonwillison.net/markdown-svg-renderer?url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F87282467acb3652e0f99c85155554a32#response https://tools.simonwillison.net/markdown-svg-renderer?url=ht... It's excellent!

swingboy 25d ago
No feet lol (also it's not animated for me)
- LtdJorge 25d ago
  Same happened to me (the feet are on the top left corner, brw). But the generated MP4 works.
- newswasboring 25d ago
  I see both feet and animation.
- guelo 25d ago
  For me it worked on Chrome but not on Firefox
- consumer451 25d ago
  For me: Firefox: No feet, no animation Chrome: Feet included, animated very nicely (uses significant CPU)
- drusepth 25d ago
  What browser are you using that doesn't have feet (or animations)? Seems to all be there in Chrome.
  - swingboy 25d ago
    Firefox
- krzyk 24d ago
  And no pedals, also no animation on Firefox.
peri-cl 25d ago
Well, the wheels are rotating the wrong way, but other than that! (Maybe it's a strobing artifact?)
- cainxinth 25d ago
  I looked closely. They are going the wrong way. Still the best one I've seen by a lot.
  - IshKebab 25d ago
    Probably some temporal aliasing on your device. The SVG elements are 100% rotating clockwise.
    - scarmig 25d ago
      Still a flaw--the SVG should account for how different client devices will render it! Existential doom averted, for now.
      - jdiff 24d ago
        Let me know when AI patches this bug in the wheels on all the vehicles on the highway, AND my box fan when it's next to the light with the bad PWM dimming. Then we can resume our existential doom.
    - jamiek88 25d ago
      Looks fine on safari iPadOS 26.6.1 wheels rotating forwards. I’m a skeptic but man, this actually has me revising my opinion a bit.
- saretup 25d ago
  Are they? Looked correct to me. At high enough speeds even real ones can look rotating either way in videos depending on fps.
- toddmerrill 24d ago
  This is an optical illusion. If I stare at it and mentally expect a direction, I can make the wheels appear to turn either way.
rafaelmn 25d ago
First time this benchmark impressed me - 10/10 I would believe a human spent hours making this. There are details a human would have done better (beak attaching to the head) but still impressive.
- psygn89 24d ago
  I like how precise it got about the math behind the chain/gear/wheel/leg movement... these are the details that ain't no one really got time for.
throw03172019 25d ago
He looks very happy riding along the beach. I love the basket fish animation as well. Impressive!
saretup 25d ago
Now that it's a solved benchmark, can we get the 3d animated version?
- consumer451 24d ago
  I can't waste Fable for play time, but I was curius, so I used plain claude.ai with Opus 5 High. It's pretty cool. I asked "create a 3d animation", and pasted the Simon’s animation code. I must admit I used a second prompt to adjust the opening camera angle, but that is all. Originally the camera was above, about 30 degrees. https://claude.ai/public/artifacts/b37a9ee2-f5bc-4ff9-ae90-a1fb8917c737 https://claude.ai/public/artifacts/b37a9ee2-f5bc-4ff9-ae90-a...
  - spiderfarmer 24d ago
    That's seriously impressive.
    - consumer451 24d ago
      I have to admit, these are not directly comparable to Simon's tests though, as I was using claude.ai, and he is using raw API correct? I suppose I could use Claude Code, and disable system prompts. That should be the same, right? If I have spare tokens at the end of the week, I will try real tests.
  - consumer451 24d ago
    I couldn't help myself. Here is the game, one-shot on claude.ai, Opus 5 Extra. There could certainly be play-ability improvements made with 1 or 2 prompts, but neato for one-shot. "Create a fun and cool game from this" - and pasted the code from above. https://claude.ai/public/artifacts/e723244b-fa52-4e63-9819-4f84e5ac0c4f https://claude.ai/public/artifacts/e723244b-fa52-4e63-9819-4... If anyone has the spare Fable tokens, would love to see the difference in the 3D animation and maybe even game.
    - stevage 24d ago
Wowfunhappy 24d ago
I find it interesting that you put the instruction ("animate this") in the system prompt, while the prompt just had the existing SVG and no actual instructions on what to do with it. Anthropic's consumer-facing tools generally don't let you do this, unless maybe you're particularly clever with Claude Code command line flags (you can start Claude Code with a custom system prompt). Do you think this had a substantial effect on the output?
- bakies 24d ago
  You can do it with the Claude cli. You can pipe into it like that and set system prompt
  - Wowfunhappy 24d ago
    Sorry, yes, that's what I meant by being "clever with Claude Code command line flags", I guess it's not really that clever. But in general if you just launch Claude Code normally and use it via the interactive CLI, you can't change the system prompt, so most users probably never do it.
stevage 24d ago
Not sure if it's a rendering artefact, but the wheels are spinning backwards for me? Also, 10 tooth sprocket with 22 tooth chainring? Not impossible, but so small!
- yreg 24d ago
  How can you tell which way are they spinning? The constant fps acts as a shutter.
chamomeal 24d ago
Dang the little fish jiggling around the basket is a nice touch

leumon 25d ago

how about trying to draw an airbus a320 in 3d space using only one brush tool that can be moved to specific x,y,z coordinates (and its color, size & hardness can be changed). i think fable 5.1 did quite a good job (reasoning high, cost $0,261): https://files.catbox.moe/umx102.png https://files.catbox.moe/umx102.png for comparision, this is fable 5: https://files.catbox.moe/ihl4m1.png https://files.catbox.moe/ihl4m1.png

pampas 24d ago

I made a 3D browser game a few months ago where you play as a pelican riding a bike via a bluetooth trainer.

alexandra_au 24d ago

Not to be "that" person but it's not solved. The feet are reversed and isn't accurate bird anatomy. In real life, what people think of as bird's feet is actually their toes, and their "knee" is actually their tarsal (ankle bone), and their actual knee is almost hidden in their feathers.

frumiousirc 24d ago
> not solved I agree. Other reasoning traces simonw quoted in his blog post showed that the model made changes to consider realistic fork rake. I think this may also be the first case where the chain went inside the seat stays. The overall bike geometry is still comical but these bits show improvement in this model over prior ones. On the other hand, given the absurdity of the original prompt, I should not necessarily expect realistic bike geometry.
stevage 24d ago
The problem is it's kind of a question without a well defined answer. If it was just drawing a picture of a pelican, we can compare against real pelicans for accuracy. But what exactly is a pelican riding a bike supposed to look like? It's impossible, so liberties have to be taken somewhere.