4 ms·

Did I miss when they managed to draw a pelican? Cause all the ones I've seen are wrong in some way.

by didibus 2mo ago

nomel 2mo ago
Perfection is such an absolutely wild requirement for what we're seeing happen here, with the level of understanding required, from a tech that was complete fiction 5 years ago. Maybe excitement and wonder, in tech, is just something for us old guys, that watched it all be birthed. Get off my lawn!
- crabmusket 2mo ago
  I don't think didibus is saying they expect perfection, just that there is still room for the pelican prompt to measure meaningful future improvement.
- didibus 2mo ago
  The article makes it sound like Pelicans riding a bike are solved, and we can move on to something else. Nothing to do with excitement or wonder of the tech, it just claims one thing and I feel I've not seen that be the case so the claim seems incorrect to me.
  - nomel 2mo ago
    > We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". First sentence of the post is clear that it's not solved.
    - didibus 2mo ago
      Might be my reading comprehension, but that reads to me like it's implying we're leaving the territory because it's solved and now need something harder.
      - nomel 2mo ago
        "Starting to leave" is not the same as "leave" or "left". When I start to leave my house in the morning, I have not yet gone through the door, but I'm on my way. I walk towards the door to leave the house. I'm leaving the house as I pass through the door. I look back and can see that I've left the house.
        didibus 2mo ago
        But why are you starting to leave your house? Likely because you're ready to leave no? To me that's where it was implying that models are ready to leave that benchmark behind, and starting the process is coming up with a new benchmark. Anyways, if it's just semantic misunderstanding, I have no issue. That means the fact is, models still can't draw a pelican riding a bike? With maybe the exception of Fable 5?
        nomel 2mo ago
        In this case, the preparation for "starting to leave" this benchmark would be getting a new benchmarks ready, because this one appears it will soon be obsolete.
redox99 2mo ago
I think the SOTA is fable max. It's up to you whether that's good enough or not. Almost all other models do make some egregious mistakes with the bike. https://static.simonwillison.net/static/2026/fable-max.jpg https://static.simonwillison.net/static/2026/fable-max.jpg
- didibus 2mo ago
  This is pretty close to "solved" I'd say. But then, it seems all the other models still need to catch up, and there's likely a little bit of room for improvement on Fable Max as well.

Did I miss when they managed to draw a pelican? Cause all the ones I've seen are wrong in some way.

by didibus 2mo ago

Did I miss when they managed to draw a pelican? Cause all the ones I've seen are wrong in some way.

nomel 2mo ago

Perfection is such an absolutely wild requirement for what we're seeing happen here, with the level of understanding required, from a tech that was complete fiction 5 years ago. Maybe excitement and wonder, in tech, is just something for us old guys, that watched it all be birthed. Get off my lawn!

crabmusket 2mo ago
I don't think didibus is saying they expect perfection, just that there is still room for the pelican prompt to measure meaningful future improvement.
didibus 2mo ago
The article makes it sound like Pelicans riding a bike are solved, and we can move on to something else. Nothing to do with excitement or wonder of the tech, it just claims one thing and I feel I've not seen that be the case so the claim seems incorrect to me.
- nomel 2mo ago
  > We're starting to leave the territory where you'd test an LLM by e.g. "create an svg of pelican on a bicycle". First sentence of the post is clear that it's not solved.
  - didibus 2mo ago
    Might be my reading comprehension, but that reads to me like it's implying we're leaving the territory because it's solved and now need something harder.
    - nomel 2mo ago
      "Starting to leave" is not the same as "leave" or "left". When I start to leave my house in the morning, I have not yet gone through the door, but I'm on my way. I walk towards the door to leave the house. I'm leaving the house as I pass through the door. I look back and can see that I've left the house.
      - didibus 2mo ago
        But why are you starting to leave your house? Likely because you're ready to leave no? To me that's where it was implying that models are ready to leave that benchmark behind, and starting the process is coming up with a new benchmark. Anyways, if it's just semantic misunderstanding, I have no issue. That means the fact is, models still can't draw a pelican riding a bike? With maybe the exception of Fable 5?
        nomel 2mo ago
        In this case, the preparation for "starting to leave" this benchmark would be getting a new benchmarks ready, because this one appears it will soon be obsolete.

redox99 2mo ago

I think the SOTA is fable max. It's up to you whether that's good enough or not. Almost all other models do make some egregious mistakes with the bike. https://static.simonwillison.net/static/2026/fable-max.jpg https://static.simonwillison.net/static/2026/fable-max.jpg

didibus 2mo ago
This is pretty close to "solved" I'd say. But then, it seems all the other models still need to catch up, and there's likely a little bit of room for improvement on Fable Max as well.