3 ms·

> That’s the problem, these tests can’t tell you how good a model is anymore because it’s trivial for labs to optimise for exactly these tests by the next relea

> That’s the problem, these tests can’t tell you how good a model is anymore because it’s trivial for labs to optimise for exactly these tests by the next release. But they don't prove the claim. Are the models amazing at recreating Minecraft, but the second you swap the word Minecraft out with another game or a custom game, it shits the bed? That's not what I see. My feed is full of people using Astra to recreate all sorts of games from Diablo to some random idea they came up with, in ridiculously polished detail like animations that would have taken me weeks of iteration in gpt-5.6-sol but it was a single shot by Astra.

tripleee 20d ago
> ridiculously polished detail Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump
- tarruda 20d ago
  X.com is full of unverified posts, but here's a few: - https://x.com/MengTo/status/2096213835460084184?s=20 https://x.com/MengTo/status/2096213835460084184?s=20 - https://x.com/birdabo/status/2096101381925114327?s=20 https://x.com/birdabo/status/2096101381925114327?s=20 Seems really impressive at first glance, but there's no guarantee that they are real
  - deleted 20d ago
    [deleted]
- simonw 20d ago
  I was impressed by this 3D version of Zork. Video: https://bsky.app/profile/emollick.bsky.social/post/3muqcwa4j4s2i https://bsky.app/profile/emollick.bsky.social/post/3muqcwa4j... Demo: https://zork-underground-empire.netlify.app https://zork-underground-empire.netlify.app Code: https://github.com/emollick/zork-underground-empire https://github.com/emollick/zork-underground-empire Prompts in a screenshot here: https://bsky.app/profile/emollick.bsky.social/post/3muqdyefcf22i https://bsky.app/profile/emollick.bsky.social/post/3muqdyefc... - I extracted them as text here: https://gist.github.com/simonw/dab1843ef17edeaeffd18b0acbba2dfc https://gist.github.com/simonw/dab1843ef17edeaeffd18b0acbba2...
  - tayo42 20d ago
    Impressed like that's cool ai is doing it? Or just impressed in general like its high quality?
    - simonw 20d ago
      I think it's a genuinely good implementation of the concept. It looks good and plays reasonably well and captures what it needs to capture from the original. (Update: OK the sword fighting isn't good. As I got deeper into the game my positive first impressions wore off.)
      - tarruda 20d ago
        I haven't tried any of these demos, but I'm not surprised they stop impressing once you go deep. What got be mostly impressed were the demos of Astra doing computer use. At my job I do some RPA and can appreciate how challenging it can be. Yet they make it look extremely easy to operate a tool like Blender at super human speeds.
        simonw 20d ago
        I've been using Blender via Astra (and Sol before they) without any UI automation at all - Blender supports writing and executing Python scripts, here's how I'm doing that: https://til.simonwillison.net/llms/blender-coding-agents-macos https://til.simonwillison.net/llms/blender-coding-agents-mac... I'd be surprised if any of the impressive Blender demos doing the rounds at the moment were built by having an agent control the mouse and keyboard against the Blender application.
        tarruda 19d ago
        Interesting, thanks for sharing! A scripting API makes the problem much more approachable, but what about those videos where Astra is drawing people from a photo? Here's one using canva: https://x.com/iam_zachi/status/2095992132620136677 https://x.com/iam_zachi/status/2095992132620136677 Is that also using scripting to batch updates? It does look as if the mouse is moving.
enraged_camel 20d ago
You are falling for selection bias. For every person using Astra to create a game from some random idea and sharing the impressive result, there are an unknown number who have tried the same thing and gave up in frustration.
- hombre_fatal 18d ago
  The posts that gain traction and hit my feed are filtered to be the best results, but I don't see what that changes. So are the Minecraft posts. Either way, you'd have to do more work to claim that providers are specifically optimizing for Minecraft vs. other hypotheses like Minecraft just having more resources, preexisting implementations, and base model knowledge than other games. If we're just going by vibes, I'd wager we're at the point that it's more straightforward to improve the model generally than try to special-case specific demos that are popular on social media.
boesboes 20d ago
I call bullshit

tripleee 20d ago

> ridiculously polished detail Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump

tarruda 20d ago
X.com is full of unverified posts, but here's a few: - https://x.com/MengTo/status/2096213835460084184?s=20 https://x.com/MengTo/status/2096213835460084184?s=20 - https://x.com/birdabo/status/2096101381925114327?s=20 https://x.com/birdabo/status/2096101381925114327?s=20 Seems really impressive at first glance, but there's no guarantee that they are real
- deleted 20d ago
  [deleted]
simonw 20d ago
I was impressed by this 3D version of Zork. Video: https://bsky.app/profile/emollick.bsky.social/post/3muqcwa4j4s2i https://bsky.app/profile/emollick.bsky.social/post/3muqcwa4j... Demo: https://zork-underground-empire.netlify.app https://zork-underground-empire.netlify.app Code: https://github.com/emollick/zork-underground-empire https://github.com/emollick/zork-underground-empire Prompts in a screenshot here: https://bsky.app/profile/emollick.bsky.social/post/3muqdyefcf22i https://bsky.app/profile/emollick.bsky.social/post/3muqdyefc... - I extracted them as text here: https://gist.github.com/simonw/dab1843ef17edeaeffd18b0acbba2dfc https://gist.github.com/simonw/dab1843ef17edeaeffd18b0acbba2...
- tayo42 20d ago
  Impressed like that's cool ai is doing it? Or just impressed in general like its high quality?
  - simonw 20d ago
    I think it's a genuinely good implementation of the concept. It looks good and plays reasonably well and captures what it needs to capture from the original. (Update: OK the sword fighting isn't good. As I got deeper into the game my positive first impressions wore off.)
    - tarruda 20d ago
      I haven't tried any of these demos, but I'm not surprised they stop impressing once you go deep. What got be mostly impressed were the demos of Astra doing computer use. At my job I do some RPA and can appreciate how challenging it can be. Yet they make it look extremely easy to operate a tool like Blender at super human speeds.
      - simonw 20d ago
        I've been using Blender via Astra (and Sol before they) without any UI automation at all - Blender supports writing and executing Python scripts, here's how I'm doing that: https://til.simonwillison.net/llms/blender-coding-agents-macos https://til.simonwillison.net/llms/blender-coding-agents-mac... I'd be surprised if any of the impressive Blender demos doing the rounds at the moment were built by having an agent control the mouse and keyboard against the Blender application.
        tarruda 19d ago
        Interesting, thanks for sharing! A scripting API makes the problem much more approachable, but what about those videos where Astra is drawing people from a photo? Here's one using canva: https://x.com/iam_zachi/status/2095992132620136677 https://x.com/iam_zachi/status/2095992132620136677 Is that also using scripting to batch updates? It does look as if the mouse is moving.

enraged_camel 20d ago

You are falling for selection bias. For every person using Astra to create a game from some random idea and sharing the impressive result, there are an unknown number who have tried the same thing and gave up in frustration.

hombre_fatal 18d ago
The posts that gain traction and hit my feed are filtered to be the best results, but I don't see what that changes. So are the Minecraft posts. Either way, you'd have to do more work to claim that providers are specifically optimizing for Minecraft vs. other hypotheses like Minecraft just having more resources, preexisting implementations, and base model knowledge than other games. If we're just going by vibes, I'd wager we're at the point that it's more straightforward to improve the model generally than try to special-case specific demos that are popular on social media.

boesboes 20d ago

I call bullshit