3 ms·

You are saying: it turns out that one-shot playable Minecraft clones are actually pretty simple. Maybe it seemed a hard problem to programmers, but why not jus

You are saying: it turns out that one-shot playable Minecraft clones are actually pretty simple. Maybe it seemed a hard problem to programmers, but why not just say that the verifiable-rewards training has shown that their skill is unusually simple?

senordevnyc 3mo ago
What a great example of the no true Scotsman fallacy.
latentsea 3mo ago
I am saying 3 years ago there wasn't a snowballs chance in hell I could one shot a playable Minecraft clone, and there has never been a snowballs chance in hell a human developer could do so in 45 minutes. The difficulty of the task and human performance on that task hasn't changed. LLMs performance on that task has changed dramatically.
- 8note 3mo ago
  it could almost certainly regurgitate something like minetest its in the training data, probably many times, and it was then, too
  - latentsea 3mo ago
    And that's exactly why 3 years ago we saw people one shotting it in 45 minutes :)
    - staticman2 3mo ago
      "Some idiot on Twitter did a game tech demo" has been a thing since, like GPT 3.5 or 4.0?
      - latentsea 3mo ago
        It has. We can see 3.5 started with text adventure games https://medium.com/@duncanariey/using-chatgpt-to-make-the-ittiest-bittiest-text-based-adventure-game-f642e8da4dad https://medium.com/@duncanariey/using-chatgpt-to-make-the-it... and with 4.0 it moved up to pong https://x.com/skirano/status/1635736107949195278 https://x.com/skirano/status/1635736107949195278 As the generations increase, so too has the complexity of the games it can one-shot.
        staticman2 3mo ago
        But nobody has nailed "this is a game anyone wants to play" or "professional game development life cycles are shorter" so it's not particularly impressive. I'm still waiting for any evidence of LLM economic impact in the real world.
        latentsea 3mo ago
        Yeah, well I doubt any economic impact will from LLMs will ever come. I doubt anyone is working on nailing those things as we speak. I doubt the results my team and I are getting with agentic engineering are real. I doubt anyone can take increasingly better tools and translate that to economic value.
        staticman2 3mo ago
        If you're doing such amazing things why the need to gush about Minecraft tweets?
        latentsea 3mo ago
        Why wouldn't I be? Those demos are put out by non-engineer AI-bros who start from a blank slate and just test raw one-shot performance of the model. In comparison I'm aware of how much additional performance you can extract out of models with a team of professional engineers that are balls-deep in harness engineering. That delta is very large. It's exciting.
cornholio 3mo ago
Can't wait to see what unusually simple Erdos problem LLMs will expose next, hiding in plain sight for decades and seemingly intractable for professional mathematicians who weren't aware just how simple the problem was.

senordevnyc 3mo ago

What a great example of the no true Scotsman fallacy.

latentsea 3mo ago

I am saying 3 years ago there wasn't a snowballs chance in hell I could one shot a playable Minecraft clone, and there has never been a snowballs chance in hell a human developer could do so in 45 minutes. The difficulty of the task and human performance on that task hasn't changed. LLMs performance on that task has changed dramatically.

8note 3mo ago
it could almost certainly regurgitate something like minetest its in the training data, probably many times, and it was then, too
- latentsea 3mo ago
  And that's exactly why 3 years ago we saw people one shotting it in 45 minutes :)
  - staticman2 3mo ago
    "Some idiot on Twitter did a game tech demo" has been a thing since, like GPT 3.5 or 4.0?
    - latentsea 3mo ago
      It has. We can see 3.5 started with text adventure games https://medium.com/@duncanariey/using-chatgpt-to-make-the-ittiest-bittiest-text-based-adventure-game-f642e8da4dad https://medium.com/@duncanariey/using-chatgpt-to-make-the-it... and with 4.0 it moved up to pong https://x.com/skirano/status/1635736107949195278 https://x.com/skirano/status/1635736107949195278 As the generations increase, so too has the complexity of the games it can one-shot.
      - staticman2 3mo ago
        But nobody has nailed "this is a game anyone wants to play" or "professional game development life cycles are shorter" so it's not particularly impressive. I'm still waiting for any evidence of LLM economic impact in the real world.
        latentsea 3mo ago
        Yeah, well I doubt any economic impact will from LLMs will ever come. I doubt anyone is working on nailing those things as we speak. I doubt the results my team and I are getting with agentic engineering are real. I doubt anyone can take increasingly better tools and translate that to economic value.
        staticman2 3mo ago
        If you're doing such amazing things why the need to gush about Minecraft tweets?
        latentsea 3mo ago
        Why wouldn't I be? Those demos are put out by non-engineer AI-bros who start from a blank slate and just test raw one-shot performance of the model. In comparison I'm aware of how much additional performance you can extract out of models with a team of professional engineers that are balls-deep in harness engineering. That delta is very large. It's exciting.

cornholio 3mo ago

Can't wait to see what unusually simple Erdos problem LLMs will expose next, hiding in plain sight for decades and seemingly intractable for professional mathematicians who weren't aware just how simple the problem was.