3 ms·

I don't know, that's why I asked b/c I always see a lot of empty platitudes when it comes to LLM praise so I'm curious to see if people can actually back up the

I don't know, that's why I asked b/c I always see a lot of empty platitudes when it comes to LLM praise so I'm curious to see if people can actually back up their claims. I haven't done any 3D modeling so I'll take your word for it but I can tell you that I am working on a very simple interpreter & bytecode compiler for a subset of Erlang & I have yet to see anything novel or even useful from any of the coding assistants. One might naively think that there is enough literature on interpreters & compilers for coding agents to pretty much accomplish the task in one go but that's not what happens in practice.

pushedx 8mo ago
Which agents are you using, and are you using them in an agent mode (Codex, Claude Code etc.)? The difference in quality of output between Claude Sonnet and Claude Opus is around an order of magnitude. The results that you can get from agent mode vs using a chat bot are around two orders of magnitude.
- measurablefunc 8mo ago
  The workflow is not the issue. You are welcome to try the same challenge yourself if you want. Extra test cases (https://drive.proton.me/urls/6Z6557R2WG#n83c6DP6mDfc https://drive.proton.me/urls/6Z6557R2WG#n83c6DP6mDfc) & specification (https://claude.ai/public/artifacts/5581b499-a471-4d58-8e05-147ab7c3ef4e https://claude.ai/public/artifacts/5581b499-a471-4d58-8e05-1...). I know enough about compilers, bytecode VMs, parsers, & interpreters to know that this is well within the capabilities of any reasonably good software engineer but the implementation from Gemini 3.1 Pro (high & low) & Claude Opus 4.6 (thinking) have been less than impressive.
  - Kim_Bruning 8mo ago
    Possibly a dumb question: but are you running this in claude code, or an ide, or basically what are you using to allow for iteration?
    - measurablefunc 8mo ago
      I'm using Google's antigravity IDE. I initially had it configured to run allowed commands (cargo add|build|check|run, testing shell scripts, performance profiling shell scripts, etc.) so that it would iterate & fix bugs w/ as little intervention from me as possible but all it did was burn through the daily allotted tokens so I switched to more "manual" guidance & made a lot more progress w/o burning through the daily limits. What I've learned from this experiment is that the hype does not actually live up to the reality. Maybe the next iteration will manage the task better than the current one but it's obvious that basic compiler & bytecode virtual machine design in a language like Rust is still beyond the capabilities of the current coding agents & whoever thinks I'm wrong is welcome to implement the linked specification to see how far they can get by just "vibing".
      - Kim_Bruning 8mo ago
        That's roughly where I'm at too. I have seen people have some more success after having practices though. Possibly the actual workflows needed for full auto are still kind of tacit. Smaller green-field projecs do work for me already though.
        measurablefunc 8mo ago
        In my experience a few hundred lines w/ a few crates w/ well-defined scopes & a detailed specification is within current capabilities, e.g. compressing wav files w/ wavelets & arithmetic coding. But it's obvious that a correct parser, compiler, & bytecode VM is still beyond current agents even if the specification is detailed enough to cover basically everything.
  - pushedx 8mo ago
    sorry, needed to edit this comment to ask the same question as the sibling: have you run these models in an agent mode that allows for executing the tests, the agent views the output, and iterates on its own for a while? up to an hour or so? you will get vastly different output if you ask the agent to write 200 of its own test cases, and then have it iterate from there
- kmaitreys 8mo ago
  Can you clarify a bit more about the this two orders of magnitude? In what context? Sure, they have "agency" and can do more than outputting text, but I would like see a proper example of this claim.
brookst 8mo ago
It’s taken me a while to get good at using them. My advice: ask for more than what you think it can do. #1 mistake is failing to give enough context about goals, constraints, priorities. Don’t ask “complete this one small task”, ask “hey I’m working on this big project, docs are here, source is there, I’m not sure how to do that, come up with a plan”
- measurablefunc 8mo ago
  The specification is linked in another comment in this thread & you can decide whether it is ambitious enough or not but what I can tell you is that none of the existing coding agents can complete the task even w/ all the details. If you do try it you will eventually get something that will mostly work on simple tests but fail miserably on slightly more complicated test cases.
deleted 8mo ago
[deleted]
joquarky 8mo ago
Most humans can't force themselves to come up with something novel immediately upon demand.
- measurablefunc 8mo ago
  Completely unrelated to the topic or any of the points I was making so did you get confused & respond to the wrong thread?

pushedx 8mo ago

Which agents are you using, and are you using them in an agent mode (Codex, Claude Code etc.)? The difference in quality of output between Claude Sonnet and Claude Opus is around an order of magnitude. The results that you can get from agent mode vs using a chat bot are around two orders of magnitude.

measurablefunc 8mo ago
The workflow is not the issue. You are welcome to try the same challenge yourself if you want. Extra test cases (https://drive.proton.me/urls/6Z6557R2WG#n83c6DP6mDfc https://drive.proton.me/urls/6Z6557R2WG#n83c6DP6mDfc) & specification (https://claude.ai/public/artifacts/5581b499-a471-4d58-8e05-147ab7c3ef4e https://claude.ai/public/artifacts/5581b499-a471-4d58-8e05-1...). I know enough about compilers, bytecode VMs, parsers, & interpreters to know that this is well within the capabilities of any reasonably good software engineer but the implementation from Gemini 3.1 Pro (high & low) & Claude Opus 4.6 (thinking) have been less than impressive.
- Kim_Bruning 8mo ago
  Possibly a dumb question: but are you running this in claude code, or an ide, or basically what are you using to allow for iteration?
  - measurablefunc 8mo ago
    I'm using Google's antigravity IDE. I initially had it configured to run allowed commands (cargo add|build|check|run, testing shell scripts, performance profiling shell scripts, etc.) so that it would iterate & fix bugs w/ as little intervention from me as possible but all it did was burn through the daily allotted tokens so I switched to more "manual" guidance & made a lot more progress w/o burning through the daily limits. What I've learned from this experiment is that the hype does not actually live up to the reality. Maybe the next iteration will manage the task better than the current one but it's obvious that basic compiler & bytecode virtual machine design in a language like Rust is still beyond the capabilities of the current coding agents & whoever thinks I'm wrong is welcome to implement the linked specification to see how far they can get by just "vibing".
    - Kim_Bruning 8mo ago
      That's roughly where I'm at too. I have seen people have some more success after having practices though. Possibly the actual workflows needed for full auto are still kind of tacit. Smaller green-field projecs do work for me already though.
      - measurablefunc 8mo ago
        In my experience a few hundred lines w/ a few crates w/ well-defined scopes & a detailed specification is within current capabilities, e.g. compressing wav files w/ wavelets & arithmetic coding. But it's obvious that a correct parser, compiler, & bytecode VM is still beyond current agents even if the specification is detailed enough to cover basically everything.
- pushedx 8mo ago
  sorry, needed to edit this comment to ask the same question as the sibling: have you run these models in an agent mode that allows for executing the tests, the agent views the output, and iterates on its own for a while? up to an hour or so? you will get vastly different output if you ask the agent to write 200 of its own test cases, and then have it iterate from there
kmaitreys 8mo ago
Can you clarify a bit more about the this two orders of magnitude? In what context? Sure, they have "agency" and can do more than outputting text, but I would like see a proper example of this claim.

brookst 8mo ago

It’s taken me a while to get good at using them. My advice: ask for more than what you think it can do. #1 mistake is failing to give enough context about goals, constraints, priorities. Don’t ask “complete this one small task”, ask “hey I’m working on this big project, docs are here, source is there, I’m not sure how to do that, come up with a plan”

measurablefunc 8mo ago
The specification is linked in another comment in this thread & you can decide whether it is ambitious enough or not but what I can tell you is that none of the existing coding agents can complete the task even w/ all the details. If you do try it you will eventually get something that will mostly work on simple tests but fail miserably on slightly more complicated test cases.

deleted 8mo ago

[deleted]

joquarky 8mo ago

Most humans can't force themselves to come up with something novel immediately upon demand.

measurablefunc 8mo ago
Completely unrelated to the topic or any of the points I was making so did you get confused & respond to the wrong thread?