3 ms·

> In the past year, agent harnesses crossed the “can it be done” rubicon. Brother, I'm still in "Can you get it right?"-mode. What am I doing wrong? (Rhetorica

by hirvi74 2mo ago

> In the past year, agent harnesses crossed the “can it be done” rubicon. Brother, I'm still in "Can you get it right?"-mode. What am I doing wrong? (Rhetorical, but advice welcomed).

MattGaiser 2mo ago
What is “it” specifically and what languages are you using?
- hirvi74 2mo ago
  "It" is a lot of things. Languages: - C#/.NET: Sufficient sometimes, but not how I'd write things. Most results at least compile, but I have noticed plenty of defiance towards particular instructions, e.g, "Do not use <x>, use <y>" -> code contains <x> and not <y>. - C#/Godot: I have noticed the greatest amounts of defiance here. Not to mention most results are an 80/20 implementation of what I asked for. And no, I am not trying to one-shot a full game or anything. - AArch64 and x86: great results surprisingly, though only small amounts were produced. Mainly, assistance with RE-ing and cracking some binaries from https://crackmes.one https://crackmes.one or where ever. - The Lord's Language (Swift): Maybe the LLMs are better at SwiftUI/Swift, but I have had some rough results going down the opposite direction of the software stack. I have on/off been working on a personal, FOSS "productivity" tool for macOS, e.g., mouseless navigation, window management, GUI automation, etc.. This type of development requires a significant amount work with C APIs like CoreGraphics, Accessibility, CoreFoundation, etc.. The code isn't the problem for me, it's the lack of useful debugging. LLMs, last I have tried (around Opus 4.6 times), seemed to really struggle with things like CoreGraphics Y-axis coordinates being inverted compared AppKit's and other stuff like that. - Applescript (GUI automation): Do not even waste your time trying (I fault no LLMs for this either). - elisp: the code is usually sufficient, though package config can be a little dicey. - Shell scripts (Zsh, Bash, Powershell): great results. - Python: I try to avoid this language unless necessary, but the results have been great. These days, I use the plain Web chat interfaces for about 95% of my usage compared to the CLI harnesses. Sometime ago, I realized I get better results that way. With the web chat, I would say my results have been outstanding.
  - jaggederest 2mo ago
    Direct feedback: You have to give up on style. "not how I'd write things" is not a blocker. Defiance of instructions is normal, you just have to steer it and correct. There's no substitute for diligence yet. 80/20 - this means your scope was too large, split the scope or tell the agent to revert, split the scope, and try again. RE and assembler: it's really good at this stuff. It can patch almost any binary with the right tools Swift: you have to give it tool usage in whatever result you're wanting. If it's a macos app, you have to let the LLM pilot it to get feedback, or build an extensive end to end test suite that it can drive autonomously. If you get into the loop on changes it'll feel awful and like no time savings. Review at the level of using the app and looking at the code, not in process or reviewing every tool call or diff. applescript: works great, I have a bunch of automation set up this way, what problems are you seeing? elisp: tough language, llms kinda hate parentheses unless you're really tight on the linting, and elisp is enough of its own animal that the training for e.g. common lisp isn't great. python: will suck unless you enable all the typechecking, make it use bdd, and have a linter/formatter run precommit and yell at the robot for you. Web vs cli: you should use the cli 100% because it lets you change the environment, if you're getting better results on web, you haven't set your local environment up very well. My personal preference is to run my own dev server on aws but that's spendy.
al_borland 2mo ago
I’ve found some success is small projects, with limited scope, in a greenfield. I’m terrified to attempt agentic anything in the repo my job actually cares about. I triggered it once by accident, when the agent was first rolled out and enabled by default… it broke everything. Now I just use ask mode, and even that is wrong half the time, and once it goes wrong it just keeps getting worse. I saw a post from Dave Plumber who vibe coded up a new cross platform task manager. He said his spec document for the AI was 107 pages long. So maybe what I’m doing wrong is not giving the AI a literal novel of spec.
- applfanboysbgon 2mo ago
  > He said his spec document for the AI was 107 pages long. This sounds like programming but with extra steps that make it take longer with less reliability.
  - 0x696C6961 2mo ago
    Ikr, at that point the code itself is a better way of encoding the information.
- hirvi74 2mo ago
  I concur with your first sentence. I have found success creating some sort of MVP, but I have had virtually no success with taking something from initiation to completion. My employer won't even provide LLMs for us, let alone allow us to use agentic coding on our repos. All our code is still USDA certified, organic, free-range code. > He said his spec document for the AI was 107 pages long. Absolutely not. My ADHD forbids such temptations of the dark arts. I'll feed any LLM a 107 page spec list, but I won't be writing nor reading that spec list.
simonw 2mo ago
Tell it to use red/green TDD and start things off with an already configured test suite, maybe with a single test that asserts 1+1==2. Make sure it know how to run the tests before it starts writing any additional code. Then set it a clear goal.
- slopinthebag 2mo ago
  Basically all the examples of LLM's building impressive things have been because they have human written tests to base the implementation on. If you have an LLM write the tests the results are far less impressive or valuable.
  - bharatsuthar 2mo ago
    Yes and LLMs are known to cheat on tests written by them.
    - slopinthebag 2mo ago
      It's not always cheating either. They aren't intelligent, so they don't actually understand the purpose of the tests or can build them to define the actual semantics of the problem space. It's literally just next-token prediction based on the codebase and prompt. Cheating implies that they have agency, and ironically agents don't.
    - andai 2mo ago
      Last year when they added computer use to Claude web I was excited to try it out. I just asked it for a code snippet and it ended up setting up a whole repo in a docker container or something. Even volunteered a test suite. This genuinely amazed me. ...until I checked the tests. It was just console.log("Tests passed!") AGI 2027
    - simonw 2mo ago
      Have you seen that recently? I used to see that happen, but I've not caught it with the more recent (Opus 4.5+, Fable 5, GOT 5.5/5.6) models.
mw888 2mo ago
You're appealing to ambiguity. All you've said is you have failed—how is anyone supposed to know what went wrong?
- hirvi74 2mo ago
  I suppose they aren't, but I am perfectly fine with reading what has worked for others should they feel inclined to share.
  - mw888 2mo ago
    Generally constraining scope and providing enough existing material until the LLM is productive. The common counter-argument is that specifying to the sufficient level is more work than just not using an LLM at all. I find that is not a universal rule.
jaggederest 2mo ago
I'd be happy to screenshare with you if you like, we can work on something trivial or open source. Half an hour should be more than enough to see whether you're doing anything obviously self-sabotaging.
dosisking 2mo ago
There are two 'camps' with respect to AI. One camp already knows that Neural Nets don't work and are a dead end. The other camp hasn't yet figured out that Neural Nets don't work, but are convinced that they do (or eventually will), because they think everything always improves over time in a linear fashion.

by hirvi74 2mo ago

> In the past year, agent harnesses crossed the “can it be done” rubicon. Brother, I'm still in "Can you get it right?"-mode. What am I doing wrong? (Rhetorical, but advice welcomed).

MattGaiser 2mo ago

What is “it” specifically and what languages are you using?

hirvi74 2mo ago
"It" is a lot of things. Languages: - C#/.NET: Sufficient sometimes, but not how I'd write things. Most results at least compile, but I have noticed plenty of defiance towards particular instructions, e.g, "Do not use <x>, use <y>" -> code contains <x> and not <y>. - C#/Godot: I have noticed the greatest amounts of defiance here. Not to mention most results are an 80/20 implementation of what I asked for. And no, I am not trying to one-shot a full game or anything. - AArch64 and x86: great results surprisingly, though only small amounts were produced. Mainly, assistance with RE-ing and cracking some binaries from https://crackmes.one https://crackmes.one or where ever. - The Lord's Language (Swift): Maybe the LLMs are better at SwiftUI/Swift, but I have had some rough results going down the opposite direction of the software stack. I have on/off been working on a personal, FOSS "productivity" tool for macOS, e.g., mouseless navigation, window management, GUI automation, etc.. This type of development requires a significant amount work with C APIs like CoreGraphics, Accessibility, CoreFoundation, etc.. The code isn't the problem for me, it's the lack of useful debugging. LLMs, last I have tried (around Opus 4.6 times), seemed to really struggle with things like CoreGraphics Y-axis coordinates being inverted compared AppKit's and other stuff like that. - Applescript (GUI automation): Do not even waste your time trying (I fault no LLMs for this either). - elisp: the code is usually sufficient, though package config can be a little dicey. - Shell scripts (Zsh, Bash, Powershell): great results. - Python: I try to avoid this language unless necessary, but the results have been great. These days, I use the plain Web chat interfaces for about 95% of my usage compared to the CLI harnesses. Sometime ago, I realized I get better results that way. With the web chat, I would say my results have been outstanding.
- jaggederest 2mo ago
  Direct feedback: You have to give up on style. "not how I'd write things" is not a blocker. Defiance of instructions is normal, you just have to steer it and correct. There's no substitute for diligence yet. 80/20 - this means your scope was too large, split the scope or tell the agent to revert, split the scope, and try again. RE and assembler: it's really good at this stuff. It can patch almost any binary with the right tools Swift: you have to give it tool usage in whatever result you're wanting. If it's a macos app, you have to let the LLM pilot it to get feedback, or build an extensive end to end test suite that it can drive autonomously. If you get into the loop on changes it'll feel awful and like no time savings. Review at the level of using the app and looking at the code, not in process or reviewing every tool call or diff. applescript: works great, I have a bunch of automation set up this way, what problems are you seeing? elisp: tough language, llms kinda hate parentheses unless you're really tight on the linting, and elisp is enough of its own animal that the training for e.g. common lisp isn't great. python: will suck unless you enable all the typechecking, make it use bdd, and have a linter/formatter run precommit and yell at the robot for you. Web vs cli: you should use the cli 100% because it lets you change the environment, if you're getting better results on web, you haven't set your local environment up very well. My personal preference is to run my own dev server on aws but that's spendy.

al_borland 2mo ago

I’ve found some success is small projects, with limited scope, in a greenfield. I’m terrified to attempt agentic anything in the repo my job actually cares about. I triggered it once by accident, when the agent was first rolled out and enabled by default… it broke everything. Now I just use ask mode, and even that is wrong half the time, and once it goes wrong it just keeps getting worse. I saw a post from Dave Plumber who vibe coded up a new cross platform task manager. He said his spec document for the AI was 107 pages long. So maybe what I’m doing wrong is not giving the AI a literal novel of spec.

applfanboysbgon 2mo ago
> He said his spec document for the AI was 107 pages long. This sounds like programming but with extra steps that make it take longer with less reliability.
- 0x696C6961 2mo ago
  Ikr, at that point the code itself is a better way of encoding the information.
hirvi74 2mo ago
I concur with your first sentence. I have found success creating some sort of MVP, but I have had virtually no success with taking something from initiation to completion. My employer won't even provide LLMs for us, let alone allow us to use agentic coding on our repos. All our code is still USDA certified, organic, free-range code. > He said his spec document for the AI was 107 pages long. Absolutely not. My ADHD forbids such temptations of the dark arts. I'll feed any LLM a 107 page spec list, but I won't be writing nor reading that spec list.

simonw 2mo ago

Tell it to use red/green TDD and start things off with an already configured test suite, maybe with a single test that asserts 1+1==2. Make sure it know how to run the tests before it starts writing any additional code. Then set it a clear goal.

slopinthebag 2mo ago
Basically all the examples of LLM's building impressive things have been because they have human written tests to base the implementation on. If you have an LLM write the tests the results are far less impressive or valuable.
- bharatsuthar 2mo ago
  Yes and LLMs are known to cheat on tests written by them.
  - slopinthebag 2mo ago
    It's not always cheating either. They aren't intelligent, so they don't actually understand the purpose of the tests or can build them to define the actual semantics of the problem space. It's literally just next-token prediction based on the codebase and prompt. Cheating implies that they have agency, and ironically agents don't.
  - andai 2mo ago
    Last year when they added computer use to Claude web I was excited to try it out. I just asked it for a code snippet and it ended up setting up a whole repo in a docker container or something. Even volunteered a test suite. This genuinely amazed me. ...until I checked the tests. It was just console.log("Tests passed!") AGI 2027
  - simonw 2mo ago
    Have you seen that recently? I used to see that happen, but I've not caught it with the more recent (Opus 4.5+, Fable 5, GOT 5.5/5.6) models.

mw888 2mo ago

You're appealing to ambiguity. All you've said is you have failed—how is anyone supposed to know what went wrong?

hirvi74 2mo ago
I suppose they aren't, but I am perfectly fine with reading what has worked for others should they feel inclined to share.
- mw888 2mo ago
  Generally constraining scope and providing enough existing material until the LLM is productive. The common counter-argument is that specifying to the sufficient level is more work than just not using an LLM at all. I find that is not a universal rule.

jaggederest 2mo ago

I'd be happy to screenshare with you if you like, we can work on something trivial or open source. Half an hour should be more than enough to see whether you're doing anything obviously self-sabotaging.

dosisking 2mo ago

There are two 'camps' with respect to AI. One camp already knows that Neural Nets don't work and are a dead end. The other camp hasn't yet figured out that Neural Nets don't work, but are convinced that they do (or eventually will), because they think everything always improves over time in a linear fashion.