4 ms·

This is fascinating to me. I completely believe you and I will not bother you with all the common "but did you try to tell it this or that" responses, but this

This is fascinating to me. I completely believe you and I will not bother you with all the common "but did you try to tell it this or that" responses, but this is such a different experience from mine. I did the exact same task with claude in the Julia language last week, and everything worked perfectly. I am now in the habit of adding "keep it simple, use only public interfaces, do not use internals, be elegant and extremely minimal in your changes" to all my requests or SKILL.md or AGENTS.md files (because of the occasional failure like the one you described). But generally speaking, such complete failures have been so very rare for me, that it is amazing to see that others have had such a completely different experience.

shafyy 6mo ago
It's almost like.... LLMs are non-deterministic and hallucinating... Oh wait?!
- krastanov 6mo ago
  non-deterministic does not mean it is not biased towards a particular type of results (helpful results)
jayd16 6mo ago
You say it doesn't fail but you also mention all these work around you know and try...sounds like it fails a lot but your tolerance is different.
- zorak8me 6mo ago
  Providing instruction and context doesn’t seem like a “workaround”.
  - theshrike79 6mo ago
    Agents are LLMs that use tools in a loop I didn't give my Agent any tools, it hallucinated code. ...well, yes. It's like evaluating a programmer's skill by having one-shot a program on paper with zero syntax errors.
- plagiarist 6mo ago
  IME it does fail pretty hard at first. One has to build up a little library of markdown and internal library of prompt techniques. Then it starts working okay. I agree there is a hurdle still, trying it on one task doesn't really get one over the hurdle.
- jmalicki 6mo ago
  Most people I've seen complain say things like "I asked it for code and it didn't compile." The real magic of LLMs comes when they iterate until completion until the code compiles and the test passes, and you don't even bother looking at it until then. Each step is pretty stupid, but the ability to very quickly doggedly keep at it until success quite often produces great work. If you don't have linters that are checking for valid syntax and approved coding style, if you don't have tests to ensure the LLM doesn't screw up the code, you don't have good CI, you're going to have a bad time. LLMs are just like extremely bright but sloppy junior devs - if you think about putting the same guardrails in place for your project you would for that case, things tend to work very well - you're giving the LLM a chance to check its work and self correct. It's the agentic loop that makes it work, not the single-shot output of an LLM.
  - alecbz 6mo ago
    Stuff like this works for things that can be verified programmatically (though I find LLMs still do occasionally ignore instructions like this), but ensuring correct functionality and sensible code organization are bigger challenges. There are techniques that can help deal with this but none of them work perfectly, and most of the time some direct oversight from me is required. And this really clips the potential productivity gains, because in order to effectively provide oversight you need to page in all the context of what's going on and how it ought to work, which is most of what the LLMs are in-theory helping you with. LLMs are still very useful for certain tasks (bootstrapping in new unfamiliar domains, tedious plumbing or test fixture code), but the massive productivity gains people are claiming or alluding to still feel out of reach.
    - jmalicki 6mo ago
      It depends - there are some very very difficult things that can still be easily verifiable! For instance, if you are working on a compiler and have a huge test database of code to compile that all has tests itself, "all sample code must compile and pass tests, ensuring your new optimizer code gets adequate branch coverage in the process" - the underlying task can be very difficult, but you have large amounts of test coverage that have a very good chance at catching errors there. At the very least "LLM code compiles, and is formatted and documented according to lint rules" is pretty basic. If people are saying LLM code doesn't compile, then yes, you are using it very incorrectly, as you're not even beginning to engage the agentic loop at all, as compiling is the simplest step. Sure, a lot of more complex cases require oversight or don't work. But "the code didn't compile" is definitely in "you're holding it wrong" territority, and it's not even subtle.
aqua_coder 6mo ago
My experience with working with AI agents is that they can be verbose and do things that are too over complicated by default. Unless directed explicitly. Which may be the reason for this discrepancy.
sigbottle 6mo ago
Idk. Had a friend recommend me gsdv2 and I wasted like 100$ + so much time trying to debug said crap. I went back to codex and it 1 shot my problems easily. And this was from two people who were 100% aligned on agentic AI coding. I've been using AI for years now and agentic AI for several months now. I was told that I bring out the worst in LLMs. Except... I was able to achieve better results, WITH LLMs, on OTHER frameworks. So like, ? It may be easier to draw the boundary between "AI and non AI users", but as AI becomes prolific, the us vs them angle that people keep using won't apply anymore. The age old "user versus tool" debate goes on, but it seems like gaslighting is popular these days. I classify it as gaslighting because I'm clearly a falsifiable test case, I'm even gung ho for LLMs, yet any kind of dissent is immediately warped into user error. It doesn't matter what you say or where you are on the spectrum, if you have 1 bad experience and speak up about it it's an issue. Guess that's not really an issue for the human condition though.

shafyy 6mo ago

It's almost like.... LLMs are non-deterministic and hallucinating... Oh wait?!

krastanov 6mo ago
non-deterministic does not mean it is not biased towards a particular type of results (helpful results)

jayd16 6mo ago

You say it doesn't fail but you also mention all these work around you know and try...sounds like it fails a lot but your tolerance is different.

zorak8me 6mo ago
Providing instruction and context doesn’t seem like a “workaround”.
- theshrike79 6mo ago
  Agents are LLMs that use tools in a loop I didn't give my Agent any tools, it hallucinated code. ...well, yes. It's like evaluating a programmer's skill by having one-shot a program on paper with zero syntax errors.
plagiarist 6mo ago
IME it does fail pretty hard at first. One has to build up a little library of markdown and internal library of prompt techniques. Then it starts working okay. I agree there is a hurdle still, trying it on one task doesn't really get one over the hurdle.
jmalicki 6mo ago
Most people I've seen complain say things like "I asked it for code and it didn't compile." The real magic of LLMs comes when they iterate until completion until the code compiles and the test passes, and you don't even bother looking at it until then. Each step is pretty stupid, but the ability to very quickly doggedly keep at it until success quite often produces great work. If you don't have linters that are checking for valid syntax and approved coding style, if you don't have tests to ensure the LLM doesn't screw up the code, you don't have good CI, you're going to have a bad time. LLMs are just like extremely bright but sloppy junior devs - if you think about putting the same guardrails in place for your project you would for that case, things tend to work very well - you're giving the LLM a chance to check its work and self correct. It's the agentic loop that makes it work, not the single-shot output of an LLM.
- alecbz 6mo ago
  Stuff like this works for things that can be verified programmatically (though I find LLMs still do occasionally ignore instructions like this), but ensuring correct functionality and sensible code organization are bigger challenges. There are techniques that can help deal with this but none of them work perfectly, and most of the time some direct oversight from me is required. And this really clips the potential productivity gains, because in order to effectively provide oversight you need to page in all the context of what's going on and how it ought to work, which is most of what the LLMs are in-theory helping you with. LLMs are still very useful for certain tasks (bootstrapping in new unfamiliar domains, tedious plumbing or test fixture code), but the massive productivity gains people are claiming or alluding to still feel out of reach.
  - jmalicki 6mo ago
    It depends - there are some very very difficult things that can still be easily verifiable! For instance, if you are working on a compiler and have a huge test database of code to compile that all has tests itself, "all sample code must compile and pass tests, ensuring your new optimizer code gets adequate branch coverage in the process" - the underlying task can be very difficult, but you have large amounts of test coverage that have a very good chance at catching errors there. At the very least "LLM code compiles, and is formatted and documented according to lint rules" is pretty basic. If people are saying LLM code doesn't compile, then yes, you are using it very incorrectly, as you're not even beginning to engage the agentic loop at all, as compiling is the simplest step. Sure, a lot of more complex cases require oversight or don't work. But "the code didn't compile" is definitely in "you're holding it wrong" territority, and it's not even subtle.

aqua_coder 6mo ago

My experience with working with AI agents is that they can be verbose and do things that are too over complicated by default. Unless directed explicitly. Which may be the reason for this discrepancy.

sigbottle 6mo ago

Idk. Had a friend recommend me gsdv2 and I wasted like 100$ + so much time trying to debug said crap. I went back to codex and it 1 shot my problems easily. And this was from two people who were 100% aligned on agentic AI coding. I've been using AI for years now and agentic AI for several months now. I was told that I bring out the worst in LLMs. Except... I was able to achieve better results, WITH LLMs, on OTHER frameworks. So like, ? It may be easier to draw the boundary between "AI and non AI users", but as AI becomes prolific, the us vs them angle that people keep using won't apply anymore. The age old "user versus tool" debate goes on, but it seems like gaslighting is popular these days. I classify it as gaslighting because I'm clearly a falsifiable test case, I'm even gung ho for LLMs, yet any kind of dissent is immediately warped into user error. It doesn't matter what you say or where you are on the spectrum, if you have 1 bad experience and speak up about it it's an issue. Guess that's not really an issue for the human condition though.