6 ms·

> try it, you might actually be amazed. I keep being told this and the tools keep falling at the first hurdle. This morning I asked Claude to use a library to

by maccard 7mo ago

> try it, you might actually be amazed. I keep being told this and the tools keep falling at the first hurdle. This morning I asked Claude to use a library to load a toml file in .net and print a value. It immediately explained how it was an easy file format to parse and didn’t need a library. I undid, went back to plan mode and it picked a library, added it and claimed it was done. Except the code didn’t compile. Three iterations later of trying to get Claude to make it compile (it changed random lines around the clear problematic line) I fixed it by following the example in the readme, and told Claude. I then asked Claude to parse the rest of the toml file, whereby it blew away the compile fix I had made.. This isn’t an isolated experience - I hit these fundamental blocking issues with pretty much every attempt to use these tools that isn’t “implement a web page”, and even when it does that it’s not long before it gets tangled up in something or other…

krastanov 7mo ago
This is fascinating to me. I completely believe you and I will not bother you with all the common "but did you try to tell it this or that" responses, but this is such a different experience from mine. I did the exact same task with claude in the Julia language last week, and everything worked perfectly. I am now in the habit of adding "keep it simple, use only public interfaces, do not use internals, be elegant and extremely minimal in your changes" to all my requests or SKILL.md or AGENTS.md files (because of the occasional failure like the one you described). But generally speaking, such complete failures have been so very rare for me, that it is amazing to see that others have had such a completely different experience.
- shafyy 7mo ago
  It's almost like.... LLMs are non-deterministic and hallucinating... Oh wait?!
  - krastanov 7mo ago
    non-deterministic does not mean it is not biased towards a particular type of results (helpful results)
- jayd16 7mo ago
  You say it doesn't fail but you also mention all these work around you know and try...sounds like it fails a lot but your tolerance is different.
  - zorak8me 7mo ago
    Providing instruction and context doesn’t seem like a “workaround”.
    - theshrike79 7mo ago
      Agents are LLMs that use tools in a loop I didn't give my Agent any tools, it hallucinated code. ...well, yes. It's like evaluating a programmer's skill by having one-shot a program on paper with zero syntax errors.
  - plagiarist 7mo ago
    IME it does fail pretty hard at first. One has to build up a little library of markdown and internal library of prompt techniques. Then it starts working okay. I agree there is a hurdle still, trying it on one task doesn't really get one over the hurdle.
  - jmalicki 7mo ago
    Most people I've seen complain say things like "I asked it for code and it didn't compile." The real magic of LLMs comes when they iterate until completion until the code compiles and the test passes, and you don't even bother looking at it until then. Each step is pretty stupid, but the ability to very quickly doggedly keep at it until success quite often produces great work. If you don't have linters that are checking for valid syntax and approved coding style, if you don't have tests to ensure the LLM doesn't screw up the code, you don't have good CI, you're going to have a bad time. LLMs are just like extremely bright but sloppy junior devs - if you think about putting the same guardrails in place for your project you would for that case, things tend to work very well - you're giving the LLM a chance to check its work and self correct. It's the agentic loop that makes it work, not the single-shot output of an LLM.
- aqua_coder 7mo ago
  My experience with working with AI agents is that they can be verbose and do things that are too over complicated by default. Unless directed explicitly. Which may be the reason for this discrepancy.
- sigbottle 7mo ago
  Idk. Had a friend recommend me gsdv2 and I wasted like 100$ + so much time trying to debug said crap. I went back to codex and it 1 shot my problems easily. And this was from two people who were 100% aligned on agentic AI coding. I've been using AI for years now and agentic AI for several months now. I was told that I bring out the worst in LLMs. Except... I was able to achieve better results, WITH LLMs, on OTHER frameworks. So like, ? It may be easier to draw the boundary between "AI and non AI users", but as AI becomes prolific, the us vs them angle that people keep using won't apply anymore. The age old "user versus tool" debate goes on, but it seems like gaslighting is popular these days. I classify it as gaslighting because I'm clearly a falsifiable test case, I'm even gung ho for LLMs, yet any kind of dissent is immediately warped into user error. It doesn't matter what you say or where you are on the spectrum, if you have 1 bad experience and speak up about it it's an issue. Guess that's not really an issue for the human condition though.
wrs 7mo ago
I’m honestly baffled by this. I don’t want to tell you “you’re holding it wrong” but if this is your normal experience there’s something weird happening. Friday afternoon I made a new directory and told Claude Code I wanted to make a Go proxy so I could have a request/callback HTTP API for a 3rd party service whose official API is only persistent websocket connections. I had it read the service’s API docs, engage in some back and forth to establish the architecture and library choices, and save out a phased implementation plan in plan mode. It implemented it in four phases with passing tests for each, then did live tests against the service in which it debugged its protocol mistakes using curl. Finally I had it do two rounds of code review with fresh context, and it fixed a race condition and made a few things cleaner. Total time, two hours. I have noticed some people I work with have more trouble, and my vague intuition is it happens when they give Claude too much autonomy. It works better when you tell it what to do, rather than letting it decide. That can be at a pretty high level, though. Basically reduce the problem to a set of well-established subproblems that it’s familiar with. Same as you’d do with a junior developer, really.
- shireboy 7mo ago
  Similar. I regularly use Github copilot (with claude models sometimes) and it works amazingly. But I see some who struggle with them. I have sort of learned to talk to it, understand what it is generating, and routinely use to generate fixes, whole features, etc. much much faster than I could before.
  - maccard 7mo ago
    I’ve found copilot to generate worse code, and the editor integrations to be unusable (lets insert this block of code at the wrong place in the file for example), but overall for it to do the right thing enough that I can patch it. I’ve found Claude code just goes mad and gets stuck in a loop and I need to pull it out and figure out where in the hundreds of lines of code it’s generates in the last few minutes it’s gone so wrong.
- thwarted 7mo ago
  > it happens when they give Claude too much autonomy. It works better when you tell it what to do, rather than letting it decide. That can be at a pretty high level, though. Basically reduce the problem to a set of well-established subproblems that it’s familiar with. Same as you’d do with a junior developer, really. Equating "junior developers" and "coding LLMs" is pretty lame. You handhold a junior developers so, eventually, you don't have to handhold anymore. The junior developer is expected to learn enough, and be trusted enough, to operate more autonomously. "Junior developers" don't exist solely to do your bidding. It may be valuable to recognize similarities between a first junior developer interaction and a first LLM interaction, but when every LLM interaction requires it to be handheld, the value of the iterative nature of having a junior developer work along side you is not at all equivalent.
DougN7 7mo ago
I have similar experiences. It has worked about half the time, but the code has to be pretty simple. I’ve many experiences, where we work on something complicated for an hour, and it is good compiling code, but then an edge case comes to mind that I ask about and Claude tells me the whole approach is doomed and will never work for that case. It has even apologized a few times for misleading me :) I feel like it’s this weird mix of brilliant moron. But yeah, ask for a simple HTML page with a few fields and it rocks.
roncesvalles 7mo ago
Did you use the best model available to you (Opus 4.6)? There is a world of difference between using the highest model vs the fast one. The fast ones are basically useless and it's a shame that all these tools default to it.
- fweimer 7mo ago
  To make this more confusing, there is a fast mode of Opus 4.6, which (as far as I understand it) is supposed to deliver the same results as the standard mode. It's much more expensive, but advertised as more interactive.
- moffkalast 7mo ago
  Opus is not cost effective.
  - deleted 7mo ago
    [deleted]
  - senordevnyc 7mo ago
    Compared to what? I spent about $700 last month on Cursor, mostly on Opus 4.6. The time it saved me is at least 10-20x more valuable than that. I keep trying to use GPT-5.3 Codex, because it's like 40% cheaper, but then the quality doesn't always seem quite as good, and it's honestly not worth it to me to save a couple hundred bucks a month for something that's 10% worse. The multiplicative value of what I'm building means it's worth it to pay extra for the SOTA model in almost all cases.
  - roncesvalles 7mo ago
    The cost-effectiveness of something that doesn't work is 0.
phromo 7mo ago
I don't know why but I find performance on c#/.net be several generations behind. Sometimes right ofc but my general experience is if you pull the generation slot machine in just about any other language it will work better. I regularly do python, typescript, ruby and rust with a better experience. It's even hard to find benchmarks where csharp is included.
JakeStone 7mo ago
If Claude ends up grabbing my C# TOML library, in my defense, I wrote it when the TOML format first came out over a dozen years ago, and never did anything more with it Sorry.
- maccard 7mo ago
  Nope! https://github.com/xoofx/Tomlyn https://github.com/xoofx/Tomlyn The code it needed to write was: var model = TomlSerializer.Deserialize<TomlTable>(toml)!; Which is in the readme of the repo. It could also have generated a class and deserialised into that. Instead it did something else (afraid I don’t have it handy sorry)
fxtentacle 7mo ago
Same here. While LLMs sometimes work surprisingly well, I also encounter edge cases where they fail surprisingly badly multiple times per day. My guess is that other people maybe just don't bother to check what the AI says which would cause them to not notify omission errors. Like when I was trying to find a physical store again with ChatGPT Pro 5.4 and asked it to prepare a list of candidates, but the shop just wasn't in the list, despite GPT claiming it to be exhaustive. When I then found it manually and asked GPT for advice on how I could improve my prompting in the future, it went full "aggressively agreeable" on me with "Excellent question! Now I can see exactly why my searches missed XY - this is a perfect learning opportunity. Here's what went wrong and what was missing: ..." and then 4 sections with 4 subsections each. It's great to see the AI reflect on how it failed. But it's also kind of painful if you know that it'll forget all of this the moment the text is sent to me and that it will never ever learn from this mistake and do better in the future.
- pdntspa 7mo ago
  [dead]
- jmalicki 7mo ago
  "I also encounter edge cases where they fail surprisingly badly multiple times per day. " If 80% of the time they 10x my output, and the other 20% I can say "well they failed, I guess this one I have to do manually" - that's still an absolutely massive productivity boost.
  - maccard 7mo ago
    But they don’t 10x my output - they write some code for a problem I/you have already thought about. The hard part isn’t writing the code, it never has been. It’s always been solving and breaking down the problem.
    - jmalicki 7mo ago
      It's not the hard part, just the tedious part that takes the most time.
foobarchu 7mo ago
The best experiences I have are those where I can describe what I want done with details. Rather than asking it add toml parsing, I would tell it to exactly which library to use ahead of time and reduce the number of decisions available to the model to make. Some of the most effective use-cases are when you have a reference to give it, e.g. "add x feature the same way as in this other project that is also in the workspace", or "make the changes I made to the contents of directory X in git commit <sha here>, but applied to directory Y instead". In both cases it's a lot of copy/paste then tweaking an obvious value (like replacing "dev" with "QA" everywhere). I try to give the model as little freedom as possible. That usually means it's not being used for novel work.
- alecbz 7mo ago
  > The best experiences I have are those where I can describe what I want done with details. But that's the hard part! You can only eke out moderate productivity gains by automating the tedium of actually writing out the code, because it's a small fraction of software engineering.
  - foobarchu 7mo ago
    That's why I don't like to claim massive productivity boosts, personally. It's helped me out with the tedious bits that are still necessary. It's also great as an idea board, where I ask for some sample approaches to a problem. That cuts way down on research time, even if a few of the options given are dead ends because e.g. they use an API that doesn't exist.
- senordevnyc 7mo ago
  I basically never have agents do work without using plan mode first (I use Cursor). So I'll start with a high-level like: "I want to plan out the architecture for feature X, I think there's a linear project already that we should update once our plan is done, but if not we'll need to create one. Here's a picture of my whiteboard, I think I also wrote some notes in Obsidian a few weeks ago that you should look up. Two things off the top of my head to think about are: should we do X or does it make more sense to do Y? And also, I'm worried about how this might interfere with the blahblah system, so can you do some research around that? <ramble ramble>" Then it crawls around for awhile, does some web searches, fetches docs from here and there, whatever. Sometimes it'll ask me some questions. And then it'll finally spit out a plan. I'll read through it and just give it a massive dump of issues, big and small, more questions I have, whatever. (I'll also often be spinning off new planning sessions for pre-work or ancillary tasks that I thought of while reviewing that plan). No structure or anything, just brain dump. Maybe two rounds of that, but usually just one. And then I'll either have it start building, or I'll have it stash in the linear agent so I can kick it off later.
ffsm8 7mo ago
> This morning I asked Claude to use a library to load a toml file in .net and print a value. Legit this morning Claude was essentially unusable for me I could explicitly state things it should adjust and it wouldn't do it. Not even after specifying again, reverting everything eventually and reprompt from the beginning etc. Even super trivial frontend things like "extract [code] into a separate component" After 30 minutes of that I relented and went on to read a book. After lunch I tried again and it's intelligence was back to normal It's so uncanny to experience how much ita performance changes - I strongly suspect anthropic is doing something whenever it's intelligence drops so much, esp. Because it's always temporary - but repeatable across sessions if occuring... Until it's normal again But ultimately just speculation, I'm just a user after all
- maccard 7mo ago
  > Legit this morning Claude was essentially unusable for me I could explicitly state things it should adjust and it wouldn't do it. Honestly, this is my experience. Every now and again it just completely self implodes and gives up, and I’m left to pick up the pieces. Look at the other replies who are making sure I’m using the agrntic loop/correct model/specific enough prompt - I don’t know what they’re doing but I would love to try the tools they’re using.
- krackers 7mo ago
  Here's an evil business idea: Use the LLMs to identify the users most likely to be "vocal influencers" and then prioritize resources for them, ensuring they get the best experience. You can engineer a bubble this way. And then the next step is to dynamically vary resources based on prediction of user stickiness. User is frustrated and thinking of trying competitor -> allocate full resources. User is profiled as prone to gambling and will tolerate intermittent rewards -> can safely forward requests to gimped models. User is an resolute AI skeptic and unlikely to ever preach the gospels of vibecoding -> no need to waste resources on him.
  - oro44 7mo ago
    "Here's an evil business idea: Use the LLMs to identify the users most likely to be "vocal influencers" and then prioritize resources for them, ensuring they get the best experience. You can engineer a bubble this way." Its quite likely this is already happening buddy... The 'random' degradation across all LLM-based services is obvious at this point.
ball_of_lint 7mo ago
Did you give claude access to run the compile step? I remember having to write code on paper for my CS exams, and they expected it to compile! It was hard but I mostly got there. definitely made a few small mistakes though
- maccard 7mo ago
  Yes.
linsomniac 7mo ago
My friend, with all due respect, I don't think this is a problem with the AI. I don't know anything about DotNet, but I just fired up Claude Code in an empty directory and asked it to create an example dotnet program using the Tomlyn library, it chugged away and ~5 minutes later I did a "grep Deserialize *" in the project and it came up with exactly the line you wanted (in your comment here) it to produce: var model = TomlSerializer.Deserialize<TomlTable>(tomlContent)!; The full results of what it produced are at https://github.com/linsomniac/tomlynexample https://github.com/linsomniac/tomlynexample That includes the prompt I used, which is: Please create a dotnet sample program that uses the library at https://github.com/xoofx/Tomlyn https://github.com/xoofx/Tomlyn to parse the TOML file given on the command line. Please only use the Tomlyn library for parsing the TOML file. I don't have any dotnet tooling installed on my system, please let me know what is needed to compile this example when we get there. Please use an agent team consisting of a dotnet expert, a qa expert, a TOML expert a devils advocate and a dotnet on Linux expert. I can't really comment on the code it produced (as I said, I don't use dotnet, I had to install dotnet on my system to try this), so I can't comment on the approach. 346 lines in Program.cs seems like a lot for an example TOML program, but I know Claude Code tends to do full error checking, etc, and it seems to have a lot of "pretty printing" code.
- maccard 7mo ago
  I'm in claude code right now, with opus 4.6. Here [0] is my prompt, plan and result. When I prompted, it went spelunking in the parent folder for a bunch of stuff, which I interrupted and told it to focus on its current folder. It came up with a sensible plan, which I let it execute. I then switched to a new tab and ran `dotnet run publishing/publish.cs` - like claude says. And the script fails at the first hurdle. The agentic loop came up with a plan, claimed it was done, but _didn't execute the verifiaction it told me it was going to execute_. [0] https://gist.github.com/donalmacc/e91955be9792a983c375a098ba1767db https://gist.github.com/donalmacc/e91955be9792a983c375a098ba...
  - wrs 7mo ago
    When CC says "Verification" at the end like that, it means it's assuming you will do the verification. But you can tell it to do it (just say "verify it yourself" or something), and then (if needed) ask it to fix the errors, etc. You could even say "verify and fix errors until it works".
deleted 7mo ago
[deleted]
senordevnyc 7mo ago
This is so perplexing to me. I've definitely hit these kinds of issues (which usually result in me cursing at the agent in all caps while telling it to get its shit together!), but it's almost always a long ways into a session where I know context rot is an issue, and the assumption it's making is a dumb one but it's also in the middle of a complex task...I just haven't had anything remotely like the situation you're describing, where Opus 4.6 can't make a simple change and verify that it compiles, can't look up docs, can't follow your instructions, etc. Bizarre.
- jmalicki 7mo ago
  It's been awhile, but I once spent four hours trying to get Opus 4.1 to change the color palette of a bar graph, that would've taken me 5 minutes. The only reason I wasted so much time was that given all of my other positive experiences, clearly I must have been prompting it incorrectly, and I should learn how to fix it. After blowing four hours I just changed the colors by hand. I took away that it's almost completely unpredictable what will be easy and what will be hard for an LLM.
prettyblocks 7mo ago
I think good results come from spending a substantial amount of time reading the claude-code documentation and setting up an environment that lets you work in this new paradigm. Asking it a few things, getting poor results, and deciding "it doesn't work when you ask it to correct itself won't get you that far.
AbanoubRodolf 7mo ago
[flagged]
the__alchemist 7mo ago
IMO you may be biting off on the hype comments and descriptions, and assuming too much of the LLM. (Easy to do if reading HN or internet articles) Approach the prompts concretely. You want it to work on x struct and y function in z module. You want it to follow the structure you already started working on. You want it to work like existing software Y, here's a link to its docs page on the topic. The result of computation A should be B, but it's currently showing as C. Then it will work great and save you time for many classes of problems.

by maccard 7mo ago

krastanov 7mo ago

This is fascinating to me. I completely believe you and I will not bother you with all the common "but did you try to tell it this or that" responses, but this is such a different experience from mine. I did the exact same task with claude in the Julia language last week, and everything worked perfectly. I am now in the habit of adding "keep it simple, use only public interfaces, do not use internals, be elegant and extremely minimal in your changes" to all my requests or SKILL.md or AGENTS.md files (because of the occasional failure like the one you described). But generally speaking, such complete failures have been so very rare for me, that it is amazing to see that others have had such a completely different experience.

shafyy 7mo ago
It's almost like.... LLMs are non-deterministic and hallucinating... Oh wait?!
- krastanov 7mo ago
  non-deterministic does not mean it is not biased towards a particular type of results (helpful results)
jayd16 7mo ago
You say it doesn't fail but you also mention all these work around you know and try...sounds like it fails a lot but your tolerance is different.
- zorak8me 7mo ago
  Providing instruction and context doesn’t seem like a “workaround”.
  - theshrike79 7mo ago
    Agents are LLMs that use tools in a loop I didn't give my Agent any tools, it hallucinated code. ...well, yes. It's like evaluating a programmer's skill by having one-shot a program on paper with zero syntax errors.
- plagiarist 7mo ago
  IME it does fail pretty hard at first. One has to build up a little library of markdown and internal library of prompt techniques. Then it starts working okay. I agree there is a hurdle still, trying it on one task doesn't really get one over the hurdle.
- jmalicki 7mo ago
  Most people I've seen complain say things like "I asked it for code and it didn't compile." The real magic of LLMs comes when they iterate until completion until the code compiles and the test passes, and you don't even bother looking at it until then. Each step is pretty stupid, but the ability to very quickly doggedly keep at it until success quite often produces great work. If you don't have linters that are checking for valid syntax and approved coding style, if you don't have tests to ensure the LLM doesn't screw up the code, you don't have good CI, you're going to have a bad time. LLMs are just like extremely bright but sloppy junior devs - if you think about putting the same guardrails in place for your project you would for that case, things tend to work very well - you're giving the LLM a chance to check its work and self correct. It's the agentic loop that makes it work, not the single-shot output of an LLM.
aqua_coder 7mo ago
My experience with working with AI agents is that they can be verbose and do things that are too over complicated by default. Unless directed explicitly. Which may be the reason for this discrepancy.
sigbottle 7mo ago
Idk. Had a friend recommend me gsdv2 and I wasted like 100$ + so much time trying to debug said crap. I went back to codex and it 1 shot my problems easily. And this was from two people who were 100% aligned on agentic AI coding. I've been using AI for years now and agentic AI for several months now. I was told that I bring out the worst in LLMs. Except... I was able to achieve better results, WITH LLMs, on OTHER frameworks. So like, ? It may be easier to draw the boundary between "AI and non AI users", but as AI becomes prolific, the us vs them angle that people keep using won't apply anymore. The age old "user versus tool" debate goes on, but it seems like gaslighting is popular these days. I classify it as gaslighting because I'm clearly a falsifiable test case, I'm even gung ho for LLMs, yet any kind of dissent is immediately warped into user error. It doesn't matter what you say or where you are on the spectrum, if you have 1 bad experience and speak up about it it's an issue. Guess that's not really an issue for the human condition though.

wrs 7mo ago

I’m honestly baffled by this. I don’t want to tell you “you’re holding it wrong” but if this is your normal experience there’s something weird happening. Friday afternoon I made a new directory and told Claude Code I wanted to make a Go proxy so I could have a request/callback HTTP API for a 3rd party service whose official API is only persistent websocket connections. I had it read the service’s API docs, engage in some back and forth to establish the architecture and library choices, and save out a phased implementation plan in plan mode. It implemented it in four phases with passing tests for each, then did live tests against the service in which it debugged its protocol mistakes using curl. Finally I had it do two rounds of code review with fresh context, and it fixed a race condition and made a few things cleaner. Total time, two hours. I have noticed some people I work with have more trouble, and my vague intuition is it happens when they give Claude too much autonomy. It works better when you tell it what to do, rather than letting it decide. That can be at a pretty high level, though. Basically reduce the problem to a set of well-established subproblems that it’s familiar with. Same as you’d do with a junior developer, really.

shireboy 7mo ago
Similar. I regularly use Github copilot (with claude models sometimes) and it works amazingly. But I see some who struggle with them. I have sort of learned to talk to it, understand what it is generating, and routinely use to generate fixes, whole features, etc. much much faster than I could before.
- maccard 7mo ago
  I’ve found copilot to generate worse code, and the editor integrations to be unusable (lets insert this block of code at the wrong place in the file for example), but overall for it to do the right thing enough that I can patch it. I’ve found Claude code just goes mad and gets stuck in a loop and I need to pull it out and figure out where in the hundreds of lines of code it’s generates in the last few minutes it’s gone so wrong.
thwarted 7mo ago
> it happens when they give Claude too much autonomy. It works better when you tell it what to do, rather than letting it decide. That can be at a pretty high level, though. Basically reduce the problem to a set of well-established subproblems that it’s familiar with. Same as you’d do with a junior developer, really. Equating "junior developers" and "coding LLMs" is pretty lame. You handhold a junior developers so, eventually, you don't have to handhold anymore. The junior developer is expected to learn enough, and be trusted enough, to operate more autonomously. "Junior developers" don't exist solely to do your bidding. It may be valuable to recognize similarities between a first junior developer interaction and a first LLM interaction, but when every LLM interaction requires it to be handheld, the value of the iterative nature of having a junior developer work along side you is not at all equivalent.

DougN7 7mo ago

I have similar experiences. It has worked about half the time, but the code has to be pretty simple. I’ve many experiences, where we work on something complicated for an hour, and it is good compiling code, but then an edge case comes to mind that I ask about and Claude tells me the whole approach is doomed and will never work for that case. It has even apologized a few times for misleading me :) I feel like it’s this weird mix of brilliant moron. But yeah, ask for a simple HTML page with a few fields and it rocks.

roncesvalles 7mo ago

Did you use the best model available to you (Opus 4.6)? There is a world of difference between using the highest model vs the fast one. The fast ones are basically useless and it's a shame that all these tools default to it.

fweimer 7mo ago
To make this more confusing, there is a fast mode of Opus 4.6, which (as far as I understand it) is supposed to deliver the same results as the standard mode. It's much more expensive, but advertised as more interactive.
moffkalast 7mo ago
Opus is not cost effective.
- deleted 7mo ago
  [deleted]
- senordevnyc 7mo ago
  Compared to what? I spent about $700 last month on Cursor, mostly on Opus 4.6. The time it saved me is at least 10-20x more valuable than that. I keep trying to use GPT-5.3 Codex, because it's like 40% cheaper, but then the quality doesn't always seem quite as good, and it's honestly not worth it to me to save a couple hundred bucks a month for something that's 10% worse. The multiplicative value of what I'm building means it's worth it to pay extra for the SOTA model in almost all cases.
- roncesvalles 7mo ago
  The cost-effectiveness of something that doesn't work is 0.

phromo 7mo ago

I don't know why but I find performance on c#/.net be several generations behind. Sometimes right ofc but my general experience is if you pull the generation slot machine in just about any other language it will work better. I regularly do python, typescript, ruby and rust with a better experience. It's even hard to find benchmarks where csharp is included.

JakeStone 7mo ago

If Claude ends up grabbing my C# TOML library, in my defense, I wrote it when the TOML format first came out over a dozen years ago, and never did anything more with it Sorry.

maccard 7mo ago
Nope! https://github.com/xoofx/Tomlyn https://github.com/xoofx/Tomlyn The code it needed to write was: var model = TomlSerializer.Deserialize<TomlTable>(toml)!; Which is in the readme of the repo. It could also have generated a class and deserialised into that. Instead it did something else (afraid I don’t have it handy sorry)

fxtentacle 7mo ago

Same here. While LLMs sometimes work surprisingly well, I also encounter edge cases where they fail surprisingly badly multiple times per day. My guess is that other people maybe just don't bother to check what the AI says which would cause them to not notify omission errors. Like when I was trying to find a physical store again with ChatGPT Pro 5.4 and asked it to prepare a list of candidates, but the shop just wasn't in the list, despite GPT claiming it to be exhaustive. When I then found it manually and asked GPT for advice on how I could improve my prompting in the future, it went full "aggressively agreeable" on me with "Excellent question! Now I can see exactly why my searches missed XY - this is a perfect learning opportunity. Here's what went wrong and what was missing: ..." and then 4 sections with 4 subsections each. It's great to see the AI reflect on how it failed. But it's also kind of painful if you know that it'll forget all of this the moment the text is sent to me and that it will never ever learn from this mistake and do better in the future.

pdntspa 7mo ago
[dead]
jmalicki 7mo ago
"I also encounter edge cases where they fail surprisingly badly multiple times per day. " If 80% of the time they 10x my output, and the other 20% I can say "well they failed, I guess this one I have to do manually" - that's still an absolutely massive productivity boost.
- maccard 7mo ago
  But they don’t 10x my output - they write some code for a problem I/you have already thought about. The hard part isn’t writing the code, it never has been. It’s always been solving and breaking down the problem.
  - jmalicki 7mo ago
    It's not the hard part, just the tedious part that takes the most time.

foobarchu 7mo ago

The best experiences I have are those where I can describe what I want done with details. Rather than asking it add toml parsing, I would tell it to exactly which library to use ahead of time and reduce the number of decisions available to the model to make. Some of the most effective use-cases are when you have a reference to give it, e.g. "add x feature the same way as in this other project that is also in the workspace", or "make the changes I made to the contents of directory X in git commit <sha here>, but applied to directory Y instead". In both cases it's a lot of copy/paste then tweaking an obvious value (like replacing "dev" with "QA" everywhere). I try to give the model as little freedom as possible. That usually means it's not being used for novel work.

alecbz 7mo ago
> The best experiences I have are those where I can describe what I want done with details. But that's the hard part! You can only eke out moderate productivity gains by automating the tedium of actually writing out the code, because it's a small fraction of software engineering.
- foobarchu 7mo ago
  That's why I don't like to claim massive productivity boosts, personally. It's helped me out with the tedious bits that are still necessary. It's also great as an idea board, where I ask for some sample approaches to a problem. That cuts way down on research time, even if a few of the options given are dead ends because e.g. they use an API that doesn't exist.
senordevnyc 7mo ago
I basically never have agents do work without using plan mode first (I use Cursor). So I'll start with a high-level like: "I want to plan out the architecture for feature X, I think there's a linear project already that we should update once our plan is done, but if not we'll need to create one. Here's a picture of my whiteboard, I think I also wrote some notes in Obsidian a few weeks ago that you should look up. Two things off the top of my head to think about are: should we do X or does it make more sense to do Y? And also, I'm worried about how this might interfere with the blahblah system, so can you do some research around that? <ramble ramble>" Then it crawls around for awhile, does some web searches, fetches docs from here and there, whatever. Sometimes it'll ask me some questions. And then it'll finally spit out a plan. I'll read through it and just give it a massive dump of issues, big and small, more questions I have, whatever. (I'll also often be spinning off new planning sessions for pre-work or ancillary tasks that I thought of while reviewing that plan). No structure or anything, just brain dump. Maybe two rounds of that, but usually just one. And then I'll either have it start building, or I'll have it stash in the linear agent so I can kick it off later.

ffsm8 7mo ago

> This morning I asked Claude to use a library to load a toml file in .net and print a value. Legit this morning Claude was essentially unusable for me I could explicitly state things it should adjust and it wouldn't do it. Not even after specifying again, reverting everything eventually and reprompt from the beginning etc. Even super trivial frontend things like "extract [code] into a separate component" After 30 minutes of that I relented and went on to read a book. After lunch I tried again and it's intelligence was back to normal It's so uncanny to experience how much ita performance changes - I strongly suspect anthropic is doing something whenever it's intelligence drops so much, esp. Because it's always temporary - but repeatable across sessions if occuring... Until it's normal again But ultimately just speculation, I'm just a user after all

maccard 7mo ago
> Legit this morning Claude was essentially unusable for me I could explicitly state things it should adjust and it wouldn't do it. Honestly, this is my experience. Every now and again it just completely self implodes and gives up, and I’m left to pick up the pieces. Look at the other replies who are making sure I’m using the agrntic loop/correct model/specific enough prompt - I don’t know what they’re doing but I would love to try the tools they’re using.
krackers 7mo ago
Here's an evil business idea: Use the LLMs to identify the users most likely to be "vocal influencers" and then prioritize resources for them, ensuring they get the best experience. You can engineer a bubble this way. And then the next step is to dynamically vary resources based on prediction of user stickiness. User is frustrated and thinking of trying competitor -> allocate full resources. User is profiled as prone to gambling and will tolerate intermittent rewards -> can safely forward requests to gimped models. User is an resolute AI skeptic and unlikely to ever preach the gospels of vibecoding -> no need to waste resources on him.
- oro44 7mo ago
  "Here's an evil business idea: Use the LLMs to identify the users most likely to be "vocal influencers" and then prioritize resources for them, ensuring they get the best experience. You can engineer a bubble this way." Its quite likely this is already happening buddy... The 'random' degradation across all LLM-based services is obvious at this point.

ball_of_lint 7mo ago

Did you give claude access to run the compile step? I remember having to write code on paper for my CS exams, and they expected it to compile! It was hard but I mostly got there. definitely made a few small mistakes though

maccard 7mo ago
Yes.

linsomniac 7mo ago

My friend, with all due respect, I don't think this is a problem with the AI. I don't know anything about DotNet, but I just fired up Claude Code in an empty directory and asked it to create an example dotnet program using the Tomlyn library, it chugged away and ~5 minutes later I did a "grep Deserialize *" in the project and it came up with exactly the line you wanted (in your comment here) it to produce: var model = TomlSerializer.Deserialize<TomlTable>(tomlContent)!; The full results of what it produced are at https://github.com/linsomniac/tomlynexample https://github.com/linsomniac/tomlynexample That includes the prompt I used, which is: Please create a dotnet sample program that uses the library at https://github.com/xoofx/Tomlyn https://github.com/xoofx/Tomlyn to parse the TOML file given on the command line. Please only use the Tomlyn library for parsing the TOML file. I don't have any dotnet tooling installed on my system, please let me know what is needed to compile this example when we get there. Please use an agent team consisting of a dotnet expert, a qa expert, a TOML expert a devils advocate and a dotnet on Linux expert. I can't really comment on the code it produced (as I said, I don't use dotnet, I had to install dotnet on my system to try this), so I can't comment on the approach. 346 lines in Program.cs seems like a lot for an example TOML program, but I know Claude Code tends to do full error checking, etc, and it seems to have a lot of "pretty printing" code.

maccard 7mo ago
I'm in claude code right now, with opus 4.6. Here [0] is my prompt, plan and result. When I prompted, it went spelunking in the parent folder for a bunch of stuff, which I interrupted and told it to focus on its current folder. It came up with a sensible plan, which I let it execute. I then switched to a new tab and ran `dotnet run publishing/publish.cs` - like claude says. And the script fails at the first hurdle. The agentic loop came up with a plan, claimed it was done, but _didn't execute the verifiaction it told me it was going to execute_. [0] https://gist.github.com/donalmacc/e91955be9792a983c375a098ba1767db https://gist.github.com/donalmacc/e91955be9792a983c375a098ba...
- wrs 7mo ago
  When CC says "Verification" at the end like that, it means it's assuming you will do the verification. But you can tell it to do it (just say "verify it yourself" or something), and then (if needed) ask it to fix the errors, etc. You could even say "verify and fix errors until it works".

deleted 7mo ago

[deleted]

senordevnyc 7mo ago

This is so perplexing to me. I've definitely hit these kinds of issues (which usually result in me cursing at the agent in all caps while telling it to get its shit together!), but it's almost always a long ways into a session where I know context rot is an issue, and the assumption it's making is a dumb one but it's also in the middle of a complex task...I just haven't had anything remotely like the situation you're describing, where Opus 4.6 can't make a simple change and verify that it compiles, can't look up docs, can't follow your instructions, etc. Bizarre.

jmalicki 7mo ago
It's been awhile, but I once spent four hours trying to get Opus 4.1 to change the color palette of a bar graph, that would've taken me 5 minutes. The only reason I wasted so much time was that given all of my other positive experiences, clearly I must have been prompting it incorrectly, and I should learn how to fix it. After blowing four hours I just changed the colors by hand. I took away that it's almost completely unpredictable what will be easy and what will be hard for an LLM.

prettyblocks 7mo ago

I think good results come from spending a substantial amount of time reading the claude-code documentation and setting up an environment that lets you work in this new paradigm. Asking it a few things, getting poor results, and deciding "it doesn't work when you ask it to correct itself won't get you that far.

AbanoubRodolf 7mo ago

[flagged]

the__alchemist 7mo ago

IMO you may be biting off on the hype comments and descriptions, and assuming too much of the LLM. (Easy to do if reading HN or internet articles) Approach the prompts concretely. You want it to work on x struct and y function in z module. You want it to follow the structure you already started working on. You want it to work like existing software Y, here's a link to its docs page on the topic. The result of computation A should be B, but it's currently showing as C. Then it will work great and save you time for many classes of problems.