4 ms·

I spent considerable time trying to coax the agentic systems into decent coding capabilities. The thing that struck me most is how creative they are at finding

I spent considerable time trying to coax the agentic systems into decent coding capabilities. The thing that struck me most is how creative they are at finding new ways to fail and make me adjust my prompt. It got tiring, so I'm on a break from ai coding until I have bandwidth to build my own agent. I don't think this is something we should be outsourcing to the likes of OpenAI, Microsoft, Anthropic, Google, Cursor, et al. Big Tech has shown their priorities lie elsewhere from our success and well being

LeafItAlone 1y ago
>The thing that struck me most is how creative they are at finding new ways to fail Wow, they are really going for that human-like behavior aren’t they?
- verdverm 1y ago
  If we're talking about emulating users, sure, but this is supposed to be a tool that helps me get my job done. If (i.e.) you dig into how something like copilot works, they do dumb things like ask^ the LLM to do glob matching after a file read (to pull in more instructions)... just use a damn glob library instead of a non-deterministic and known to be unreliable method ^ it's just a table in the overall context, so "asking" is a bit anthropomorphizing
  - cluckindan 1y ago
    I would consider a bunch of ”dumb/power user” agents more useful than coding agents. The more they fail to use my software, the better!
  - zahlman 1y ago
    > ^ it's just a table in the overall context, so "asking" is a bit anthropomorphizing I interpreted GP as just saying that you are already anthropomorphizing too much by supposing that the models "find" new ways to fail (as if trying to defy you).
    - verdverm 1y ago
      most humans do not seek out ways to defy after a certain age I did not mean to imply active choice by "find", more that they are reliably non-deterministic and have a hard time sticking to, or easy time ignoring, the instructions I did write
prmph 1y ago
Exactly my experience too. I'm now using AI like 25% of the time or less. I always get to a point where I see that using agentic coding is making me not want to actually think, there's no way anyone can convince me that that is a superior approach, because every time I took days off the agents to actually think, I came up with a far superior architecture and code that even rendered much of what the agents were hammering away at moot. Agentic coding is like a drug or slot machine, it slowly draws you in with the implicit promise of getting much for little. The only ways it is useful to me now is for very focused tasks where I have spent a lot of time defining the architecture down to the last detail, and the agents are used to fill in the blanks, as it were. I also think I could write a better agent, and as to why the bog corps have not done so is baffling to me. Just event getting the current agents to obey the guidelines in the agent .md files is a struggle. They forget pretty much everything two prompts down the line. Why can't the CLI systemically prompt them to check every time, etc.? Something tell me the future is about domain-aware agents that help users to wring better performance out of the models, based on some domain-specific deterministic guardrails.
- csallen 1y ago
  I've had experiences like this before but if that's the ONLY experience you've had, or if you have that experience 75% of the way, I think you're doing something wrong. Or perhaps you're just in a very different coding domain than I am (web dev, HTML/CSS/JS) where the AI happens to suck. The biggest mistakes imo are: 1. Underplanning. Trying to do huge projects in one go, rather than breaking them down into small projects, and breaking those small projects down into well thought out plans. 2. Too much of a focus on prompting rather than context. Prompt engineering is obsessing with the perfect way to say or phrase something. Whereas context engineering putting relevant information into the LLM's working memory, which requires you to go out and gather that info (or use the LLM to get it).
  - verdverm 1y ago
    I've had my share of good and bad experiences, one section of an existing project more than 90% ai created. How you say things is equally important to the context you provide, partly because the agents will start trying to decide what is and is not good context, which they are unreliable in doing, even after you give them the limited context and tell them not to edit other files or bring in more context. For example, if you use a lot of colloquial phrases, you activate that area of the network, taking away from using other parts (MoE activation, also lower level too) They are not good readers (see research results around context collapse and context poisoning)

LeafItAlone 1y ago

>The thing that struck me most is how creative they are at finding new ways to fail Wow, they are really going for that human-like behavior aren’t they?

verdverm 1y ago
If we're talking about emulating users, sure, but this is supposed to be a tool that helps me get my job done. If (i.e.) you dig into how something like copilot works, they do dumb things like ask^ the LLM to do glob matching after a file read (to pull in more instructions)... just use a damn glob library instead of a non-deterministic and known to be unreliable method ^ it's just a table in the overall context, so "asking" is a bit anthropomorphizing
- cluckindan 1y ago
  I would consider a bunch of ”dumb/power user” agents more useful than coding agents. The more they fail to use my software, the better!
- zahlman 1y ago
  > ^ it's just a table in the overall context, so "asking" is a bit anthropomorphizing I interpreted GP as just saying that you are already anthropomorphizing too much by supposing that the models "find" new ways to fail (as if trying to defy you).
  - verdverm 1y ago
    most humans do not seek out ways to defy after a certain age I did not mean to imply active choice by "find", more that they are reliably non-deterministic and have a hard time sticking to, or easy time ignoring, the instructions I did write

prmph 1y ago

Exactly my experience too. I'm now using AI like 25% of the time or less. I always get to a point where I see that using agentic coding is making me not want to actually think, there's no way anyone can convince me that that is a superior approach, because every time I took days off the agents to actually think, I came up with a far superior architecture and code that even rendered much of what the agents were hammering away at moot. Agentic coding is like a drug or slot machine, it slowly draws you in with the implicit promise of getting much for little. The only ways it is useful to me now is for very focused tasks where I have spent a lot of time defining the architecture down to the last detail, and the agents are used to fill in the blanks, as it were. I also think I could write a better agent, and as to why the bog corps have not done so is baffling to me. Just event getting the current agents to obey the guidelines in the agent .md files is a struggle. They forget pretty much everything two prompts down the line. Why can't the CLI systemically prompt them to check every time, etc.? Something tell me the future is about domain-aware agents that help users to wring better performance out of the models, based on some domain-specific deterministic guardrails.

csallen 1y ago
I've had experiences like this before but if that's the ONLY experience you've had, or if you have that experience 75% of the way, I think you're doing something wrong. Or perhaps you're just in a very different coding domain than I am (web dev, HTML/CSS/JS) where the AI happens to suck. The biggest mistakes imo are: 1. Underplanning. Trying to do huge projects in one go, rather than breaking them down into small projects, and breaking those small projects down into well thought out plans. 2. Too much of a focus on prompting rather than context. Prompt engineering is obsessing with the perfect way to say or phrase something. Whereas context engineering putting relevant information into the LLM's working memory, which requires you to go out and gather that info (or use the LLM to get it).
- verdverm 1y ago
  I've had my share of good and bad experiences, one section of an existing project more than 90% ai created. How you say things is equally important to the context you provide, partly because the agents will start trying to decide what is and is not good context, which they are unreliable in doing, even after you give them the limited context and tell them not to edit other files or bring in more context. For example, if you use a lot of colloquial phrases, you activate that area of the network, taking away from using other parts (MoE activation, also lower level too) They are not good readers (see research results around context collapse and context poisoning)