9 ms·

The quality of AI-assisted software depends on unit of work management

170 points by mogambo1 1y ago

datadrivenangel 1y ago
Keep your scope as small as necessary, but no smaller. This has been fundamentally true for project management work breakdown structures for decades.
- BinaryIgor 1y ago
  Interesting that it turns out to be true for code generation as well!
- igouy 1y ago
  But not actionable?
liszper 1y ago
most SWE folks still have no idea how big the difference is between the coding agents they tried a year ago and declared as useless and chatgpt 5 paired with Codex or Cursor today thanks for the article, it's a good one
- TheRoque 1y ago
  It's true that I haven't been a hardcore agent-army vibe coder, I just try the popular ones once in a while in a naive way (isn't it the point of these tools, to have little friction ?), claude code for example. And it's cool ! But imperfect, and as this article attests, there's a lot of mental overhead to even have a shot at getting a decent output. And even if it's decent, it still needs to be reviewed and could include logical flaws. I'd rather use it the other way, I'm the one in charge, and the AI reviews any logical flaw or things that I would have missed. I don't even have to think about context window since it'll only look at my new code logic. So yeah, 3 years after the first ChatGPT and Copilot, I don't feel huge changes regarding "automated" AI programming, and I don't have any AI tool in my IDE, I pefer to have a chat using their website, to brainstorm, or occasionally find a solution to something I'm stuck on.
- blibble 1y ago
  > most SWE folks still have no idea how big the difference is between the coding agents they tried a year ago and declared as useless and chatgpt 5 paired with Codex or Cursor today yes, just as was said each and every previous time OpenAI/anthropic shit out a new model "now it doesn't suck!"
  - Filligree 1y ago
    Each and every new model expands the scope of what you can do. You notice that, get elated when things that didn’t work start working, then three weeks later the honeymoon period is over and you notice the remaining limits. The hedonic treadmill ensures it feels the same way each time. But that doesn’t mean the models aren’t improving, nor that the scope isn’t expanding. If you compare today’s tools to those a year ago, the difference is stark.
  - thrawa8387336 1y ago
    She is choosing GPT5 as the good example? Maybe Claude, maybe..
- zeroonetwothree 1y ago
  I use agents for coding small stuff at work almost every day. I would say there has been some improvement compared to a year ago but it’s not any sort of step change. They still are only able to complete simple “intern-level” tasks around 50% of the time. Which is helpful but not revolutionary.
- angusturner 1y ago
  I think most SWEs do have a good idea where I work. They know that its a significant, but not revolutionary improvement. If you supervise and manage your agents closely on well scoped (small) tasks they are pretty handy. If you need a prototype and don't care about code quality or maintenance, they are great. Anyone claiming 2x, 5x, 10x etc is absolutely kidding themselves for any non-trivial software.
  - liszper 1y ago
    I'd argue this just proves my point.
  - bluefirebrand 1y ago
    > If you supervise and manage your agents closely on well scoped (small) tasks they are pretty handy Compared to just doing it yourself though? Imagine having to micromanage a junior developer like this to get good results Ridiculous tbh
  - dingnuts 1y ago
    if the benefit is less than 2x then we're talking about AI assisted coding as being a very, very expensive IntelliSense. 1.x improvement just isn't much. My mind goes back to that study showing engineers claimed a 20% improvement and measured 20% reduction in productivity -- this is all encouraging me to just keep using traditional tools.
    - rmunn 1y ago
      The only AI-assisted software work I've seen actually have a benefit is the way my coworker use Supermaven, where it's basically Intellisense but suggesting filling in the function parameters for you as well. He'll type `MergeEx` and it will not just suggest `MergeExample(` as Intellisense would have done, but also suggest `MergeExample(oldExample, newExample, mergeOptions)` based on the variable names in scope at the moment and which ones line up with the types. Then he presses Tab and moves on, saving 10-15 seconds of typing. Repeat that multiple times through the day and it might be a 10% improvement, with no time lost on fiddling with prompts to get the AI to correct its mistakes. (Here, if the suggestion is wrong, he just ignores it and keeps typing, and the second he types a character that wasn't the next one in the suggestion it goes away and a new suggestion might be calculated, but the cognitive load in ignoring the incorrect suggestion is minimal).
- kibwen 1y ago
  Last week I wanted to generate some test data for some unit tests for a certain function in a C codebase. It's an audio codec library, so I could have modified the function to dump its inputs to disk and then run the library on any audio file and then hardcoded the input into the unit tests. Instead, I decided I wanted to save a few bytes and wanted to look at generating dummy data dynamically. I wanted to try out Claude for generating the code that would generate the data, so to keep the context manageable I extracted the function and all its dependencies into a self-contained C program (less than 200 lines altogether) and asked it to write a function that would generate dummy data, in C. Impressively, it recognized the structure of the code and correctly identified it as a component of an audio codec library, and provided a reasonably complete description of many minute details specific to this codec and the work that the function was doing. Rather less impressively, it decided to ignore my request and write a function that used C++ features throughout, such as type inference and lambdas, or should I say "lambdas" because it was actually just a function-defined-within-a-function that tried to access and mutate variables outside of its own function scope, like we were writing Javascript or something. Even apart from that, the code was rife with the sorts of warnings that even a default invocation of gcc would flag. I can see why people would be wowed by this on its face. I wouldn't expect any average developer to have such a depth of knowledge and breadth of pattern-matching ability to be able to identify the specific task that this specific function in this specific audio codec was performing. At the same time, this is clearly not a tool that's suitable for letting loose on a codebase without EXTREME supervision. This was a fresh session (no prior context to confuse it) using a tightly crafted prompt (a small, self-contained C program doing one thing) with a clear goal, and it still required constant handholding. At the end of the day, I got the code working by editing it manually, but in an honest retrospective I would have to admit that the overall process actually didn't save me any time at all. Ironically, despite how they're sold, these tools are infinitely better at going from code to English than going the other way around.
  - angusturner 1y ago
    I feel this. I've had a few tasks now where in honest retrospect I find myself asking "did that really speed me up". Its a bit demoralising cause not only do you waste time, you have a worse mental model of the resulting code and feel less sense of ownership over the result. Brainstorming, ideation and small, well defined tasks where I can quickly vet the solution : these feel like the sweet spot for current frontier model capabilities. (Unless you are pumping out some sloppy React SPA that you don't care about anything except get it working as fast as possible - fine, get Claude code to one shot it)
- rco8786 1y ago
  I still use Claude Code and Cursor and tbh still run into a lot of the same issues. Hallucinating code, hallucinating requirements, even when scoped to a very simple "make this small change". It's good enough that it helps, particularly in areas or languages that I'm unfamiliar with. But I'm constantly fighting with it.
- realusername 1y ago
  I tried again recently and I see absolutely no difference. If there's been some improvement, it's very subtle. There's a big difference with their benchmarks and real world coding.
jonstewart 1y ago
I first tried getting specific with Claude Code. I made the Claude.md, I detailed how to do TDD, what steps it should take, the commands it should run. It was imperfect. Then I had it plan (think hard) and write the plan to a file. I’d clear context, have it read the plan, ask me questions, and then have it decompose the plan into a detailed plan of discrete tasks. Have it work its way through that. It would inevitably go sideways halfway through, even clearing context between each task. It wouldn’t run tests, it would commit breakage, it would flip flop between two different broken approaches, it was just awful. Now I’ve just been vibing, writing as little as possible and seeing what happens. That sucks, too. It’s amazing at reviewing code. It will identify what you fear, the horrors that lie within the codebase, and it’ll bring them out into the sunlight and give you a 7 step plan for fixing them. And the coding model is good, it can write a function. But it can’t follow a plan worth shit. And if I have to be extremely detailed at the function by function level, then I should be in the editor coding. Claude code is an amazing niche tool for code reviews and dialogue and debugging and coping with new technologies and tools, but it is not a productivity enhancement for daily coding.
- liszper 1y ago
  With all due respect, you sound like someone who is just getting familiar with these tools. 100 more hours spent with AI coding and you will be much more productive. Coding with AI is a slightly different skill from coding, similar how managing software engineers is different from writing software.
  - TheRoque 1y ago
    Then, it's the job of someone else to use these tools, not developers
    - liszper 1y ago
      I agree with your point. I think this is the reason why most developers still don't get it, because AI coding ultimately requires a "higher level" methodology.
xmpir 1y ago
Same as for human software engineers... We'll see Conway' law all again with agentic coding!
- bryanrasmussen 1y ago
  maybe it just works that way for Agents because they see in the data it works that way for humans.
tedggh 1y ago
I found out that summarizing a completed task and feeding it to a new context works better than staying on the same context for multiple tasks. So let’s say I have a sprint with tasks 1, 2 and 3. I start by creating a project with general information including the spec, git issues, code base, folder trees, etc then work on Task 1. When done I ask for a summary using a template, which gives me a txt file describing what the original goal was, what we changed and what the next steps are. Then I repeat the process for Task 2 and I feed the summary from Task 1. At least in ChatGPT keeping the same context for multiple tasks has lots of issues like speed, increased hallucinations, and ChatGPT referencing content from old files.
- smw 1y ago
  Hell, claude even makes that part of the standard workflow, with /compact; cleverly using the llm itself to summarize the previous context
  - furyofantares 1y ago
    /compact is really poor imo. Quality just falls off a ledge after it. I much prefer to choose tasks that can be done with 25%+ context left and then just start the next task with fresh context. If I'm getting low on context I have it summarize the plan and progress in a text file rather than use /compact and then start a fresh context and reference that file, which I can then edit and try again if I'm not getting good results.
  - bryanlarsen 1y ago
    Once you see that message it's time to finish the task without AI because Claude will start crapping over your codebase if you let it continue.
    - ewoodrich 1y ago
      I just got burned by that, it compacted and then immediately dropped what it was in the middle of working on to redo something it had already finished half an hour earlier. Which, predictably, sent it into "systematically destroy the entire working codebase" mode because the code it was now reading didn't match expectations of the original instructions. So it started extrapolating like "huh, it looks like the function already exists, therefore, user must have meant [bizarre completely out of left field guess] instead" in an escalating loop of confusion and code mangling.
OptionOfT 1y ago
And the value of AI as pushed to us by these companies is in doing larger units of work. But... reviewing code is harder than writing code. Expressing how I want something to be done in natural language is incredibly hard. So over time I'm spending a lot of energy in those things, and only getting it 80% right. Not to mention I'm constantly in this highly suspicious mode, trying to pierce through the veil of my own prompt and the code generated, because it's the edge cases that make work hard. The end result is exhaustion. There is no recharge. Plans are front-loaded, and then you switch to auditing mode. Whereas with code you front-load a good amount of design, but you can make changes as you go, and since you know your own code the effort to make those are much lower.
- palmotea 1y ago
  That makes very clear these tools are not meant to serve you, you are meant to serve these tools.
  - ryoshu 1y ago
    Reverse centaur.
  - fragmede 1y ago
    What is this, that episode of The Twilight Zone called "to serve man"?
- nicce 1y ago
  > Expressing how I want something to be done in natural language is incredibly hard Surprise, surprise… that is why programming languages were created.
  - dragonwriter 1y ago
    Programming languages don’t solve that problem, since someone still has to explain what needs to be done in natural language unless the end customer is also the programmer. Programming languages were created because of the different problem of “its very hard to get computers to understand natural language even if you know how to express what you want in it”.
elpakal 1y ago
I agree with the premise of the article, and have felt that we're probably seeing AI code gen tools being limited by the constraints being put on them by traditional source code management tools like git and GitHub. Those tools were designed for incremental changes (patches), and have worked well for humans to organize changes so they could be more easily reviewed, maintained and reasoned about. Units of work in the form of features, patches etc rely on "pull requests" which are a function of the above.
marstall 1y ago
doing things in small chunks is good. so is it doing things in large chunks sometimes. In AI, like in life, there are no hard and fast rules and we're all figuring it out as we go. Like with "vibe coding" - sometimes it's ok to not even look at the code AI is generated, sometimes you need to understand every line. It feels like part of my journey to being an "AI developer" is being present for those tradeoffs, metabolizing each one into my craft. AI is a fickle, but powerful horse. I'm finding it a privilege to learn how to be a rider.
danparsonson 1y ago
I seem to be in a minority but I find user stories or features to be really awkward and unnatural units of work for building software. Sure these things help to define the expected result but they shouldn't directly drive the development process. Imagine building a house that way - you don't build the living room, then the kitchen, then the bathroom etc.; you build floors, walls, the roof... The 'features' or use cases for the building arise out of the combination of different elements that were put into it, and usually right near the end of the build. The same is true for basically anything else that we build or create - if you're making a sculpture, do you finish working on one leg first before you move onto some other part? Features are vertical slices through the software cake, but the cake is actually made out of horizontal layers. Creating a bunch of servings of cake and then trying to stick them together just results in a fragile mess that's difficult to work with and easy to break.
- nemomarx 1y ago
  They make more sense if you think about adding to an existing house. "I want to open up this wall to serve X function" kinda work they're very well adapted to legacy enterprise work
  - gizmo686 1y ago
    Even there, "open up this wall" is not a unit of work. You need to: * Evaluate what, if any, structural implications removing the wall has * Tear down the existing wall * Redo any plumbing, ductwork, wiring, etc that was hiding in the wall * Remediate structural concerns from removing the wall. * Redo the flooring * Repair and repaint any damage done to remaining drywall If this is part of a larger renovation, you will likely schedule work so the above tasks happen at the same time as other similar tasks. E.g. A meaningful unit of work might be "electrical roughing", which would include both moving wires that were previously in the wall, and running a new circuit to the garage for a car charger. No user story covers those to tasks, but the nature of renovating a house means that it makes sense to do them together.
- galbar 1y ago
  My take on this is that, from a SW development POV, user stories are not the right unit of work. Instead, I treat user stories as "Epics". Stake holders can track that Epic for progress, as the unit of work from their POV. Internally, the team splits Epics into "Spikes" (figure out what to do) and "Tasks" (executing on the things we need to do). - Spikes are scoped to up to 3 days and their outcome is usually a doc and either a follow-up Spike or Tasks to execute. - Tasks must be as small and unambiguous as possible (within reason).
jaaron 1y ago
Agreed that picking the right size of work is critical. I didn't know about Kiro specs. I've been playing around with my own org-mode based approach with mixed success in keeping dev agent work tracked: https://github.com/farra/dev-agent-work https://github.com/farra/dev-agent-work
stpedgwdgfhgdd 1y ago
Test driven development, there is nothing more to say for the coming year
rglover 1y ago
What I've found works best: 1. Assume that any model will start to lose focus beyond 50K-100K tokens (even with a huge context window). 2. Be gluttonous with chats. At the first sign of confusion or mistakes, tell it to generate a new prompt and move to a new chat. 3. Write detailed prompts with clear expectations (from how the code should be written to the specific implementation that's required). Combine these with context like docs to get a fairly consistent hit rate. 4. Use tools like Cline that let you switch between an "Act" and "Plan" mode. This saves a ton of tokens but also avoids the LLM getting stuck on a loop when it's debugging. I recently wrote this short blog post related to this: https://ryanglover.net/blog/treat-the-ai-like-it-s-yourself https://ryanglover.net/blog/treat-the-ai-like-it-s-yourself The above approach helped me to implement a full-blown database wrapper around LMDB for Node.js in ~2 weeks of slow back-and-forth (link to code in post for those who are curious).
chrisrickard 1y ago
We are tackling this at https://userdoc.fyi https://userdoc.fyi - we help you build your specs (epics, stories, acceptance criteria, tech notes, test cases, etc) - then you can generate what we call Dev Plans, one or more requirement layers for implementation. e.g maybe a dev plan is all your authentication feature requirements, or in the house of analogy – all the requirements for the rooms, but with instructions to actually just first build the floor, and the walls. Dev plans then slice the reqs into meaningful units of work, as mentioned in the article – a feature/story, is often too large of a checkpoint, or often needs to be implemented in collaboration with other features/stories, so it understands the correct architectural context,. You can then implement Dev plans over MCP, or copy to .md for tools like Lovable or V0.
zmmmmm 1y ago
I prefer small units of work. It really surprises me how fast people have leapt from the 2x speedup (mind blowing level of productivity increase) to full agentic coding without really questioning if it's a good idea. When I have let Claude loose and vibe coded up hundreds of lines at a time that I have no familiarity with, I viscerally feel how I no longer understand or can maintain the app I've built. If I can't get Claude to do the next change I need, I'm screwed. I'm very satisfied at the moment to be wielding LLMs as a tool at the individual function / microfeature level and getting a very satisfying productivity improvement.
deleted 1y ago
[deleted]
trash_cat 1y ago
>> "Turns out the major bottleneck is not intelligence, but rather providing the correct context." But this has more or less always been the case for LLMs. The challenge becomes context capure. Which in my opinion is the real challenge with LLM adoption. Without the right contex, some tasks just cannot be reliably completed.

170 points by mogambo1 1y ago

Keep your scope as small as necessary, but no smaller. This has been fundamentally true for project management work breakdown structures for decades.

BinaryIgor 1y ago
Interesting that it turns out to be true for code generation as well!
igouy 1y ago
But not actionable?

liszper 1y ago

most SWE folks still have no idea how big the difference is between the coding agents they tried a year ago and declared as useless and chatgpt 5 paired with Codex or Cursor today thanks for the article, it's a good one

TheRoque 1y ago
It's true that I haven't been a hardcore agent-army vibe coder, I just try the popular ones once in a while in a naive way (isn't it the point of these tools, to have little friction ?), claude code for example. And it's cool ! But imperfect, and as this article attests, there's a lot of mental overhead to even have a shot at getting a decent output. And even if it's decent, it still needs to be reviewed and could include logical flaws. I'd rather use it the other way, I'm the one in charge, and the AI reviews any logical flaw or things that I would have missed. I don't even have to think about context window since it'll only look at my new code logic. So yeah, 3 years after the first ChatGPT and Copilot, I don't feel huge changes regarding "automated" AI programming, and I don't have any AI tool in my IDE, I pefer to have a chat using their website, to brainstorm, or occasionally find a solution to something I'm stuck on.
blibble 1y ago
> most SWE folks still have no idea how big the difference is between the coding agents they tried a year ago and declared as useless and chatgpt 5 paired with Codex or Cursor today yes, just as was said each and every previous time OpenAI/anthropic shit out a new model "now it doesn't suck!"
- Filligree 1y ago
  Each and every new model expands the scope of what you can do. You notice that, get elated when things that didn’t work start working, then three weeks later the honeymoon period is over and you notice the remaining limits. The hedonic treadmill ensures it feels the same way each time. But that doesn’t mean the models aren’t improving, nor that the scope isn’t expanding. If you compare today’s tools to those a year ago, the difference is stark.
- thrawa8387336 1y ago
  She is choosing GPT5 as the good example? Maybe Claude, maybe..
zeroonetwothree 1y ago
I use agents for coding small stuff at work almost every day. I would say there has been some improvement compared to a year ago but it’s not any sort of step change. They still are only able to complete simple “intern-level” tasks around 50% of the time. Which is helpful but not revolutionary.
angusturner 1y ago
I think most SWEs do have a good idea where I work. They know that its a significant, but not revolutionary improvement. If you supervise and manage your agents closely on well scoped (small) tasks they are pretty handy. If you need a prototype and don't care about code quality or maintenance, they are great. Anyone claiming 2x, 5x, 10x etc is absolutely kidding themselves for any non-trivial software.
- liszper 1y ago
  I'd argue this just proves my point.
- bluefirebrand 1y ago
  > If you supervise and manage your agents closely on well scoped (small) tasks they are pretty handy Compared to just doing it yourself though? Imagine having to micromanage a junior developer like this to get good results Ridiculous tbh
- dingnuts 1y ago
  if the benefit is less than 2x then we're talking about AI assisted coding as being a very, very expensive IntelliSense. 1.x improvement just isn't much. My mind goes back to that study showing engineers claimed a 20% improvement and measured 20% reduction in productivity -- this is all encouraging me to just keep using traditional tools.
  - rmunn 1y ago
    The only AI-assisted software work I've seen actually have a benefit is the way my coworker use Supermaven, where it's basically Intellisense but suggesting filling in the function parameters for you as well. He'll type `MergeEx` and it will not just suggest `MergeExample(` as Intellisense would have done, but also suggest `MergeExample(oldExample, newExample, mergeOptions)` based on the variable names in scope at the moment and which ones line up with the types. Then he presses Tab and moves on, saving 10-15 seconds of typing. Repeat that multiple times through the day and it might be a 10% improvement, with no time lost on fiddling with prompts to get the AI to correct its mistakes. (Here, if the suggestion is wrong, he just ignores it and keeps typing, and the second he types a character that wasn't the next one in the suggestion it goes away and a new suggestion might be calculated, but the cognitive load in ignoring the incorrect suggestion is minimal).
kibwen 1y ago
Last week I wanted to generate some test data for some unit tests for a certain function in a C codebase. It's an audio codec library, so I could have modified the function to dump its inputs to disk and then run the library on any audio file and then hardcoded the input into the unit tests. Instead, I decided I wanted to save a few bytes and wanted to look at generating dummy data dynamically. I wanted to try out Claude for generating the code that would generate the data, so to keep the context manageable I extracted the function and all its dependencies into a self-contained C program (less than 200 lines altogether) and asked it to write a function that would generate dummy data, in C. Impressively, it recognized the structure of the code and correctly identified it as a component of an audio codec library, and provided a reasonably complete description of many minute details specific to this codec and the work that the function was doing. Rather less impressively, it decided to ignore my request and write a function that used C++ features throughout, such as type inference and lambdas, or should I say "lambdas" because it was actually just a function-defined-within-a-function that tried to access and mutate variables outside of its own function scope, like we were writing Javascript or something. Even apart from that, the code was rife with the sorts of warnings that even a default invocation of gcc would flag. I can see why people would be wowed by this on its face. I wouldn't expect any average developer to have such a depth of knowledge and breadth of pattern-matching ability to be able to identify the specific task that this specific function in this specific audio codec was performing. At the same time, this is clearly not a tool that's suitable for letting loose on a codebase without EXTREME supervision. This was a fresh session (no prior context to confuse it) using a tightly crafted prompt (a small, self-contained C program doing one thing) with a clear goal, and it still required constant handholding. At the end of the day, I got the code working by editing it manually, but in an honest retrospective I would have to admit that the overall process actually didn't save me any time at all. Ironically, despite how they're sold, these tools are infinitely better at going from code to English than going the other way around.
- angusturner 1y ago
  I feel this. I've had a few tasks now where in honest retrospect I find myself asking "did that really speed me up". Its a bit demoralising cause not only do you waste time, you have a worse mental model of the resulting code and feel less sense of ownership over the result. Brainstorming, ideation and small, well defined tasks where I can quickly vet the solution : these feel like the sweet spot for current frontier model capabilities. (Unless you are pumping out some sloppy React SPA that you don't care about anything except get it working as fast as possible - fine, get Claude code to one shot it)
rco8786 1y ago
I still use Claude Code and Cursor and tbh still run into a lot of the same issues. Hallucinating code, hallucinating requirements, even when scoped to a very simple "make this small change". It's good enough that it helps, particularly in areas or languages that I'm unfamiliar with. But I'm constantly fighting with it.
realusername 1y ago
I tried again recently and I see absolutely no difference. If there's been some improvement, it's very subtle. There's a big difference with their benchmarks and real world coding.

jonstewart 1y ago

I first tried getting specific with Claude Code. I made the Claude.md, I detailed how to do TDD, what steps it should take, the commands it should run. It was imperfect. Then I had it plan (think hard) and write the plan to a file. I’d clear context, have it read the plan, ask me questions, and then have it decompose the plan into a detailed plan of discrete tasks. Have it work its way through that. It would inevitably go sideways halfway through, even clearing context between each task. It wouldn’t run tests, it would commit breakage, it would flip flop between two different broken approaches, it was just awful. Now I’ve just been vibing, writing as little as possible and seeing what happens. That sucks, too. It’s amazing at reviewing code. It will identify what you fear, the horrors that lie within the codebase, and it’ll bring them out into the sunlight and give you a 7 step plan for fixing them. And the coding model is good, it can write a function. But it can’t follow a plan worth shit. And if I have to be extremely detailed at the function by function level, then I should be in the editor coding. Claude code is an amazing niche tool for code reviews and dialogue and debugging and coping with new technologies and tools, but it is not a productivity enhancement for daily coding.

liszper 1y ago
With all due respect, you sound like someone who is just getting familiar with these tools. 100 more hours spent with AI coding and you will be much more productive. Coding with AI is a slightly different skill from coding, similar how managing software engineers is different from writing software.
- TheRoque 1y ago
  Then, it's the job of someone else to use these tools, not developers
  - liszper 1y ago
    I agree with your point. I think this is the reason why most developers still don't get it, because AI coding ultimately requires a "higher level" methodology.

xmpir 1y ago

Same as for human software engineers... We'll see Conway' law all again with agentic coding!

bryanrasmussen 1y ago
maybe it just works that way for Agents because they see in the data it works that way for humans.

tedggh 1y ago

I found out that summarizing a completed task and feeding it to a new context works better than staying on the same context for multiple tasks. So let’s say I have a sprint with tasks 1, 2 and 3. I start by creating a project with general information including the spec, git issues, code base, folder trees, etc then work on Task 1. When done I ask for a summary using a template, which gives me a txt file describing what the original goal was, what we changed and what the next steps are. Then I repeat the process for Task 2 and I feed the summary from Task 1. At least in ChatGPT keeping the same context for multiple tasks has lots of issues like speed, increased hallucinations, and ChatGPT referencing content from old files.

smw 1y ago
Hell, claude even makes that part of the standard workflow, with /compact; cleverly using the llm itself to summarize the previous context
- furyofantares 1y ago
  /compact is really poor imo. Quality just falls off a ledge after it. I much prefer to choose tasks that can be done with 25%+ context left and then just start the next task with fresh context. If I'm getting low on context I have it summarize the plan and progress in a text file rather than use /compact and then start a fresh context and reference that file, which I can then edit and try again if I'm not getting good results.
- bryanlarsen 1y ago
  Once you see that message it's time to finish the task without AI because Claude will start crapping over your codebase if you let it continue.
  - ewoodrich 1y ago
    I just got burned by that, it compacted and then immediately dropped what it was in the middle of working on to redo something it had already finished half an hour earlier. Which, predictably, sent it into "systematically destroy the entire working codebase" mode because the code it was now reading didn't match expectations of the original instructions. So it started extrapolating like "huh, it looks like the function already exists, therefore, user must have meant [bizarre completely out of left field guess] instead" in an escalating loop of confusion and code mangling.

OptionOfT 1y ago

And the value of AI as pushed to us by these companies is in doing larger units of work. But... reviewing code is harder than writing code. Expressing how I want something to be done in natural language is incredibly hard. So over time I'm spending a lot of energy in those things, and only getting it 80% right. Not to mention I'm constantly in this highly suspicious mode, trying to pierce through the veil of my own prompt and the code generated, because it's the edge cases that make work hard. The end result is exhaustion. There is no recharge. Plans are front-loaded, and then you switch to auditing mode. Whereas with code you front-load a good amount of design, but you can make changes as you go, and since you know your own code the effort to make those are much lower.

palmotea 1y ago
That makes very clear these tools are not meant to serve you, you are meant to serve these tools.
- ryoshu 1y ago
  Reverse centaur.
- fragmede 1y ago
  What is this, that episode of The Twilight Zone called "to serve man"?
nicce 1y ago
> Expressing how I want something to be done in natural language is incredibly hard Surprise, surprise… that is why programming languages were created.
- dragonwriter 1y ago
  Programming languages don’t solve that problem, since someone still has to explain what needs to be done in natural language unless the end customer is also the programmer. Programming languages were created because of the different problem of “its very hard to get computers to understand natural language even if you know how to express what you want in it”.

elpakal 1y ago

I agree with the premise of the article, and have felt that we're probably seeing AI code gen tools being limited by the constraints being put on them by traditional source code management tools like git and GitHub. Those tools were designed for incremental changes (patches), and have worked well for humans to organize changes so they could be more easily reviewed, maintained and reasoned about. Units of work in the form of features, patches etc rely on "pull requests" which are a function of the above.

marstall 1y ago

doing things in small chunks is good. so is it doing things in large chunks sometimes. In AI, like in life, there are no hard and fast rules and we're all figuring it out as we go. Like with "vibe coding" - sometimes it's ok to not even look at the code AI is generated, sometimes you need to understand every line. It feels like part of my journey to being an "AI developer" is being present for those tradeoffs, metabolizing each one into my craft. AI is a fickle, but powerful horse. I'm finding it a privilege to learn how to be a rider.

danparsonson 1y ago

I seem to be in a minority but I find user stories or features to be really awkward and unnatural units of work for building software. Sure these things help to define the expected result but they shouldn't directly drive the development process. Imagine building a house that way - you don't build the living room, then the kitchen, then the bathroom etc.; you build floors, walls, the roof... The 'features' or use cases for the building arise out of the combination of different elements that were put into it, and usually right near the end of the build. The same is true for basically anything else that we build or create - if you're making a sculpture, do you finish working on one leg first before you move onto some other part? Features are vertical slices through the software cake, but the cake is actually made out of horizontal layers. Creating a bunch of servings of cake and then trying to stick them together just results in a fragile mess that's difficult to work with and easy to break.

nemomarx 1y ago
They make more sense if you think about adding to an existing house. "I want to open up this wall to serve X function" kinda work they're very well adapted to legacy enterprise work
- gizmo686 1y ago
  Even there, "open up this wall" is not a unit of work. You need to: * Evaluate what, if any, structural implications removing the wall has * Tear down the existing wall * Redo any plumbing, ductwork, wiring, etc that was hiding in the wall * Remediate structural concerns from removing the wall. * Redo the flooring * Repair and repaint any damage done to remaining drywall If this is part of a larger renovation, you will likely schedule work so the above tasks happen at the same time as other similar tasks. E.g. A meaningful unit of work might be "electrical roughing", which would include both moving wires that were previously in the wall, and running a new circuit to the garage for a car charger. No user story covers those to tasks, but the nature of renovating a house means that it makes sense to do them together.
galbar 1y ago
My take on this is that, from a SW development POV, user stories are not the right unit of work. Instead, I treat user stories as "Epics". Stake holders can track that Epic for progress, as the unit of work from their POV. Internally, the team splits Epics into "Spikes" (figure out what to do) and "Tasks" (executing on the things we need to do). - Spikes are scoped to up to 3 days and their outcome is usually a doc and either a follow-up Spike or Tasks to execute. - Tasks must be as small and unambiguous as possible (within reason).

jaaron 1y ago

Agreed that picking the right size of work is critical. I didn't know about Kiro specs. I've been playing around with my own org-mode based approach with mixed success in keeping dev agent work tracked: https://github.com/farra/dev-agent-work https://github.com/farra/dev-agent-work

stpedgwdgfhgdd 1y ago

Test driven development, there is nothing more to say for the coming year

rglover 1y ago

What I've found works best: 1. Assume that any model will start to lose focus beyond 50K-100K tokens (even with a huge context window). 2. Be gluttonous with chats. At the first sign of confusion or mistakes, tell it to generate a new prompt and move to a new chat. 3. Write detailed prompts with clear expectations (from how the code should be written to the specific implementation that's required). Combine these with context like docs to get a fairly consistent hit rate. 4. Use tools like Cline that let you switch between an "Act" and "Plan" mode. This saves a ton of tokens but also avoids the LLM getting stuck on a loop when it's debugging. I recently wrote this short blog post related to this: https://ryanglover.net/blog/treat-the-ai-like-it-s-yourself https://ryanglover.net/blog/treat-the-ai-like-it-s-yourself The above approach helped me to implement a full-blown database wrapper around LMDB for Node.js in ~2 weeks of slow back-and-forth (link to code in post for those who are curious).

chrisrickard 1y ago

We are tackling this at https://userdoc.fyi https://userdoc.fyi - we help you build your specs (epics, stories, acceptance criteria, tech notes, test cases, etc) - then you can generate what we call Dev Plans, one or more requirement layers for implementation. e.g maybe a dev plan is all your authentication feature requirements, or in the house of analogy – all the requirements for the rooms, but with instructions to actually just first build the floor, and the walls. Dev plans then slice the reqs into meaningful units of work, as mentioned in the article – a feature/story, is often too large of a checkpoint, or often needs to be implemented in collaboration with other features/stories, so it understands the correct architectural context,. You can then implement Dev plans over MCP, or copy to .md for tools like Lovable or V0.

zmmmmm 1y ago

I prefer small units of work. It really surprises me how fast people have leapt from the 2x speedup (mind blowing level of productivity increase) to full agentic coding without really questioning if it's a good idea. When I have let Claude loose and vibe coded up hundreds of lines at a time that I have no familiarity with, I viscerally feel how I no longer understand or can maintain the app I've built. If I can't get Claude to do the next change I need, I'm screwed. I'm very satisfied at the moment to be wielding LLMs as a tool at the individual function / microfeature level and getting a very satisfying productivity improvement.

deleted 1y ago

[deleted]

trash_cat 1y ago

>> "Turns out the major bottleneck is not intelligence, but rather providing the correct context." But this has more or less always been the case for LLMs. The challenge becomes context capure. Which in my opinion is the real challenge with LLM adoption. Without the right contex, some tasks just cannot be reliably completed.