6 ms·

I built a package which I use for large codebase work[0]. It starts with /feature, and takes a description. Then it analyzes the codebase and asks questions.

I built a package which I use for large codebase work[0]. It starts with /feature, and takes a description. Then it analyzes the codebase and asks questions. Once I’ve answered questions, it writes a plan in markdown. There will be 8-10 markdowns files with descriptions of what it wants to do and full code samples. Then it does a “code critic” step where it looks for errors. Importantly, this code critic is wrong about 60% of the time. I review its critique and erase a bunch of dumb issues it’s invented. By that point, I have a concise folder of changes along with my original description, and it’s been checked over. Then all I do is say “go” to Claude Code and it’s off to the races doing each specific task. This helps it keep from going off the rails, and I’m usually confident that the changes it made were the changes I wanted. I use this workflow a few times per day for all the bigger tasks and then use regular Claude code when I can be pretty specific about what I want done. It’s proven to be a pretty efficient workflow. [0] GitHub.com/iambateman/speedrun

j45 1y ago
This looks very cool. I see it has a pseudo code step, was it helpful at all to try to define a workflow, process or procedure beforehand? I've also heard that keeping each file down to 100 lines is critical before connecting them. Noticed the same but haven't tried it in depth.
- dhorthy 1y ago
  File size matters if you don’t have strategically placed “read the entire file” instructions for certain parts of the workflow (we do)
eddywebs 1y ago
The biggest challenge i found with LLMs on large codebase is making the same mistakes again and again How do keep track of the architecture decisions in context of every tasks on the large codebase ?
- tom_m 1y ago
  Very very clear, unambiguous, prompts and agent rules. Use strong language like "must" and "critical" and "never" etc. I would also try working on smaller sections of a large codebase at a time too if things are too inaccurate. The AI coding tools are going to be looking at other files in the project to help with context. Ambiguity is the death of AI effectiveness. You have to keep things clear and so that may require addressing smaller sections at a time. Unless you can really configure the tools in ways to isolate things. This is why I like tools that have a lot of control and are transparent. If you ask a tool what the full system and user prompt is and it doesn't tell you? Run away from that tool as fast as you can. You need to have introspections here. You have to be able to see what causes a behavior you don't want and be able to correct it. Any tool that takes that away from you is one that won't work.
  - sarchertech 1y ago
    > Use strong language like "must" and "critical" and "never" etc. Truly we live in the stupidest timeline. Imagine if you had a domestic robot but when you asked it make you breakfast you had to preface your request with “it’s critical that you don’t kill me.” Or when you asked it to do the laundry you had to remember to tell it that it “must not make its own doors by knocking holes in the wall” and hope that it listens.
    - phs318u 1y ago
      > Truly we live in the stupidest timeline. Wholeheartedly agree. Truly, I look around and see no shortage of evidence for this assertion. EDITED to make it clear I am agreeing with parent.
    - sally_glance 1y ago
      There is even a chance our timeline might include that robot too at some point... Book recommendation no one asked for but which is essentially about some guy living through multiple more or less stupid timelines: Count to Eschaton series by John C. Wright
    - 1y ago
scuff3d 1y ago
I will never understand why anyone wants to go through all this. I don't believe for a second this is more productive than regular coding with a little help from the LLM.
- mattigames 1y ago
  There is a chunk of devs using AI that do it not because they believe it makes them more productive in the present but because it might do so in the near future thanks to advances on AI tech/models, and then some do it because they think it might be required from them to do it this way by their bosses at some point in the future, so they can show preparedness and give the impression of being up to date with how the field evolves, even if at the end it turns out it doesn't speed up things that much.
  - scuff3d 1y ago
    That line of thinking makes no sense to me honestly. We are years into this, and while the models have gotten better, the guard rails that have to be put on these things to keep the outputs even semi useful are crazy. Look into the system prompts for Claude sometime. And then we have to layer all these additional workflows on top... Despite the hype I don't see any way we get to this actually being a more productive way to work anytime soon. And not only are we paying money for the privilege to work slower (in some cases people are shelling out for multiple services) but we're paying with our time. There is no way working this way doesn't degrade your fundamental skills, and (maybe) worse the understanding of how things actually work. Although I suppose we can all take solice in the fact that our jobs aren't going anywhere soon. If this is what it takes to make these things work.
    - ktzar 1y ago
      And most importantly, we're paying with our brain and skills degradation. Once all these services stop being subsidised there will be a massive amount of programmers who no longer can code.
    - athrowaway3z 1y ago

j45 1y ago

This looks very cool. I see it has a pseudo code step, was it helpful at all to try to define a workflow, process or procedure beforehand? I've also heard that keeping each file down to 100 lines is critical before connecting them. Noticed the same but haven't tried it in depth.

dhorthy 1y ago
File size matters if you don’t have strategically placed “read the entire file” instructions for certain parts of the workflow (we do)

eddywebs 1y ago

The biggest challenge i found with LLMs on large codebase is making the same mistakes again and again How do keep track of the architecture decisions in context of every tasks on the large codebase ?

tom_m 1y ago
Very very clear, unambiguous, prompts and agent rules. Use strong language like "must" and "critical" and "never" etc. I would also try working on smaller sections of a large codebase at a time too if things are too inaccurate. The AI coding tools are going to be looking at other files in the project to help with context. Ambiguity is the death of AI effectiveness. You have to keep things clear and so that may require addressing smaller sections at a time. Unless you can really configure the tools in ways to isolate things. This is why I like tools that have a lot of control and are transparent. If you ask a tool what the full system and user prompt is and it doesn't tell you? Run away from that tool as fast as you can. You need to have introspections here. You have to be able to see what causes a behavior you don't want and be able to correct it. Any tool that takes that away from you is one that won't work.
- sarchertech 1y ago
  > Use strong language like "must" and "critical" and "never" etc. Truly we live in the stupidest timeline. Imagine if you had a domestic robot but when you asked it make you breakfast you had to preface your request with “it’s critical that you don’t kill me.” Or when you asked it to do the laundry you had to remember to tell it that it “must not make its own doors by knocking holes in the wall” and hope that it listens.
  - phs318u 1y ago
    > Truly we live in the stupidest timeline. Wholeheartedly agree. Truly, I look around and see no shortage of evidence for this assertion. EDITED to make it clear I am agreeing with parent.
  - sally_glance 1y ago
    There is even a chance our timeline might include that robot too at some point... Book recommendation no one asked for but which is essentially about some guy living through multiple more or less stupid timelines: Count to Eschaton series by John C. Wright
  - 1y ago

scuff3d 1y ago

I will never understand why anyone wants to go through all this. I don't believe for a second this is more productive than regular coding with a little help from the LLM.

mattigames 1y ago
There is a chunk of devs using AI that do it not because they believe it makes them more productive in the present but because it might do so in the near future thanks to advances on AI tech/models, and then some do it because they think it might be required from them to do it this way by their bosses at some point in the future, so they can show preparedness and give the impression of being up to date with how the field evolves, even if at the end it turns out it doesn't speed up things that much.
- scuff3d 1y ago
  That line of thinking makes no sense to me honestly. We are years into this, and while the models have gotten better, the guard rails that have to be put on these things to keep the outputs even semi useful are crazy. Look into the system prompts for Claude sometime. And then we have to layer all these additional workflows on top... Despite the hype I don't see any way we get to this actually being a more productive way to work anytime soon. And not only are we paying money for the privilege to work slower (in some cases people are shelling out for multiple services) but we're paying with our time. There is no way working this way doesn't degrade your fundamental skills, and (maybe) worse the understanding of how things actually work. Although I suppose we can all take solice in the fact that our jobs aren't going anywhere soon. If this is what it takes to make these things work.
  - ktzar 1y ago
    And most importantly, we're paying with our brain and skills degradation. Once all these services stop being subsidised there will be a massive amount of programmers who no longer can code.
  - athrowaway3z 1y ago