6 ms·

Claude Code is ahead of anything else, in a very noticeable way. (I've been writing my own cli tooling for AI codegen from 2023 - and in that journey I've tried

by jeswin 1y ago

Claude Code is ahead of anything else, in a very noticeable way. (I've been writing my own cli tooling for AI codegen from 2023 - and in that journey I've tried most of the options out there. It has been a big part of my work - so that's how I know.) I agree with many things that the author is doing: 1. Monorepos can save time 2. Start with a good spec. Spend enough time on the spec. You can get AI to write most of the spec for you, if you provide a good outline. 3. Make sure you have tests from the beginning. This is the most important part. Tests (along with good specs) are how an AI agent can recurse into a good solution. TDD is back. 4. Types help (a lot!). Linters help as well. These are guard rails. 5. Put external documentation inside project docs, for example in docs/external-deps. 6. And finally, like every tool it takes time to figure out a technique that works best for you. It's arguably easier than it was (especially with Claude Code), but there's still stuff to learn. Everyone I know has a slightly different workflow - so it's a bit like coding. I vibe coded quite a lot this week. Among them, Permiso [1] - a super simple GraphQL RBAC server. It's nowhere close to best tested and reviewed, but can be quite useful already if you want something simple (and can wait until it's reviewed.) [1]: https://github.com/codespin-ai/permiso https://github.com/codespin-ai/permiso

unshavedyak 1y ago
> 2. Start with a good spec. Spend enough time on the spec. You can get AI to write most of the spec for you, if you provide a good outline. Curious how you outline the spec, concretely. A sister markdown document? How detailed is it? etc. > 3. Make sure you have tests from the beginning. This is the most important part. Tests (along with good specs) are how an AI agent can recurse into a good solution. TDD is back. Ironically i've been struggling with this. For best results i've found claude to do best with a test hook, but then claude loses the ability to write tests before code works to validate bugs/assumptions, it just starts auto fixing things and can get a bit wonky. It helps immensely to ensure it doesn't forget anything or abandon anything, but it's equally harmful at certain design/prototype stages. I've taken to having a flag where i can enable/disable the test behavior lol.
- jeswin 1y ago
  > Curious how you outline the spec, concretely. A sister markdown document? How detailed is it? etc. Yes. I write the outline in markdown. And then get AI to flesh it out. The I generate a project structure, with stubbed API signatures. Then I keep refining until I've achieved a good level of detail - including full API signatures and database schemas. > Ironically i've been struggling with this. For best results i've found claude to do best with a test hook, but then claude loses the ability to write tests before code works to validate bugs/assumptions, it just starts auto fixing things and can get a bit wonky. I generate a somewhat basic prototype first. At which point I have a good spec, and a good project structure, API and db schemas. Then continuously refine the tests and code. Like I was saying, types and linting are also very helpful.
  - rane 1y ago
    I don't even write the outline myself. I tell CC to come up with a plan, and then we iterate on that together with CC and I might also give it to Gemini for review and tell CC to apply Gemini's suggestions.
    - jerpint 1y ago
      I built a tool for this exact workflow in mind but with MCP and versioning included so you can easily track and edit the files on any platform including cursor, Claude desktop etc https://github.com/jerpint/context-llemur https://github.com/jerpint/context-llemur
  - skydhash 1y ago
    What kind or projects are more suitable for this approach? Because my workflow, sans LLM agents, have been to rely on frameworks to provide a base abstraction for me to build upon. The hardest is to nail down the business domain, done over rounds of discussions with stakeholders. Coding is pretty breezy in comparison.
    - dm3 1y ago
      That's why you see such a difference in time saved using LLMs for programming across the population. If you have all the domain knowledge and the problem is generic enough it's a 100x multiplier. Otherwise your experience can easily range from 0.1x to 10x.
nico 1y ago
Agreed, for CC to work well, it needs quite a bit of structure I’ve been working on a Django project with good tests, types and documentation. CC mostly does great, even if it needs guidance from time to time Recently also started a side project to try to run CC offline with local models. Got a decent first version running with the help of ChatGPT, then decided to switch to CC. CC has been constantly trying to avoid solving the most important issues, sidestepping errors and for almost everything just creating a new file/script with a different approach (instead of fixing or refactoring the current code)
- wenc 1y ago
  I've also found that structure is key instead of trusting its streams of consciousness. For unit testing, I actually pre-write some tests so it can learn what structure I'm looking for. I go as far as to write mocks and test classes that *constrain* what it can do. With constraints, it does a much better job than if it were just starting from scratch and improvising. There's a numerical optimization analogy to this: if you just ask a solver to optimize a complicated nonlinear (nonconvex) function, you will likely get stuck or hit a local optimum. But if you carefully constrain its search space, and guide it, you increase your chances of getting to the optimum. LLMs are essentially large function evaluators with a huge search space. The more you can herd it (like herding a flock into the right pen), the better it will converge.
arevno 1y ago
> 1. Monorepos can save time Yes they can save you some time, but at the cost of Claude's time and lots of tokens making tool calls attempting to find what it needs to find. Aider is much nicer, from the standpoint that you can add the files you need it to know about, and send it off to do its thing. I still don't understand why Claude is more popular than Aider, which is by nearly every measure a better tool, and can use whatever LLM is more appropriate for the task at hand.
- KronisLV 1y ago
  > Aider is much nicer, from the standpoint that you can add the files you need it to know about, and send it off to do its thing. As a user, I don't want to sit there specifying about 15-30 files, then realize that I've missed some and that it ruins everything. I want to just point the tool at the codebase and tell it: "Go do X. Look at the current implementation and patterns, as well as the tests, alongside the docs. Update everything as needed along the way, here's how you run the tests..." Indexing the whole codebase into Qdrant might also help a little.
  - macNchz 1y ago
    I think it makes sense to want that, but at least for me personally I’ve had dramatically better overall results when manually managing the context in Aider than letting Claude Code try to figure out for itself what it needs. It can be annoying, but I think it both helps me be more aware of what’s being changed (vs just seeing a big diff after a while), and lends itself to working on smaller subtasks that are more likely to work on the first try.
    - rane 1y ago
      You get much better results in CC as well if you're able to give the relevant files as a starting point. In that regard these two tools are not all that different.
  - zmmmmm 1y ago
    Aider does know the whole repository tree (it scans the git index). It just doesn't read the files until you tell it to. If it thinks it needs access to a file, it will prompt you to add it. I find this to be a fairly good model. Obviously it doesn't work off line though.
benhurmarcel 1y ago
> Put external documentation inside project docs Most projects have their documentation on their website. Do you spend time formatting it into a clean Markdown file?

by jeswin 1y ago

unshavedyak 1y ago

> 2. Start with a good spec. Spend enough time on the spec. You can get AI to write most of the spec for you, if you provide a good outline. Curious how you outline the spec, concretely. A sister markdown document? How detailed is it? etc. > 3. Make sure you have tests from the beginning. This is the most important part. Tests (along with good specs) are how an AI agent can recurse into a good solution. TDD is back. Ironically i've been struggling with this. For best results i've found claude to do best with a test hook, but then claude loses the ability to write tests before code works to validate bugs/assumptions, it just starts auto fixing things and can get a bit wonky. It helps immensely to ensure it doesn't forget anything or abandon anything, but it's equally harmful at certain design/prototype stages. I've taken to having a flag where i can enable/disable the test behavior lol.

jeswin 1y ago
> Curious how you outline the spec, concretely. A sister markdown document? How detailed is it? etc. Yes. I write the outline in markdown. And then get AI to flesh it out. The I generate a project structure, with stubbed API signatures. Then I keep refining until I've achieved a good level of detail - including full API signatures and database schemas. > Ironically i've been struggling with this. For best results i've found claude to do best with a test hook, but then claude loses the ability to write tests before code works to validate bugs/assumptions, it just starts auto fixing things and can get a bit wonky. I generate a somewhat basic prototype first. At which point I have a good spec, and a good project structure, API and db schemas. Then continuously refine the tests and code. Like I was saying, types and linting are also very helpful.
- rane 1y ago
  I don't even write the outline myself. I tell CC to come up with a plan, and then we iterate on that together with CC and I might also give it to Gemini for review and tell CC to apply Gemini's suggestions.
  - jerpint 1y ago
    I built a tool for this exact workflow in mind but with MCP and versioning included so you can easily track and edit the files on any platform including cursor, Claude desktop etc https://github.com/jerpint/context-llemur https://github.com/jerpint/context-llemur
- skydhash 1y ago
  What kind or projects are more suitable for this approach? Because my workflow, sans LLM agents, have been to rely on frameworks to provide a base abstraction for me to build upon. The hardest is to nail down the business domain, done over rounds of discussions with stakeholders. Coding is pretty breezy in comparison.
  - dm3 1y ago
    That's why you see such a difference in time saved using LLMs for programming across the population. If you have all the domain knowledge and the problem is generic enough it's a 100x multiplier. Otherwise your experience can easily range from 0.1x to 10x.

nico 1y ago

Agreed, for CC to work well, it needs quite a bit of structure I’ve been working on a Django project with good tests, types and documentation. CC mostly does great, even if it needs guidance from time to time Recently also started a side project to try to run CC offline with local models. Got a decent first version running with the help of ChatGPT, then decided to switch to CC. CC has been constantly trying to avoid solving the most important issues, sidestepping errors and for almost everything just creating a new file/script with a different approach (instead of fixing or refactoring the current code)

wenc 1y ago
I've also found that structure is key instead of trusting its streams of consciousness. For unit testing, I actually pre-write some tests so it can learn what structure I'm looking for. I go as far as to write mocks and test classes that *constrain* what it can do. With constraints, it does a much better job than if it were just starting from scratch and improvising. There's a numerical optimization analogy to this: if you just ask a solver to optimize a complicated nonlinear (nonconvex) function, you will likely get stuck or hit a local optimum. But if you carefully constrain its search space, and guide it, you increase your chances of getting to the optimum. LLMs are essentially large function evaluators with a huge search space. The more you can herd it (like herding a flock into the right pen), the better it will converge.

arevno 1y ago

> 1. Monorepos can save time Yes they can save you some time, but at the cost of Claude's time and lots of tokens making tool calls attempting to find what it needs to find. Aider is much nicer, from the standpoint that you can add the files you need it to know about, and send it off to do its thing. I still don't understand why Claude is more popular than Aider, which is by nearly every measure a better tool, and can use whatever LLM is more appropriate for the task at hand.

KronisLV 1y ago
> Aider is much nicer, from the standpoint that you can add the files you need it to know about, and send it off to do its thing. As a user, I don't want to sit there specifying about 15-30 files, then realize that I've missed some and that it ruins everything. I want to just point the tool at the codebase and tell it: "Go do X. Look at the current implementation and patterns, as well as the tests, alongside the docs. Update everything as needed along the way, here's how you run the tests..." Indexing the whole codebase into Qdrant might also help a little.
- macNchz 1y ago
  I think it makes sense to want that, but at least for me personally I’ve had dramatically better overall results when manually managing the context in Aider than letting Claude Code try to figure out for itself what it needs. It can be annoying, but I think it both helps me be more aware of what’s being changed (vs just seeing a big diff after a while), and lends itself to working on smaller subtasks that are more likely to work on the first try.
  - rane 1y ago
    You get much better results in CC as well if you're able to give the relevant files as a starting point. In that regard these two tools are not all that different.
- zmmmmm 1y ago
  Aider does know the whole repository tree (it scans the git index). It just doesn't read the files until you tell it to. If it thinks it needs access to a file, it will prompt you to add it. I find this to be a fairly good model. Obviously it doesn't work off line though.

benhurmarcel 1y ago

> Put external documentation inside project docs Most projects have their documentation on their website. Do you spend time formatting it into a clean Markdown file?