4 ms·

I think there will be lessons learned here as well for better agentic systems writing code more generally; instead of “committing” to code as of the first token

I think there will be lessons learned here as well for better agentic systems writing code more generally; instead of “committing” to code as of the first token generated, first generate overall structure of code base, with abstractions, and only then start writing code. I usually instruct Claude/chatGPT/etc not to generate any code until I tell it to, as they are eager to do so and often box themselves in a corner early on

rmbyrro 2y ago
aider.chat has an /architect mode where you can discuss the architecture first and later ask it to execute the architectural decisions works pretty well, especially because you can use a more capable model for architecting and a cheaper one to code
- namanyayg 2y ago
  I didn't know about this, thanks for sharing
thegeomaster 2y ago
This is literally chain-of-thought! Even better than generic chain-of-thought prompting ("Think step by step and write down your thought process."), you're doing a domain-specific CoT, where you use some of your human intuition on how to approach a problem and imparting the LLM with it.
j_bum 2y ago
Yes I frequently do this too. In fact I often ask whatever model I’m interacting with to not do anything until we’ve devised a plan. This goes for search, code, commands, analysis, etc. It often leads to better results for me across the board. But often I need to repeat those instructions as the chat gets longer. These models are so hyped to generate something even if it’s not requested.
Kinrany 2y ago
We already have languages for expressing abstractions, they're called programming languages. Working software is always built interactively, with a combination of top-down and bottom-up reasoning and experimentation. The problem is not in starting with real code, the problem is in being unable to keep editing the draft.
- qup 2y ago
  Not a problem with the correct tooling.
- Yoric 2y ago
  I agree. On the other hand, I expect that programming languages will keep evolving, and the next generation or so might be designed with LLMs in mind. For instance, there's a conversation in the Rust's lang forum on how to best extract API documentation for processing by an LLM. Will this help? No idea. But it's an interesting experiment nevertheless.
namanyayg 2y ago
That's exactly what I've understood, and this becomes even more important as the size of codebase scales. Ultimately, LLMs (like humans) can keep a limited context in their "brains". To use them effectively, we have to provide the right context.

rmbyrro 2y ago

aider.chat has an /architect mode where you can discuss the architecture first and later ask it to execute the architectural decisions works pretty well, especially because you can use a more capable model for architecting and a cheaper one to code

namanyayg 2y ago
I didn't know about this, thanks for sharing

thegeomaster 2y ago

This is literally chain-of-thought! Even better than generic chain-of-thought prompting ("Think step by step and write down your thought process."), you're doing a domain-specific CoT, where you use some of your human intuition on how to approach a problem and imparting the LLM with it.

j_bum 2y ago

Yes I frequently do this too. In fact I often ask whatever model I’m interacting with to not do anything until we’ve devised a plan. This goes for search, code, commands, analysis, etc. It often leads to better results for me across the board. But often I need to repeat those instructions as the chat gets longer. These models are so hyped to generate something even if it’s not requested.

Kinrany 2y ago

We already have languages for expressing abstractions, they're called programming languages. Working software is always built interactively, with a combination of top-down and bottom-up reasoning and experimentation. The problem is not in starting with real code, the problem is in being unable to keep editing the draft.

qup 2y ago
Not a problem with the correct tooling.
Yoric 2y ago
I agree. On the other hand, I expect that programming languages will keep evolving, and the next generation or so might be designed with LLMs in mind. For instance, there's a conversation in the Rust's lang forum on how to best extract API documentation for processing by an LLM. Will this help? No idea. But it's an interesting experiment nevertheless.

namanyayg 2y ago

That's exactly what I've understood, and this becomes even more important as the size of codebase scales. Ultimately, LLMs (like humans) can keep a limited context in their "brains". To use them effectively, we have to provide the right context.