3 ms·
> If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster. I thought so to
by 0x457 1y ago
> If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster.
I thought so too, but sometimes I had better results with one sentence prompt (+README.md) where it delivered the exact thing I wanted. I also had a very detailed prompt with multiple subtasks, all were very detailed +README.md +AGENTS.md and results were very poor.
- stillpointlab 1y agoThis is true in my experience but it doesn't go against my larger point. Choosing the "goldilocks" context is a bit of an art, not too big not too small. It reminds of a famous witty quote [1]: “I apologize for such a long letter - I didn't have time to write a short one.” If you send too much info at once it does seem to confuse the agent, just like if you ask it to do too much all at once. That is yet another property it shares with a junior engineer. It is easy to overwhelm a new contributor to a project with too much information, especially if it isn't strictly relevant.
- daveguy 1y agoAlso, regardless of the prompt they only get ~80% accuracy on coding benchmarks. So even with the absolute perfect prompt incantation, you can expect it to fail 1 out of 5 times.
- 0x457 1y agoThat's why I make initial context (e.g. AGENTS.md) is about how to bootstrap context for a current task/project. Now, my prompts only need to be good enough to hint how to read the graph correctly.
- gaeb69 1y agoReasoning model prompting is relevant here. OpenAI in their docs state that giving detailed, step-by-step instructions often hinders reasoning models. It's better to clearly define the outcome along with what you're working with. Then let the mode interpolate.