3 ms·
I am getting disproportionately good results with the models by following a process: spec -> plan -> critique -> improve plan -> implement plan.
by pgt 7mo ago
I am getting disproportionately good results with the models by following a process: spec -> plan -> critique -> improve plan -> implement plan.
- CharlesW 7mo agoIf I may "yes, and" this: spec → plan → critique → improve plan → implement plan → code review It may sound absurd to review an implementation with the same model you used to write it, but it works extremely well. You can optionally crank the "effort" knob (if your model has one) to "max" for the code review.
- frumiousirc 7mo agoA blanket follow-up "are you sure this is the best way to do it?" Frequently returns, "Oh, you are absolutely correct, let me redo this part better."
- jgilias 7mo agoYou should start a new session for the code review to make sure the context window is not polluted with the work on implementation itself. At the end of the day it’s an autocomplete. So if you ask “are you sure?” then “oh, actually” is a statistically likely completion.
- CharlesW 7mo ago> You should start a new session for the code review to make sure the context window is not polluted with the work on implementation itself. I'm just a sample size of one, but FWIW I didn't find that this noticably improved my results. Not having to completely recreate all the LLM context neccessary to understand the literal context and the spectrum of possible solutions (which the LLM still "knows" before you clear the session) saves lots of time and tokens.
- jgilias 7mo agoInteresting, I definitely see better results on a clean session. On a “dirty” session it’s more likely to go with “this is what we implemented, it’s good, we could improve it this way”, whereas on a clean session it’s a lot more likely to find actual issues or things that were overlooked in the implementation session.
- deleted 7mo ago[deleted]
- afroisalreadyin 7mo agoCan you give a little more detail how you execute these steps? Is there a specific tool you use, or is it simply different kinds of prompts?
- pgt 7mo agoI wrote it down here: https://x.com/BraaiEngineer/status/2016887552163119225 https://x.com/BraaiEngineer/status/2016887552163119225 However, I have since condensed this into 2 prompts: 1. Write plan in Plan Mode 2. (Exit Plan Mode) Critique -> Improve loop -> Implement.
- ramoz 7mo agoI follow a very similar workflow, with manual human review of plans and continuous feedback loops with the plan iterations See me in action here. It's a quick demo: https://youtu.be/a_AT7cEN_9I https://youtu.be/a_AT7cEN_9I
- 4b11b4 7mo agosimilar approach