5 ms·
I follow the same process. I have a design in mind for the problem at hand, but I don't reveal it to Codex. I go back and forth a bit to see if its proposals ar
by throwaway7783 5mo ago
I follow the same process. I have a design in mind for the problem at hand, but I don't reveal it to Codex. I go back and forth a bit to see if its proposals are better than mine. I go back and forth on tradeoffs of various approaches. And then I ask it to compare its proposals with mine. I "win" most of the time but there are many times where it shows a me a better, or simpler approach, or makes me rethink the solution altogether.
Once this is done, the mechanical coding parts are mostly routine (for codex)
- hackermanai 5mo agoI think this approach is more common than the hype for actual work. I do something similar, many back and forth, then settle on something often with now known tradeoffs, written by hand to spot issues as a final guard/ keep consistent naming etc.
- yread 5mo ago> I go back and forth a bit to see if its proposals are better than mine I find it useful to let it generate benchmarks comparing the approaches. Turns out AI is terrible at guessing whats faster or allocates less
- puilp0502 4mo ago> Turns out AI is terrible at guessing whats faster or allocates less s/AI/a human being/ would work equally well, lol. Jokes aside, I do like the approach of letting the AI build something deterministic and make decisions based on that.
- chris_st 4mo agoYup, just like people!
- ptsneves 4mo agoI had the exact experience yesterday. I have a performance problem and went down the path of optimising part of a pipeline that when benchmarked was not the bottleneck, even if it looked plausible for me and the llm. When I asked it to make a final benchmark for documentation I found most of the work I did improved 30% while another path would have improved a magnitude more. Thankfully iteration is now faster than ever and given how fast it creates tests, previous tests created for the aborted optimisation were helpful.
- a_bonobo 5mo agoI really like this pattern and use it often, this 'not showing my cards'. The second I hint towards the LLM what I prefer it will become sycophantic and invent nonsense why my preferred solution is better. I'm sure there's an interesting study on how users 'leak' their preference unintentionally to the LLM; perhaps when users list their options, they often put their prefered option first; but not showing the cards on my hand has been very useful when thinking through a problem with LLMs.
- HarHarVeryFunny 4mo agoYes, outside of coding too, it’s a good idea to ask open ended questions rather than ask for confirmation, to avoid this sycophantic bias
- nickcw 5mo agoLLMs are very prone to priming in my experience. That is the human psychology name for what you are describing; whether it should be applied to LLMs I don't know, but it describes the phenomenon perfectly.
- jimbokun 4mo agoMakes sense as priming is at the core of how an LLM is trained. “Given these words, predict the next word.”
- williamdclt 4mo agoSame. Alternatively (or in addition), I sometimes present my preferred idea as being a "bad/naive/stupid option" (or a suggestion from someone who can't be trusted) to see how it stands up to sycophancy to it being bad. As expected the LLM will usually say "yeah it's bad!" and give plausible-sounding reasons for it, but if these reasons are nonsensical it's a good sign that I'm not missing anything
- avadodin 4mo agoIt's not limited to arguing with LLMs but if you want a honest opinion you should remember to push back even when it agrees with your hidden preference at first. Sometimes it is only being contrarian or supporting the underdog. Steelman the opposition.
- revv00 4mo agoi bet you've contributed a lot of training trajectories for those AI's.
- chris_st 4mo agoGood!
- daniel3303 4mo ago[flagged]