All these "constantly challenge your own assumptions" prompts really do in practice is make it second guess assumptions that actually are obvious and true, whic

All these "constantly challenge your own assumptions" prompts really do in practice is make it second guess assumptions that actually are obvious and true, which pollutes the output further. In a sense, it bikesheds[1] itself. It's been told it needs more discussion/debate to solve a problem, but it's only smart enough to tackle minor details that might not matter. And when it does try to tackle major details, it screws them up, causing a cascade of compounding bullshit. And it also tends to cause it to roleplay someone who constantly bungles things, which is bad because it takes it away from its default RLHF tuning for factually accurate outputs on first shot (not that it lives up to this goal, but that's how they've tried to train it). Getting this right seems to be a very tricky problem. 1: https://bikeshed.com/ https://bikeshed.com/