There was an interesting substack that went through the logic of this type of failure[1]. The tl;dr is that phrasing the question as a Yes/No forces the answer

There was an interesting substack that went through the logic of this type of failure[1]. The tl;dr is that phrasing the question as a Yes/No forces the answer into, well, a yes or a no. Without pre-answer reasoning trace, the LLM is forced to make a decision based on it's training data, which here is more likely to not be from 2025, so it picks no. Any further output cannot change the previous output. [1] https://ramblingafter.substack.com/p/why-does-chatgpt-think-mammoths-were https://ramblingafter.substack.com/p/why-does-chatgpt-think-...