3 ms·

Yeah, you asked it specific follow-up questions, designed to guide it towards the correct answer. That feels like cheating, because it's not generalizable. I'd

Yeah, you asked it specific follow-up questions, designed to guide it towards the correct answer. That feels like cheating, because it's not generalizable. I'd like to see an evaluation with a generic prompt like "Is that the best solution? Make sure you look for trick questions.", because that is applicable to any input. Or something like "Pick apart this answer like a pedantic HN commenter. Do your best to prove it wrong, or begrudgingly say it's correct if you can't: <previous answer here>"

Hugsun 2y ago
That is exactly what I did in the section I'm referring to. I asked both suggestive and non-suggestive questions. It needed the suggestion to get the right answer. Just asking it to elaborate didn't change it's answer.
- EnergyAmy 2y ago
  Maybe I'm just being dense, but are the questions like "One of the trips is not necessary. Which one is it and why?" and "Is there another way to solve the problem?" part of separate conversations? I read those as a single conversation where you first asked it "One of the trips is not necessary. Which one is it and why?", and then later asked it "Is there another way to solve the problem?", so that the leading question was part of the earlier conversation when you asked it "Is there another way to solve the problem?". If those were separate conversations, then the last example of "Is there another way to solve the problem?" is more what I'm talking about, yes. I'd be interested in seeing something a little more specific (but still generalizable) like "Make sure you've looked for trick questions" EDIT: I just tried it, and "Think carefully about every entity and every step." as you tried didn't work, but something like "Make sure you check to see if it's a trick question" works, which is what I'm talking about. That's something you could put in any prompt, and helps it reason.

Hugsun 2y ago

That is exactly what I did in the section I'm referring to. I asked both suggestive and non-suggestive questions. It needed the suggestion to get the right answer. Just asking it to elaborate didn't change it's answer.

EnergyAmy 2y ago
Maybe I'm just being dense, but are the questions like "One of the trips is not necessary. Which one is it and why?" and "Is there another way to solve the problem?" part of separate conversations? I read those as a single conversation where you first asked it "One of the trips is not necessary. Which one is it and why?", and then later asked it "Is there another way to solve the problem?", so that the leading question was part of the earlier conversation when you asked it "Is there another way to solve the problem?". If those were separate conversations, then the last example of "Is there another way to solve the problem?" is more what I'm talking about, yes. I'd be interested in seeing something a little more specific (but still generalizable) like "Make sure you've looked for trick questions" EDIT: I just tried it, and "Think carefully about every entity and every step." as you tried didn't work, but something like "Make sure you check to see if it's a trick question" works, which is what I'm talking about. That's something you could put in any prompt, and helps it reason.