2 ms·
Challenging Big-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- isaacfrond 4y agoFascinating. They gave various reasoning tasks to GPT-3. The model was helped by adding in the question the phrase 'Let's think step by step'. As a result performance increased to above human level on 17 of 23 tasks. For example, this was one the questions Q: If you follow these instructions, do you return to the starting point? Turn left. Turn right. Take 5 steps. Take 4 steps. Turn around. Take 9 steps. Options: - Yes - No A: Let's think step by step And this was the model output: We start at the origin (0, 0), facing the positive y-axis. (1) Turn left: (0, 0), facing the negative x-axis. (2) Turn right: (0, 0), facing the positive y-axis. (3) Take 5 steps: (0, 5), facing the positive y-axis. (4) Take 4 steps: (0, 9), facing the positive y-axis. (5) Turn around: (0, 9), facing the negative y-axis. (6) Take 9 steps: (0, 0), facing the negative y-axis. Since (0, 0) is (0, 0), we are indeed where we started. So the answer is Yes.