I understand what you are saying, and I know that chat GPT generates nonsense on things it doesn't know and also does not respond that it doesn't know. But let

I understand what you are saying, and I know that chat GPT generates nonsense on things it doesn't know and also does not respond that it doesn't know. But let's consider that we feed GPT a blob of text, just like in the demo. Then ask it to solve a problem based on it and tell it to also show us the reasoning steps. If it just generates some meaningful text but does not solve the problem then we are sure it can't reason. But my question is how can it show me the steps? If it can't reason, where is the reasoning coming from? I am trying to compare this with a human student. Let's say I gave a student the tax codes and nothing else. Then I give that student a tax calculation problem which involves knowing that the problem contains general tax calculation steps and also some special rules which are embedded somewhere in the tax code. When the student comes up to me with a correct answer, I don't immediately know that the student understands tax codes. To really know, I have to ask him how did you come up with the solution. If he can show me the steps then I will know he understood the tax codes. Beyond that there is no way to look into his brain and confirm that he has understood the problem. So if GPT-4 can do the same, why would I not assume it has reasoning abilities? Maybe the process by which it comes up with the reasoning steps is different from us, but they are nonetheless reasoning steps. I am not arguing that GPT can reason. I personally don't believe that. I am trying to understand how it can generate reasoning steps if it can not reason.