The lack of iterative decision making is definitely a huge problem in studies like this. My own experience with GPTs for debugging software show that iterative

The lack of iterative decision making is definitely a huge problem in studies like this. My own experience with GPTs for debugging software show that iterative performance is honestly atrocious. The machine will go all-in on dead-end solutions, forget the context of the problem, and fail to rule things out that should be obvious. Even very simple tasks like locating a misconfigured IDE setting have just awful performance in real world iterative problem solving. It’s not to say the GPTs are currently useless though, they can provide a starting off point or remind us of things we may have missed. Full iterative diagnosis seems really far off though.