At least in the write up he talked about trying with 3 different models. He could look at the progress at each step and evaluate whether it was making progress

At least in the write up he talked about trying with 3 different models. He could look at the progress at each step and evaluate whether it was making progress or stuck. He was able to understand the partial and complete solutions. I think part of what it shows is how much a good value function provides in a search problem. But we’ve always known that brute force is also a valid solution to search problems. It seems that what’s changed is that it’s much easier than before to run brute force, so it’s going to be applied more often and by a base rate argument “solve” more problems. What remains to be seen is whether human + new tech = better than the parts, as has been the case with most previous new technologies.