If I understand correctly, they're using an LLM to write a series of computer programs exploring a large solution space and feeding the output of those programs

If I understand correctly, they're using an LLM to write a series of computer programs exploring a large solution space and feeding the output of those programs into a separate validation program written by a human. To take the calculator analogy, it's more like having the human give a set of constraints on a solution and having the calculator decide what buttons to push. Real progress is always incremental. I wouldn't be surprised if 5-10 years from now we have similar kinds of systems discovering new materials or new candidates for dark matter.