It depends what they mean by automating. But the classic argument goes: * Each new generation of models has emergent capabilities we did not anticipate. * We

It depends what they mean by automating. But the classic argument goes: * Each new generation of models has emergent capabilities we did not anticipate. * We already have trouble monitoring and controlling the current generation (see HuggingFace incident). * The more we let models shape their successors, the more out-of-distribution each generation's learning environment becomes. * If not done carefully, we risk creating extraordinarily intelligent and powerful models with unintended behaviors, like deceptiveness or power-seeking.