I think the paper would similarly claim something like "a chess ai trained only on human games can't play better than the humans who played those games" I thin

I think the paper would similarly claim something like "a chess ai trained only on human games can't play better than the humans who played those games" I think that's false - you could imagine it playing at GM level without making mistakes that real GMs make But it's missing an important fact: chess ais aren't limited to learning only from human games. They can learn from self play. You can extend this to physics (if we can simulate physics precisely enough, there's no reason ai can't make fundamental discoveries in physics)