More specifically, the article assumes that the only way "AI"s will ever be produced is the way that LLMs are -- by being trained off pre-existing human generat

More specifically, the article assumes that the only way "AI"s will ever be produced is the way that LLMs are -- by being trained off pre-existing human generated media, and existing as static objects after that. In particular, this assumes that the system doesn't subsequently modify itself based on further interactions with the real world (e.g., doing experiments, and taking note of the results). That's assuming an awful lot, to put it mildly.