Sufficiently advanced random mixture of features extracted from real photos is indistinguishable from genuine understanding. That's why I don't understand this

Sufficiently advanced random mixture of features extracted from real photos is indistinguishable from genuine understanding. That's why I don't understand this argument. It's not true - you can make the AI generate specific samples, but most samples can not be found in the training data - but even if it was true, it wouldn't prove anything anyways, because you can always say "well the features it copies are just smaller" until the features are just combinations of pixels. It's unfalsifiable even for human artists.