Sure - one only needs a sufficiently powerful learning architecture to extract the signal from the data, and assuming that is the case then what is learnt depen

Sure - one only needs a sufficiently powerful learning architecture to extract the signal from the data, and assuming that is the case then what is learnt depends on the data not the architecture. That said, I don't think that LLMs are yet close to fully utilizing the training data. e.g. they are prone to hallucinating due to not thinking ahead and backing themselves into a corner where they have to say something. One obvious improvement for that one is more forward looking prediction (i.e. "thinking ahead" - engage brain before opening mouth), which for LLMs can be addressed by tree of thought rollouts and RL learning. So, while architecture is not important if all architectures are equally powerful (in which case you'll learn what's available to learn), it certainly does matter if not all are fully up to the job, as it would appear none currently are.