No, the person you're responding to is absolutely right. The easy test (which has been done in papers again and again) is the ability to train linear probes (or

No, the person you're responding to is absolutely right. The easy test (which has been done in papers again and again) is the ability to train linear probes (or non-linear classifier heads) on the current hidden representations to predict the nth-next token, and the fact that these probes have very high accuracy.