> I feel this misses the point and I think it dismisses attention heads and transformers Those just makes it better at completing the text, but for very common

> I feel this misses the point and I think it dismisses attention heads and transformers Those just makes it better at completing the text, but for very common riddles those tools still gets easily overruled by pretty simple text completion logic since the weights for those will be so extremely strong. The point is that if you understand its a text completer then its easy to understand why it fails at these. To fix these properly you need to make it no longer try to complete text, and that is hard to do without breaking it.