I wonder if this relation still holds with newer models that have have even more compute thrown at them? My intuition is that the structure inherent to languag

I wonder if this relation still holds with newer models that have have even more compute thrown at them? My intuition is that the structure inherent to language makes Word2Vec possible. Then training on terabytes of human text encoded with Word2Vec + Positional Encoding makes it possible to then have the ability to predict the next encoding at superhuman levels of cognition (while training!). It's my sense that the bag of words (as input/output method) combined with limited context windows (to make Positional Encoding work) is a huge impedance mismatch to the internal cognitive structure. Thus I think that given the orders of magnitude more compute thrown at GPT-4 et al, it's entirely possible new forms of representation evolved and remain to be discovered by humans probing through all the weights. I also think that MemGPT could, eventually, become an AGI because of the unlimited long term memory. More likely, though, I think it would be like the protagonist in Memento[1]. [1] https://en.wikipedia.org/wiki/Memento_(film) https://en.wikipedia.org/wiki/Memento_(film) [edit - revise to address question]