"That’s basically the claim of the “Blurry JPEG of the Web” article. I agree that to a large degree that’s probably what’s going on here." I strongly disagree

"That’s basically the claim of the “Blurry JPEG of the Web” article. I agree that to a large degree that’s probably what’s going on here." I strongly disagree with that view, and encourage reading the "Sparks of Artificial General Intelligence" paper[1] and watching the associated video[2], then just experiment with GPT4 using novel information that has been created after its training data cutoff and you'll see it reason about it. It can also output novel creative works about things that didn't exist in its training data. Sure, there's some "compression" going on in LLMs, but that's far from everything that's going on. [1] - https://arxiv.org/abs/2303.12712 https://arxiv.org/abs/2303.12712 [2] - https://m.youtube.com/watch?v=qbIk7-JPB2c https://m.youtube.com/watch?v=qbIk7-JPB2c