4 ms·
> It literally has zero access to the far-enough past. > GPT is completely memoryless: it has access to the past 1024 symbols This is the critique I have of t
by abernard1 6y ago
> It literally has zero access to the far-enough past.
> GPT is completely memoryless: it has access to the past 1024 symbols
This is the critique I have of that.
This model is so huge, the "memory" is encoded in the trained model parameters. Since there is so much repetition and structure in comments in the corpus, I believe we are seeing a large scale social memory compressed in this {model parameters x symbol memory} structure. That space is gigantic.
> If you scale it up to more symbols that effect will go away.
This though still remains to be proven. Does any significant further correlation of note happen when you have {175B model parameters x 1M symbol memory} structure? No one knows quite yet. But we do for sure know that humans can learn things with an extraordinarily smaller "training" set.
- nullc 6y agoI think this thread has lost the point being discussed (much like GPT's limited window!)-- I was replying to this: > If I have a chat with the model and I ask it "what did we talk about thirty minutes ago" it's as clueless as anything. This criticism is true, but would almost certainly be eliminated if the transformer window was increased go back far enough. That wouldn't give it an arbitrary memory, sure. But the specific complaint that it forgets the current conversation doesn't require an arbitrary memory.