4 ms·
LLMs don't memorize and reuse. They don't really have a memory of any kind. I think the problem is they're more different from human learning than you think.
by olddustytrail 3y ago
LLMs don't memorize and reuse. They don't really have a memory of any kind. I think the problem is they're more different from human learning than you think.
- mitthrowaway2 3y agoThe LLM's trained parameters are a lossy memory of their training data.
- olddustytrail 3y agoNo, they aren't.
- mitthrowaway2 3y agoCare to explain why not? An LLM's working memory is just its context window, but the LLM also has embedded data in its parameters, which are set during training to minimize loss. This is effectively a memory of the training data, just as much as a digital photo of my face is a memory of the photons reflecting from my face.
- Der_Einzige 3y agoWhy would you go on HN and say blatantly wrong information AI is literally 1-1 equivalent to compression. If you don't believe me, you should check out this demo of GPT-2 as a (for awhile SOTA) text compressor. Oh its down now, but here's the HN thread to prove this exists: https://news.ycombinator.com/item?id=23618465 https://news.ycombinator.com/item?id=23618465