4 ms·
> But how can a LLM not know the answer if it was trained on essentially the entire internet of data and certainly most likely all the data in IMDB? The LLM do
by acchow 2y ago
> But how can a LLM not know the answer if it was trained on essentially the entire internet of data and certainly most likely all the data in IMDB?
The LLM doesn't memorize the input during training. If it encounters the same information a few times, it has a higher chance of getting compressed into the network. But a tiny nudge along a gradient descent does not store all the input.
- mr_toad 2y ago> getting compressed into the network The idea that neural network weights are compressed data is a distressingly common human hallucination.
- Terr_ 2y agoThe overlaps are obvious, so on what grounds are you claiming the overall system not like compression? (Dynamic, lossy, predictive.)
- mr_toad 2y agoNeural networks are no more like compressed data than a Huffman tree, or frequency counts of letters, or the weights of a linear model, or even a sample average. You can use them to build a compression algorithm (and people have), but they are not compressed data.
- comex 2y agoNeural networks can repeat significant amounts of their training data verbatim or near-verbatim. None of the other things you mentioned are capable of doing that. They don’t store nearly enough information.