4 ms·
This speaks very much to the idea that LLMs are in some sense a ridiculously effective, somewhat lossy, compression algorithm that has been applied to the whole
by gcanyon 8mo ago
This speaks very much to the idea that LLMs are in some sense a ridiculously effective, somewhat lossy, compression algorithm that has been applied to the whole internet.
- vizzier 8mo agoI've thought of them for a while as just a really complicated indexing strategy.
- r_lee 8mo agoI mean, the transformer is basically like a big query engine and the model is the dataset + some logic or whatever it's kind of like that by definition, with the whole Attention stuff etc.
- in-silico 8mo agoIt's a good way to frame base models that have only been pretrained. However, modern frontier models have undergone rounds of fine-tuning, RLHF (reinforcement learning from human feedback), and RLVR (RL from verifiable rewards) that turn them into something else. The compressed internet is still in there, but it's wrapped in problem-solving and people-pleasing circuitry.