5 ms·
The reason this works is because LLMs are trained to minimize the empirical risk over a large set of data (https://en.wikipedia.org/wiki/Empirical_risk_minimiza
by tysam_and 3y ago
The reason this works is because LLMs are trained to minimize the empirical risk over a large set of data (https://en.wikipedia.org/wiki/Empirical_risk_minimization https://en.wikipedia.org/wiki/Empirical_risk_minimization), and it is where the log-likelihood-based measure comes from, oftentimes represented simply as the perplexity.
This is precisely because LLMs learn a representation of the nth-order Markov Chain, which by its nature is extraordinarily sparse. This allows for excellent compression, and I would assume (and have for a while, I suppose) the nearly-linear nature of LLMs allows for excellent interpolation between these very sparse states.
This lets us 'break' the hard-combinatorial problem into one that can be softly, albeit with the fuzzy error that comes with any estimator.
Since computers cannot tractably compress markov chains past a certain point, this allows LLMs some opportunity to outpace traditional methods for compression, at least in theory.
Also, this method technically cheats since you need the language model to decompress, which counts towards your dictionary size. In a truly unbounded case, I would be interested to see which method wins. I'm assuming it's the LLM.
Apologies for any factual inaccuracies or such, I am still learning many of these things. Many thanks and much love! <3 :))))
- blovescoffee 3y agoIf you want to learn something cool and related, checkout Autoencoders (or VAEs). They effectively compress information by forming representations of some data.
- tysam_and 3y agoIndeed. They are extremely cool! <3 :) I think in a way, every neural network works on compression! Though AEs & VAEs are a bit more blatant in how they do it. It's just always turned around a little, here and there. I have a pet theory that no neural network can work without compression (even generative ones that have constantly increasing layer depths! :D :)))) )
- nerdponx 3y agoIn a very loose sense, even plain linear regression is a form of lossy compression in the form of an orthogonal projection.
- tysam_and 3y agoOh, I appreciate this point, thanks for making it! I like it a lot. <3 :)))) I think, perhaps, if inputs > outputs and there is some dimensionality reduction (though I think orthogonality would be a trait of an ideal system [i.e. an emergent property approached in the limit], not one that is explicitly enforced each step).
- nl 3y agoThe link between compression and intelligence is a popular theory. Indeed the linked work here came out of the Hutter Prize: > The goal of the Hutter Prize is to encourage research in artificial intelligence (AI). The organizers believe that text compression and AI are equivalent problems. Hutter proved that the optimal behavior of a goal-seeking agent in an unknown but computable environment is to guess at each step that the environment is probably controlled by one of the shortest programs consistent with all interaction so far. https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize
- tysam_and 3y agoI agree that it is most certainly necessary to have compression to have intelligence, after all, it is the bridge from empirical examples to a learned policy. Oftentimes the subtle switch comes when people say that compression IS intelligence. Similar to how metabolism in many ways is required for life, yet metabolism itself isn't life. One of the challenges is that the minimum MDL (Minimum Descriptor Length -- https://en.wikipedia.org/wiki/Minimum_description_length https://en.wikipedia.org/wiki/Minimum_description_length) is intractable to prove directly, we can only prove that we are a bit closer to it than we were before. This of course becomes even more difficult in the temporal regime, as the amount of information to prove that an iterative mapping (i.e. a decision-making algorithm or what have you, in this case) over each time slice in a temporal system is nigh-impossible. We can tell _something_ about it because of the attractors generated by such a system, but even then, I think it's something basically impossible to do in a closed-form manner. That being said, I do believe compression is required for intelligence, and that deliciously drags in all of the info theory stuff, which is very fun indeed. Just seems like it gets really messy with that time component, but that's just my 2 cents at least. :'( I didn't know that was related to the Hutter Prize. Very cool. I'd read a little bit about that prize before, and I'll take another look at it now. Probably won't start any work on anything because I really, really, really do not want to Collatz Conjecture myself again.
- angch 3y agoNNCP (Bellard's prelude to ts_zip, using similar techniques) is not qualified for Hutter Prize, btw, because hardware and speed limitations specified by Hutter Prize. "Must run in ≲50 hours using a single CPU core and <10GB RAM and <100GB HDD on our test machine." Which is an Intel Core i7-620M
- fnordpiglet 3y agoI would note you can externalize your dictionary in a lot of schemes. For tight protocol level encoding in my custom protocols I often build the dictionary as part of compilation and deploy it compiled into both sides. It saves a lot of overhead on small message protocols. In this case the dictionary is probably absurdly large, but I think we are at a point where new compression techniques are going to have weird and unexpected trade offs.
- tysam_and 3y agoYes, in this case I'm counting the mathematical definition for a decoder that's required when compressing large corpii of text.