5 ms·
> GPT-2 is not just a dictionary, it's enormous non-linear function. Dictionaries are non-linear functions. > This compressor gets compression levels which a
by blackbear_ 6y ago
> GPT-2 is not just a dictionary, it's enormous non-linear function.
Dictionaries are non-linear functions.
> This compressor gets compression levels which are unachievable with a dictionary.
I don't see why this is the case, care to elaborate?
- croddin 6y agoA dictionary doesn’t give a prediction about what the next word in string will be. GPT-2 does and those predictions can be used so save space.
- cma 6y agoA dictionary of n-grams does, which I think is what he is describing.
- sidarape 6y agoGPT is way better at next-word predictions than n-grams. However, I agree with OP that the resulting compression scheme is probably not very practical all things considered.
- TheRealPomax 6y agoA dictionary of n-grams does not change its probabilities based on "text seen so far", so you'd need an unbounded n. In practice, n-gram dictionaries don't encode for more than very low values of n, so while yes, abstractly we can say that GPT-2 is equivalent to an n-gram dictionary with some high n, those dictionary cannot be constructed, let alone used, leaving the equivalency is purely theoretical. GPT-2 is not a dictionary.
- cma 6y ago>GPT-2 is not a dictionary. I was only responding to the statement that dictionary methods don't give a prediction of the next word in a string will be. I wasn't in any way saying GPT-2 is a dictionary.