4 ms·
A dictionary doesn’t give a prediction about what the next word in string will be. GPT-2 does and those predictions can be used so save space.
by croddin 6y ago
A dictionary doesn’t give a prediction about what the next word in string will be. GPT-2 does and those predictions can be used so save space.
- cma 6y agoA dictionary of n-grams does, which I think is what he is describing.
- sidarape 6y agoGPT is way better at next-word predictions than n-grams. However, I agree with OP that the resulting compression scheme is probably not very practical all things considered.
- TheRealPomax 6y agoA dictionary of n-grams does not change its probabilities based on "text seen so far", so you'd need an unbounded n. In practice, n-gram dictionaries don't encode for more than very low values of n, so while yes, abstractly we can say that GPT-2 is equivalent to an n-gram dictionary with some high n, those dictionary cannot be constructed, let alone used, leaving the equivalency is purely theoretical. GPT-2 is not a dictionary.
- cma 6y ago>GPT-2 is not a dictionary. I was only responding to the statement that dictionary methods don't give a prediction of the next word in a string will be. I wasn't in any way saying GPT-2 is a dictionary.