3 ms·
Is there not a way for me to express the crucial difference between non-language character repetition (or repetition of any string) and the ability or the abili
by DiggyJohnson 3y ago
Is there not a way for me to express the crucial difference between non-language character repetition (or repetition of any string) and the ability or the ability to interpret and respond to human language in the same language?
I just feel like we're not in a position to even begin understanding our disagreement until you at least recognize the question or point I'm trying to get here. If you disagree with the question, or don't understand it: why?
- JoshuaDavid 3y agoI don't think that's where the difference lies, no. Sampling from a markov model built from English n-gram frequencies will produce non-repeated, mostly grammatical English text, which frequently is even sensible-sounding once n >= 4 or so. But I don't think there's anything going on in language generation beyond "based on the current context, produce an appropriate output token for that context based on the observed and inferred distribution of training inputs". I think that's also how human language generation works, though "the context" for humans includes a lot more than just a few thousand words of text. But I think that the surprising thing about e.g. GPT-4 is how well it does the thing, rather than the fact that it does the thing at all.