3 ms·
The only way to make it sound less like a word salad is to increase the n-gram length (e.g. use the last 10 words to predict the next), but at that point it sta
by gattilorenz 3y ago
The only way to make it sound less like a word salad is to increase the n-gram length (e.g. use the last 10 words to predict the next), but at that point it starts repeating the corpus.
I guess technically you can train on a huge corpus like those of NNs to mitigate that, but you’ll end up with more refined word salads then (edit: refined as in “locally coherent but globally still a word salad”)
- sgt101 3y agowell - the lawyers from the NYT called, and they disagree...
- gattilorenz 3y agoI didn’t get the call, so it’s not clear what they disagree on :) I doubt the have anything to say about Markov chains. We’re talking about technical possibilities, not legality of training corpora
- sgt101 3y agoThe NN's trained on huge corpus's appear to be prone to memorization as well.
- microtonal 3y agoYes, the problem with such long context lengths is sparsity - you still don't have enough data points to make representative many-gram distributions. This was exactly the motivation for Bengio and others to propose neural language models in 2003: https://dl.acm.org/doi/10.5555/944919.944966 https://dl.acm.org/doi/10.5555/944919.944966 I recommend everyone interested in neural language models and/or wondering why we can't scale up Markov models by just using longer ngrams to read this paper. It's pretty accessible and explains how we got to where we are now in 2023.