4 ms·

I wonder how this would behave if trained on 1 trillion words like the LLMs. Also, Is training a Markov cheaper that training neural nets? It would be a great

by max_ 3y ago

I wonder how this would behave if trained on 1 trillion words like the LLMs. Also, Is training a Markov cheaper that training neural nets? It would be a great way to cut AI costs if they could be made as effective as neural nets. There is also an interesting post by Orange duck. [0] [0] https://theorangeduck.com/page/17-line-markov-chain https://theorangeduck.com/page/17-line-markov-chain

stabbles 3y ago
Yeah, a century old approach using frequency tables where the next word is generated based just on the previous two words is highly competitive with contemporary neural nets. To produce coherent sentences, do you ever need to remember more than the last two words you said? I doubt it.
- CamperBob2 3y ago
  Now ask it to render a deepfake of Noam Chomsky waving his hands saying "Nothing to see here."
gattilorenz 3y ago
The only way to make it sound less like a word salad is to increase the n-gram length (e.g. use the last 10 words to predict the next), but at that point it starts repeating the corpus. I guess technically you can train on a huge corpus like those of NNs to mitigate that, but you’ll end up with more refined word salads then (edit: refined as in “locally coherent but globally still a word salad”)
- sgt101 3y ago
  well - the lawyers from the NYT called, and they disagree...
  - gattilorenz 3y ago
    I didn’t get the call, so it’s not clear what they disagree on :) I doubt the have anything to say about Markov chains. We’re talking about technical possibilities, not legality of training corpora
    - sgt101 3y ago
      The NN's trained on huge corpus's appear to be prone to memorization as well.
- microtonal 3y ago
  Yes, the problem with such long context lengths is sparsity - you still don't have enough data points to make representative many-gram distributions. This was exactly the motivation for Bengio and others to propose neural language models in 2003: https://dl.acm.org/doi/10.5555/944919.944966 https://dl.acm.org/doi/10.5555/944919.944966 I recommend everyone interested in neural language models and/or wondering why we can't scale up Markov models by just using longer ngrams to read this paper. It's pretty accessible and explains how we got to where we are now in 2023.

by max_ 3y ago

stabbles 3y ago

Yeah, a century old approach using frequency tables where the next word is generated based just on the previous two words is highly competitive with contemporary neural nets. To produce coherent sentences, do you ever need to remember more than the last two words you said? I doubt it.

CamperBob2 3y ago
Now ask it to render a deepfake of Noam Chomsky waving his hands saying "Nothing to see here."

gattilorenz 3y ago

The only way to make it sound less like a word salad is to increase the n-gram length (e.g. use the last 10 words to predict the next), but at that point it starts repeating the corpus. I guess technically you can train on a huge corpus like those of NNs to mitigate that, but you’ll end up with more refined word salads then (edit: refined as in “locally coherent but globally still a word salad”)

sgt101 3y ago
well - the lawyers from the NYT called, and they disagree...
- gattilorenz 3y ago
  I didn’t get the call, so it’s not clear what they disagree on :) I doubt the have anything to say about Markov chains. We’re talking about technical possibilities, not legality of training corpora
  - sgt101 3y ago
    The NN's trained on huge corpus's appear to be prone to memorization as well.
microtonal 3y ago
Yes, the problem with such long context lengths is sparsity - you still don't have enough data points to make representative many-gram distributions. This was exactly the motivation for Bengio and others to propose neural language models in 2003: https://dl.acm.org/doi/10.5555/944919.944966 https://dl.acm.org/doi/10.5555/944919.944966 I recommend everyone interested in neural language models and/or wondering why we can't scale up Markov models by just using longer ngrams to read this paper. It's pretty accessible and explains how we got to where we are now in 2023.