4 ms·

It doesn’t sound like you understand what a Markov chain nor an LLM is, you’re just parroting something you heard and that stuck in your head. Please don’t?

ftxbro 4y ago
I mean if you read the definitions you can see that it's a Markov chain. I don't know what else to tell you. Maybe you have the idea that only low order Markov chains, like second or third order Markov chains are "true" Markov chains and that 2048th order Markov chains shouldn't be called Markov chains anymore? Maybe you think that the transition matrix must be constructed explicitly as a square of numbers stored in the memory of the computer for it to count as a Markov chain? It's not true, you can have the transition matrix be implicit and it's still a Markov chain. Maybe you think not just any transition matrix is allowed for a Markov chain, and that the ones for LLMs are (implicitly) defined in ways that disallow it from being a Markov chain? For example, maybe you think the transition probabilities are required to be constructed by counting how many times the next token followed the context in the training text. Well that wouldn't be true, you can for example make a transition matrix that gives nonzero probability to a token even if that one has never followed its context in the training data, and it would still be a Markov chain. Maybe you think that if I'm defending someone who calls an LLM a Markov chain, then I'm also defending the "stochastic parrot" take and the takes of ones like Gary Marcus. Well that's not true. I can recognize the LLM is a gigantic Markov chain and I also think the "stochastic parrot" take isn't a good one. My favorite take is the simulators one https://generative.ink/posts/simulators/ https://generative.ink/posts/simulators/
- jdthedisciple 4y ago
  Markov Chains are analytical models, as opposed to LLMs. Big difference.
  - ftxbro 4y ago
    An LLM (one that for example gives a probability distribution over tokens that follow a 2048 token context sequence) is a giant high-order Markov chain with an implicit transition matrix. I guess you are hung up on the fact that the full transition matrix of an LLM is never actually constructed in memory in closed form. Maybe according to your definition that makes it not an 'analytical model'.
    - jdthedisciple 4y ago
      besides, where did you get the 2048 token number from?
      - ftxbro 4y ago
        The LLM GPT-3 for example has a 2048-token-long context.
    - deleted 4y ago
      [deleted]

ftxbro 4y ago

I mean if you read the definitions you can see that it's a Markov chain. I don't know what else to tell you. Maybe you have the idea that only low order Markov chains, like second or third order Markov chains are "true" Markov chains and that 2048th order Markov chains shouldn't be called Markov chains anymore? Maybe you think that the transition matrix must be constructed explicitly as a square of numbers stored in the memory of the computer for it to count as a Markov chain? It's not true, you can have the transition matrix be implicit and it's still a Markov chain. Maybe you think not just any transition matrix is allowed for a Markov chain, and that the ones for LLMs are (implicitly) defined in ways that disallow it from being a Markov chain? For example, maybe you think the transition probabilities are required to be constructed by counting how many times the next token followed the context in the training text. Well that wouldn't be true, you can for example make a transition matrix that gives nonzero probability to a token even if that one has never followed its context in the training data, and it would still be a Markov chain. Maybe you think that if I'm defending someone who calls an LLM a Markov chain, then I'm also defending the "stochastic parrot" take and the takes of ones like Gary Marcus. Well that's not true. I can recognize the LLM is a gigantic Markov chain and I also think the "stochastic parrot" take isn't a good one. My favorite take is the simulators one https://generative.ink/posts/simulators/ https://generative.ink/posts/simulators/

jdthedisciple 4y ago
Markov Chains are analytical models, as opposed to LLMs. Big difference.
- ftxbro 4y ago
  An LLM (one that for example gives a probability distribution over tokens that follow a 2048 token context sequence) is a giant high-order Markov chain with an implicit transition matrix. I guess you are hung up on the fact that the full transition matrix of an LLM is never actually constructed in memory in closed form. Maybe according to your definition that makes it not an 'analytical model'.
  - jdthedisciple 4y ago
    besides, where did you get the 2048 token number from?
    - ftxbro 4y ago
      The LLM GPT-3 for example has a 2048-token-long context.
  - deleted 4y ago
    [deleted]