5 ms·

I've never seen a Markov chain do anything like GPT4. I'm not sure how you can say with a straight face they are basically the same.

PheonixPharts 3y ago
LLMs are Markov chains in latent space, it's the latent representation that give them their power, but ultimately there's not as much difference as one would suspect.
- esafak 3y ago
  They're different because Markov models are stateless whereas LLMs are stateful. https://en.wikipedia.org/wiki/Markov_property https://en.wikipedia.org/wiki/Markov_property
  - Jensson 3y ago
    Current LLMs are stateless as far as we know, their state when computing a new token is only the preceding text tokens, they don't store any metadata or save state from the previous calculations.
adamnemecek 3y ago
No one has spent 100M on training Markov chains.
- hiddencost 3y ago
  Google did it. Amazon did it. Plenty of others did. What do you think they were doing before recurrent neural networks?
  - ben_w 3y ago
    100 million USD? No. I'd believe $40 for the energy cost, $120 for very slightly increased wear on their hard drives, and $400 for one engineer's 20% time project for one week. And that's if it was trained on Google's entire internet cache rather than, say, just a Wikipedia snapshot from 2004, which sounds like the kind of thing that Google might have set as a pre-interview coding challenge.
- edgyquant 3y ago
  For the first few years of smartphones markov chains were used for autocorrect.
- Al-Khwarizmi 3y ago
  Not 100M I guess, but this is fresh from today's arXiv: "Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens" https://arxiv.org/pdf/2401.17377.pdf https://arxiv.org/pdf/2401.17377.pdf
- Sohcahtoa82 3y ago
  The trick with Markov chains is that you don't need to. Markov Chains are dead simple. There's not really a "training" as much as it's simply reading data and collecting statistics. They're so simple that you can probably build one nearly as fast as you can read the training data.
- ben_w 3y ago
  I think it would take some spectacularly bad engineering to be that wasteful. It would need to be so inefficient that getting ChatGPT to write the code won't be bad enough.
- esafak 3y ago
  It would make no sense; they are not powerful models worthy of the spend.
thwarted 3y ago
A LLM is a Markov chain with billions of associations and weights. A Makov chain is an LLM of maybe a few dozen associations and weights (so an LM, without the first L). The difference is in the data structure and the size of the atoms/n-grams. The data structure Markov chain implementations use is not efficient for billions of parameters, either in storage or in processing. But the idea is the same: give a likely next token given the last n tokens. The value for n is a narrow window for Markov chains and an extremely wide window for LLM. LLM are able to maintain massive amounts of state compared to a Markov chain implementation.
- causalmodels 3y ago
  An LLM is a Markov chain in the same sense that a cat is a tiger, technically true but it misses the qualia.
  - candiodari 3y ago
    It's not. There's fundamental architectural differences that couldn't be bigger. A better comparison would be that it's like a windup toy versus a group of humans moving an entire civilization. They both move along a distance, but just listing the systems that the human group has that the windup toy doesn't is too long to fit on a page.
    - Jensson 3y ago
      > It's not. There's fundamental architectural differences that couldn't be bigger. LLM architecture is a markov chain to the core. It isn't a lookup table like old markov chains but it is still a markov chain: next word prediction based on previous words.
      - fjkdlsjflkds 3y ago
        Thanks for repeating this. Seems like most people fail to understand that LLMs (as they are implemented these days) are markov chains by definition, regardless of how much "better" they are compared to "Dissociated Press"-style markov chains based on lookup tables. > A Markov chain or Markov process is a stochastic model describing a sequence of possible events in which *the probability of each event depends only on the state attained in the previous event*. Is the process calculating a probability distribution over "next token", based on a bounded-size context of "previous tokens"? Yes? Then it is a markov chain, by definition. It's like saying that a "human" is not an "animal", since it is so much "better" and "capable" than (other/usual) animals. The more you argue, the more I'll be convinced that you either don't know what the definition of a "human" is, or that you don't know what the definition of an "animal" is (or both).
idontpost 3y ago
[dead]
k__ 3y ago
Fair, but maybe it's more of a computer science-cy type of comparison? We say systems can perform the same types of computations if they're both Turing complete. Yet, we wouldn't implement everything in every "language" that is Turing complete. Perhaps, every LLM could be represented as a Markov chain, and for some it even makes sense (e.g., easier to train, easier to reason about), but in most cases it's a bad idea (e.g., expensive, bad performance).

PheonixPharts 3y ago

LLMs are Markov chains in latent space, it's the latent representation that give them their power, but ultimately there's not as much difference as one would suspect.

esafak 3y ago
They're different because Markov models are stateless whereas LLMs are stateful. https://en.wikipedia.org/wiki/Markov_property https://en.wikipedia.org/wiki/Markov_property
- Jensson 3y ago
  Current LLMs are stateless as far as we know, their state when computing a new token is only the preceding text tokens, they don't store any metadata or save state from the previous calculations.

adamnemecek 3y ago

No one has spent 100M on training Markov chains.

hiddencost 3y ago
Google did it. Amazon did it. Plenty of others did. What do you think they were doing before recurrent neural networks?
- ben_w 3y ago
  100 million USD? No. I'd believe $40 for the energy cost, $120 for very slightly increased wear on their hard drives, and $400 for one engineer's 20% time project for one week. And that's if it was trained on Google's entire internet cache rather than, say, just a Wikipedia snapshot from 2004, which sounds like the kind of thing that Google might have set as a pre-interview coding challenge.
edgyquant 3y ago
For the first few years of smartphones markov chains were used for autocorrect.
Al-Khwarizmi 3y ago
Not 100M I guess, but this is fresh from today's arXiv: "Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens" https://arxiv.org/pdf/2401.17377.pdf https://arxiv.org/pdf/2401.17377.pdf
Sohcahtoa82 3y ago
The trick with Markov chains is that you don't need to. Markov Chains are dead simple. There's not really a "training" as much as it's simply reading data and collecting statistics. They're so simple that you can probably build one nearly as fast as you can read the training data.
ben_w 3y ago
I think it would take some spectacularly bad engineering to be that wasteful. It would need to be so inefficient that getting ChatGPT to write the code won't be bad enough.
esafak 3y ago
It would make no sense; they are not powerful models worthy of the spend.

thwarted 3y ago

A LLM is a Markov chain with billions of associations and weights. A Makov chain is an LLM of maybe a few dozen associations and weights (so an LM, without the first L). The difference is in the data structure and the size of the atoms/n-grams. The data structure Markov chain implementations use is not efficient for billions of parameters, either in storage or in processing. But the idea is the same: give a likely next token given the last n tokens. The value for n is a narrow window for Markov chains and an extremely wide window for LLM. LLM are able to maintain massive amounts of state compared to a Markov chain implementation.

causalmodels 3y ago
An LLM is a Markov chain in the same sense that a cat is a tiger, technically true but it misses the qualia.
- candiodari 3y ago
  It's not. There's fundamental architectural differences that couldn't be bigger. A better comparison would be that it's like a windup toy versus a group of humans moving an entire civilization. They both move along a distance, but just listing the systems that the human group has that the windup toy doesn't is too long to fit on a page.
  - Jensson 3y ago
    > It's not. There's fundamental architectural differences that couldn't be bigger. LLM architecture is a markov chain to the core. It isn't a lookup table like old markov chains but it is still a markov chain: next word prediction based on previous words.
    - fjkdlsjflkds 3y ago
      Thanks for repeating this. Seems like most people fail to understand that LLMs (as they are implemented these days) are markov chains by definition, regardless of how much "better" they are compared to "Dissociated Press"-style markov chains based on lookup tables. > A Markov chain or Markov process is a stochastic model describing a sequence of possible events in which *the probability of each event depends only on the state attained in the previous event*. Is the process calculating a probability distribution over "next token", based on a bounded-size context of "previous tokens"? Yes? Then it is a markov chain, by definition. It's like saying that a "human" is not an "animal", since it is so much "better" and "capable" than (other/usual) animals. The more you argue, the more I'll be convinced that you either don't know what the definition of a "human" is, or that you don't know what the definition of an "animal" is (or both).

idontpost 3y ago

[dead]

k__ 3y ago

Fair, but maybe it's more of a computer science-cy type of comparison? We say systems can perform the same types of computations if they're both Turing complete. Yet, we wouldn't implement everything in every "language" that is Turing complete. Perhaps, every LLM could be represented as a Markov chain, and for some it even makes sense (e.g., easier to train, easier to reason about), but in most cases it's a bad idea (e.g., expensive, bad performance).