4 ms·
It’s funny that people say it’s a glorified Markov chain. No. It’s not. A Markov chain has some very specific properties that are absolutely not fulfilled by G
by toxik 5y ago
It’s funny that people say it’s a glorified Markov chain.
No. It’s not. A Markov chain has some very specific properties that are absolutely not fulfilled by GPT-3 models.
Just say “stochastic” if you want a buzzword. Stop appropriating Markov chains.
- sillysaurusx 5y ago“It’s a stochastic” doesn’t flow, though I guess I could use "stochastic random walk." What properties does a gpt-3 model have that a Markov chain doesn’t? (Other than effectiveness.)
- exdsq 5y agoANNs aren't Markov chains
- sillysaurusx 5y agoWell, yes. And apples aren't oranges, but they share a lot of similar traits. "Given a prompt, provide a completion" is what a Markov chain does. GPT-3 is exactly the same, in the sense that apples and oranges both satisfy your hunger.
- tomrod 5y agoIs calculator.exe the equivalent to TI-84?
- sillysaurusx 5y agoI would say that a TI-84 is a glorified calculator.exe. The TI-84 is usually more effective, but they’re both calculators.
- toxik 5y agoWillfully misunderstanding your interlocutor seems ungracious, you apparently know stochastic is an adjective, so why did you try to use it as a noun? GPT-3 is conditioned on the entire input sequence as well as its own output, which is strictly NON-MARKOVIAN. In fact, the point in saying something is Markovian is exactly that: the state transition probability only depends on the current state.
- immibis 5y agoBut the current state could include all of the last N output tokens.
- immibis 5y agoActually GPT-3 is a Markov chain. A Markov chain is a very general term. Just because the simplest ones are stupid, doesn't mean there aren't smarter ones. A Markov chain is a model where: (1) there is a state, (2) the probabilities of the next state depend only on the current state. That could describe anything from a wet fart to a deterministic computer to GPT-3 to the human mind to quantum mechanics (the real one, not a simulation). If I unplug the Internet and all USB devices, my computer is a Markov chain with at least several trillion bits of state, so 2^(several trillion) possible states. And there is one next state, which has a probability of 1, and all other possible states have a probability of 0. That's a Markov chain. GPT models choose the next word probabilistically, with the probabilities chosen by feeding the previous N words into a neural network. That sounds like a Markov chain to me!