2 ms·
While it is obviously trivially (in the mathematical sense of "trivially") possible to make a Markov chain that produces the same f(input context window) => {ou
by ben_w 2y ago
While it is obviously trivially (in the mathematical sense of "trivially") possible to make a Markov chain that produces the same f(input context window) => {output distribution}, that doesn't make it useful to describe LLMs as a kind of Markov chain.
After all, in the same sense I can also make a Markov chain that reproduces the behaviour of any human, or even any Turing machine with finite tape — it's just that a computer with 1 GB of memory (all kinds including non-volatile) has 2^(8*(10^30)) states.
- benchmarkist 2y agoI've seen this nonsensical equivalence several times now so I'm going to make an example out of you to prove a point. Define the state space for the human Markov chain.
- ben_w 2y ago> Define the state space for the human Markov chain. At most the Berkenstein bound for the mass of a human brain. Is that big? Yes, too huge to bother calculating. That's why I'm using it as an example to say "it's a Markov chain" is not helpful. But it is finite, and that means it's mathematically equivalent. And that's before the thing about transition matrices being (in general if not in particular) square, from any one state to any other. For fun, here's the Berkenstein bound for the equivalent transition matrix for the much smaller GPT-3 with 2048 context length, measured in (amongst other things) multiples of the size of the universe: http://www.wolframalpha.com/input/?i=sqrt%2812%2A%28%2850000%5E20%29%5E2%29%2Aplank%20constant%2AGravitational%20constant%2Aln%282%29%2F%28pi%2A%28speed%20of%20light%5E3%29%29%29 http://www.wolframalpha.com/input/?i=sqrt%2812%2A%28%2850000... (Assuming I didn't fluff the algebra). Edit: I fluffed the agebra, see if you can spot the typo. It's much much bigger.
- benchmarkist 2y agoWhat are the states? I didn't say give me a count of the states. I said give me the algorithm for determining whether any given state of the universe corresponds to whatever human you are reducing to a Markov chain. I don't care if it's practical or not because it seems like you're convinced this algorithm exists even though you have never actually gone through the trouble of actually formalizing it. So I'm doing you a favor, you will either learn about your ignorance or you will make your argument algorithmically formal and actually prove that people are equivalent to Markov chains.
- ben_w 2y agoSeriously? Of all the things I've heard people being dismissive of, you're the first I've heard suggest it might possibly be more than trivial to create a map from physical objects to some enumeration. It's not like simulated physics is somehow a novel field I've had to invent in my head, it's a field that's been performing such mappings, even for quantum systems (on smaller scales than a brain, but you yourself say "I don't care if it's practical or not") for decades already. In extremis, the Bekenstein bound, you've got a surface which — at a resolution of (IIRC) 2π bits per square Planck area — contains all the information about the interior. There's a lot of ways to map the surface of a sphere into Planck areas, pick any you want. That ordering itself defines a state down to the quantum level. There's in the order of 10^17 of those Planck areas if a human brain was compressed into a black hole, so that's the maximum information content of any given possible arrangement of matter and energy with the mass of a human brain. The only thing I'm struggling with right now is why you might be so confident that this is difficult that you're even thinking to try and make an example out of me?
- benchmarkist 2y agoThat's a lot of words and yet I have not seen an actual proof of equivalence. Do you even know the definition of a Markov chain?
- ben_w 2y ago> Do you even know the definition of a Markov chain? Dude, I've written several Markov chains, starting aged something like six while still learning to read. (Family had a Commodore 64, one of the books it came with had one as a coding example). Markov chains are stochastic processes where the transitions between possible nodes have probabilities depending only on the current node, i.e. history is irrelevant. So, right back'atcha, if you can't see how what I wrote is a proof of equivalence — you've got a way to map reality (of a human brain) onto a number between 0 and ~2^(10^17)*. Or perhaps, despite your previous question being answered, your actual issue is that you were unaware that the laws of quantum mechanics give you a stochastic processes where the transitions between possible states have probabilities depending only on the current state? * exponent times whatever the constant of proportionality is
- sigbottle 2y agoSo I'm confused. In my college classes, sometimes we'd get problems involving "construct a markov chain to model this process". Doing the naive, obvious solution wouldn't make the probabilities work because we'd "forget" crucial information, so we'd just add on extra bits to the state (e.g. "flag1 flag2 flag3 ...") that would make the number of nodes exponentially larger. But the infinite information part of markov chains is really helpful sometimes too, e.g. the position of your random walk wouldn't be encoded in your "state" (which could be infinite), it'd just be returned as a value (if you were trying to compute the distance from the origin or something). Are you guys just arguing over "at what point does this shoving a billion bits into my state make it not really a 'markov' chain like you'd traditionally think about it"? Basically the formal definition vs what we intuitively think about as the 'point' of markov processes and why the markov observation is important?
- ben_w 2y ago> Are you guys just arguing over "at what point does this shoving a billion bits into my state make it not really a 'markov' chain like you'd traditionally think about it"? Basically the formal definition vs what we intuitively think about as the 'point' of markov processes and why the markov observation is important? That's an accurate description of my perception of the conversation, I can't speak for the others of course.