3 ms·

I'm on board with a lot of what's in this deck, but I take issue with the argument on slide 9. Roughly, the probability that an LLM-provided answer is fully cor

by abeppu 4y ago

I'm on board with a lot of what's in this deck, but I take issue with the argument on slide 9. Roughly, the probability that an LLM-provided answer is fully correct decreases exponentially with the length of the answer. I think that's trivially true, but it's also true for human-provided answers (a full non-fiction book is going to have some errors), so it doesn't really get to the core problem with LLMs specifically. In much of the rest of the deck, it's just presumed that any variable named x comes from the world in some generic way, which doesn't really distinguish why those are a better basis for knowledge or reasoning than the linguistic inputs to LLMs. I think we're at the point where people working in these areas need some exposure to the prior work on philosophy of mind and philosophy of language.

elefanten 4y ago
Appreciate your take, not arguing with it. But I’d imagine Lecun has more than passing familiarity with those. This deck was put out with the Philosophy dept and he had a panel debate with NYU profs across depts (inc phil) recently on this topic. I suspect this is all pushing the top Phil Lang and Phil Mind to their limits too. Besides, if those subjects were anywhere near resolved (or even… decently understood), they probably wouldn’t be in the Phil dept any more.
- Grothendank 4y ago
  I feel like Lecun did a poor job of localizing the technique relative to any philosophy, or really explaining what it does for a nontechnical audience. Saying he gives talks to philosophers, or saying this pushes philosophy to its limits doesn't fix the problem that lecun does a poor job - in this presentation - of philosophically motivating the proposal. Perhaps I am wrong, and you can point out exactly how lecun explicates the philosophy in the presentation - perhaps it's really embedded in the maths, which I have not appreciated. Appealing to lecun's authority won't fix the opacity of the presentation. But interpreting it can help! Are you up for it?
  - elefanten 4y ago
    I haven't even given it a deep read, so I unfortunately can't help shed any light. From my quick read, it didn't seem to me that laying out a digestible or rigorous philosophical perspective was really the point here. It seemed more directly biology inspired than philosophy. I also don't seek to valorize Lecun. But he was a very early figure working on these technologies and from the beginning there was a neuroscience-inspired impetus to machine learning. My point was sort of the opposite: that assuming Lecun doesn't clear the bar of having "exposure to the prior work on philosophy of mind and philosophy of language" seems like a weak bet. edit: For clarity, I'm making the assumption that lifelong AI researchers who put time into learning from neuroscience... would also gravitate towards and seek to learn from the nearest relevant branches of philosophy.
gsjbjt 4y ago
The point is that LLMs can’t backtrack after deciding on a token. So the probability at least one token along a long generation will lead you down the wrong path does indeed increase as the sequence gets longer (especially since we typically sample from these things), whereas humans can plan their outputs in advance, revise/refine, etc.
- taylorius 4y ago
  I could be wrong, but I think the only probabilistic component of an LLM is the statistical word fragment selection at the end. Assuming this is true, one could theoretically run the program multiple times, making different fragment choices. This (while horribly inefficient) would allow a sort of backtracking.
- morkalork 4y ago
  To extend the non-fiction book example, the author doesn't write it in a one-shot forward pass, they go back and edit.
- wordpad25 4y ago
  nothing is stopping you from doing the same with LLMs, you can have it go back over it’s response and check it for accuracy
  - andrewprock 4y ago
    What is stopping the LLMs from doing the same?
- bobbylarrybobby 4y ago
  Do you know of any work on holistic response quality in LLMs? We currently have the LLM equivalent of the html line break and hyphenation algorithm, when what we want is the LaTeX version of that algorithm.
- abeppu 4y ago
  Humans can backtrack, but the probability of an "correct" output is still (1-epsilon)^n. Not only can any token introduce an error, but the human author will not perfectly catch errors they have previously introduced. The epsilon ought to be lower for humans, but it's not zero. But more to the point, in the deck provided, Lecun's point is _not_ about backtracking per se. The highlighted / red text on the preceding slide is: > LLMs have no knowledge of the underlying reality > They have no common sense & they can't plan their answer Now, we generally generate from LLMs by sampling uniformly forward, but it isn't hard to use essentially the same structure to generate tokens conditioned on both preceding and following sequences. If you ran generation for tokens 1...n, and then ran m iterations of re-sampling internal token i based on (1..i-1, i+1..n), it would sometimes "fix" issues created initial generation pass. It would sometimes introduce new issues, which were fine upon original generation. Process-wise, it would look a lot like MCMC at generation-time. The ability to "backtrack" does _not_ on its own add knowledge of reality, common sense, or "planning". When a human edits, they're reconciling their knowledge of the world and their intended impact on their expected audience, neither of which the LLM has.
oezi 4y ago
I also don't get why this line of reasoning doesn't take into account that LLMs can just emit a backspace token in the same way that humans can ask for you to discard their statement. The probability of a correct answer doesn't have to decrease in the length of the answer. Epistemologically it certainly makes sense that you need to run a randomized controlled trial at some point to ascertain facts about nature, but alas we humans very rarely do. We primarily consume information and process that.
- Hendrikto 4y ago
  These LLMs are trained with lots and lots of text from many sources, 99.9+% of which do not contain backspaces. They could theoretically output them, given the right tokenizer, but they don‘t. And they especially don‘t do it to correct errors.
  - cma 4y ago
    They could potentially be trained on applying diffs though. Wikipedia has full source control version history of how things evolved, and so do most of the source code corpuses they use like github. Google has keystroke by keystroke history of all google docs, could pull an Adobe Stock and opt everyone in with some TOS update but the blowback would be huge.
HervalFreire 4y ago
That probability formula is too general. It can practically model everything. Yan is just hiding behind epsilon. It's like assigning a probability to the origin of the universe. You can just make shit up with a algebraic letter representing a probability. To illustrate the absurdity of it consider this: I have a theory for everything. I can predict future. The mathematical equation for that is simply (1 - EPSILON) where EPSILON is the probability of any event not happening. What about the probability of a event happening as a result of the first event? Well that's (1 - EPSILON)^2. Since this model can model practically everything it just means everything diverges and everything fundamentally can't be controlled. It's basically just modelling entropy. Really? No. He just tricked himself. The Key here is we need to understand WHAT EPSILON is. We don't, and Yan bringing this formula up ultimately says nothing about anything because it's too general. Not to mention, Every token should have a different probability. You can't have the same epsilon for every token. You have zero knowledge of the probabilities of every single token therefore they cannot share the same variable, unless you KNOW for a fact the probabilities are the same. It should be: P_n(Correct) = (1 - Epsilon_n)^n
- HervalFreire 4y ago
  Error: formula above should be a recurrence relation: P_n(correct) = (1 - Epsilon_n)*P_(n-1)(correct) Where P_0 = (1 - Epsilon_0)
Al-Khwarizmi 4y ago
I may be missing something as I don't claim to be as smart as LeCun, but to me that probabilistic argument is not even wrong. The "probability e that a produced token takes us outside of the set of correct answers" is likely to vary so wildly due to a plethora of factors like filler words vs. keywords, hard vs. easy parts of the question, previous tokens generated (I know abuse of statistical independence assumptions is common and often tolerated, but here it's doing a lot of heavy lifting), parts of the answer that you can express in many ways vs. concrete parts that you must get exactly right, etc. that I don't think simplifying it as he does makes any sense. Yes, I know, abstraction is useful, models are always simplifications, that probability doesn't need to be even close to a constant for the gist of the argument to stand. But everything can be bad in excess and in this case, the simplification is extreme. One can very easily imagine a long answer where the bulk of that probability is concentrated into a single word, while the rest have near-zero probability. Under such circumstances, I don't think his model is meaningful at all. Connecting with your comment, if someone made that kind of claim about humans, I suppose most people would find it ridiculous (or at the very least, meaningless/irrelevant). With LLMs we find it more palatable, because we are primed to think about LLMs in terms of "probability of generating the next token". But I don't think it really makes much more sense for LLMs than for humans, as the problems with the argument are more in how language works than in how the words are generated.

by abeppu 4y ago

elefanten 4y ago

Appreciate your take, not arguing with it. But I’d imagine Lecun has more than passing familiarity with those. This deck was put out with the Philosophy dept and he had a panel debate with NYU profs across depts (inc phil) recently on this topic. I suspect this is all pushing the top Phil Lang and Phil Mind to their limits too. Besides, if those subjects were anywhere near resolved (or even… decently understood), they probably wouldn’t be in the Phil dept any more.

Grothendank 4y ago
I feel like Lecun did a poor job of localizing the technique relative to any philosophy, or really explaining what it does for a nontechnical audience. Saying he gives talks to philosophers, or saying this pushes philosophy to its limits doesn't fix the problem that lecun does a poor job - in this presentation - of philosophically motivating the proposal. Perhaps I am wrong, and you can point out exactly how lecun explicates the philosophy in the presentation - perhaps it's really embedded in the maths, which I have not appreciated. Appealing to lecun's authority won't fix the opacity of the presentation. But interpreting it can help! Are you up for it?
- elefanten 4y ago
  I haven't even given it a deep read, so I unfortunately can't help shed any light. From my quick read, it didn't seem to me that laying out a digestible or rigorous philosophical perspective was really the point here. It seemed more directly biology inspired than philosophy. I also don't seek to valorize Lecun. But he was a very early figure working on these technologies and from the beginning there was a neuroscience-inspired impetus to machine learning. My point was sort of the opposite: that assuming Lecun doesn't clear the bar of having "exposure to the prior work on philosophy of mind and philosophy of language" seems like a weak bet. edit: For clarity, I'm making the assumption that lifelong AI researchers who put time into learning from neuroscience... would also gravitate towards and seek to learn from the nearest relevant branches of philosophy.

gsjbjt 4y ago

The point is that LLMs can’t backtrack after deciding on a token. So the probability at least one token along a long generation will lead you down the wrong path does indeed increase as the sequence gets longer (especially since we typically sample from these things), whereas humans can plan their outputs in advance, revise/refine, etc.

taylorius 4y ago
I could be wrong, but I think the only probabilistic component of an LLM is the statistical word fragment selection at the end. Assuming this is true, one could theoretically run the program multiple times, making different fragment choices. This (while horribly inefficient) would allow a sort of backtracking.
morkalork 4y ago
To extend the non-fiction book example, the author doesn't write it in a one-shot forward pass, they go back and edit.
wordpad25 4y ago
nothing is stopping you from doing the same with LLMs, you can have it go back over it’s response and check it for accuracy
- andrewprock 4y ago
  What is stopping the LLMs from doing the same?
bobbylarrybobby 4y ago
Do you know of any work on holistic response quality in LLMs? We currently have the LLM equivalent of the html line break and hyphenation algorithm, when what we want is the LaTeX version of that algorithm.
abeppu 4y ago
Humans can backtrack, but the probability of an "correct" output is still (1-epsilon)^n. Not only can any token introduce an error, but the human author will not perfectly catch errors they have previously introduced. The epsilon ought to be lower for humans, but it's not zero. But more to the point, in the deck provided, Lecun's point is _not_ about backtracking per se. The highlighted / red text on the preceding slide is: > LLMs have no knowledge of the underlying reality > They have no common sense & they can't plan their answer Now, we generally generate from LLMs by sampling uniformly forward, but it isn't hard to use essentially the same structure to generate tokens conditioned on both preceding and following sequences. If you ran generation for tokens 1...n, and then ran m iterations of re-sampling internal token i based on (1..i-1, i+1..n), it would sometimes "fix" issues created initial generation pass. It would sometimes introduce new issues, which were fine upon original generation. Process-wise, it would look a lot like MCMC at generation-time. The ability to "backtrack" does _not_ on its own add knowledge of reality, common sense, or "planning". When a human edits, they're reconciling their knowledge of the world and their intended impact on their expected audience, neither of which the LLM has.

oezi 4y ago

I also don't get why this line of reasoning doesn't take into account that LLMs can just emit a backspace token in the same way that humans can ask for you to discard their statement. The probability of a correct answer doesn't have to decrease in the length of the answer. Epistemologically it certainly makes sense that you need to run a randomized controlled trial at some point to ascertain facts about nature, but alas we humans very rarely do. We primarily consume information and process that.

Hendrikto 4y ago
These LLMs are trained with lots and lots of text from many sources, 99.9+% of which do not contain backspaces. They could theoretically output them, given the right tokenizer, but they don‘t. And they especially don‘t do it to correct errors.
- cma 4y ago
  They could potentially be trained on applying diffs though. Wikipedia has full source control version history of how things evolved, and so do most of the source code corpuses they use like github. Google has keystroke by keystroke history of all google docs, could pull an Adobe Stock and opt everyone in with some TOS update but the blowback would be huge.

HervalFreire 4y ago

That probability formula is too general. It can practically model everything. Yan is just hiding behind epsilon. It's like assigning a probability to the origin of the universe. You can just make shit up with a algebraic letter representing a probability. To illustrate the absurdity of it consider this: I have a theory for everything. I can predict future. The mathematical equation for that is simply (1 - EPSILON) where EPSILON is the probability of any event not happening. What about the probability of a event happening as a result of the first event? Well that's (1 - EPSILON)^2. Since this model can model practically everything it just means everything diverges and everything fundamentally can't be controlled. It's basically just modelling entropy. Really? No. He just tricked himself. The Key here is we need to understand WHAT EPSILON is. We don't, and Yan bringing this formula up ultimately says nothing about anything because it's too general. Not to mention, Every token should have a different probability. You can't have the same epsilon for every token. You have zero knowledge of the probabilities of every single token therefore they cannot share the same variable, unless you KNOW for a fact the probabilities are the same. It should be: P_n(Correct) = (1 - Epsilon_n)^n

HervalFreire 4y ago
Error: formula above should be a recurrence relation: P_n(correct) = (1 - Epsilon_n)*P_(n-1)(correct) Where P_0 = (1 - Epsilon_0)

Al-Khwarizmi 4y ago

I may be missing something as I don't claim to be as smart as LeCun, but to me that probabilistic argument is not even wrong. The "probability e that a produced token takes us outside of the set of correct answers" is likely to vary so wildly due to a plethora of factors like filler words vs. keywords, hard vs. easy parts of the question, previous tokens generated (I know abuse of statistical independence assumptions is common and often tolerated, but here it's doing a lot of heavy lifting), parts of the answer that you can express in many ways vs. concrete parts that you must get exactly right, etc. that I don't think simplifying it as he does makes any sense. Yes, I know, abstraction is useful, models are always simplifications, that probability doesn't need to be even close to a constant for the gist of the argument to stand. But everything can be bad in excess and in this case, the simplification is extreme. One can very easily imagine a long answer where the bulk of that probability is concentrated into a single word, while the rest have near-zero probability. Under such circumstances, I don't think his model is meaningful at all. Connecting with your comment, if someone made that kind of claim about humans, I suppose most people would find it ridiculous (or at the very least, meaningless/irrelevant). With LLMs we find it more palatable, because we are primed to think about LLMs in terms of "probability of generating the next token". But I don't think it really makes much more sense for LLMs than for humans, as the problems with the argument are more in how language works than in how the words are generated.