4 ms·

To be clear, did you have any trouble figuring out the correct answer in the motivating example in the article's introduction? I'm asking because sure, a human

To be clear, did you have any trouble figuring out the correct answer in the motivating example in the article's introduction? I'm asking because sure, a human programmer might well get tripped up by a name swap if they had missed the swap, or forgot it, but when presented with a short test of three lines like the one in the article's motivating example, the majority of human programmers would pass the test easily. That's what the experiment is testing. Three short lines and a question. The LLMs fail it. How often would humans fail it? (Note I think the experiment is not very informative for other reasons, but I'm specifically pointing out that most humans would pass the test anyway).

kenjackson 3y ago
I’m not sure MOST humans would pass the test. Maybe most decent SW engineers would.
TeMPOraL 3y ago
My take is: does a human programmer in this case would have to consciously talk to themselves to not make a mistake? "Remember, remember, 'el eye en' prints, 'pee ar aye en tee' computes length". Would they have to double-check their output? If yes, then don't expect LLMs to do better in general case, unless you make them do those steps. This is what I mean when I keep saying that a good analogy for current LLMs, GPT-4 in particular, is the "inner voice" - that bit in your head that spits out streams of thought using language. If you look at LLM output as you'd look at the thoughts popping into your head when you read the prompt, before you consciously fix/filter/"return to sender" them, I think you'd find the two very similar. My objection to the article's test is thus that it's intentionally set up to confuse one's gut feel. "len" and "print" are not opaque tokens, they are words with whole lot of associations, and those associations are precisely why they were chosen to name those specific functions. Even with a clear problem statement, at intuitive level, your intuition would still cling to those associations (and heaps of Python code you may have read and wrote in your life) - you need to override that at a conscious level. Since I claim that LLM output is equivalent to the thoughts produced by intuition, and there is no "conscious level" equivalent (unless you supply it yourself by a back-and-forth), I'd expect LLMs to have trouble with this test - and I see it disputing nothing but a bad mental model of the authors. BTW. there is a way to test my claim, which is to construct similar tests explicitly designed to avoid going against training data and word-level associations (i.e. everything else the word used as function name, like "print" here, evokes). If GPT-4 fails that, where a typical person's "gut feel" wouldn't, then I'd see this as significant, and would be very interested in the exact details of the failure. In fact, time allowing today/tomorrow, I'll try to do some tests like these myself.
- mrtranscendence 3y ago
  > Even with a clear problem statement, at intuitive level, your intuition would still cling to those associations (and heaps of Python code you may have read and wrote in your life) - you need to override that at a conscious level. For the simple problem posed here, I dispute this. I do not need to rely on consciously overriding my intuition in order to replace "print" with "len" in this simple of an example. It's immediately obvious and, frankly, intuitive.
  - TeMPOraL 3y ago
    FWIW, I copy-pasted the example to GPT-4 and played with it a bit, and doing so I noticed that I do need to keep consciously overriding my intuition at all times when reading the output - the moment I stop, I get confused, and have to re-read the last 1-3 lines with even more concentration.
- YeGoblynQueenne 3y ago
  >> My take is: does a human programmer in this case would have to consciously talk to themselves to not make a mistake? "Remember, remember, 'el eye en' prints, 'pee ar aye en tee' computes length". Would they have to double-check their output? If yes, then don't expect LLMs to do better in general case, unless you make them do those steps. I'm pretty sure most people would not need to do any of that to answer the question in the motivating example correctly. Remember: it's three lines and a question. Pretty much the Country Music of experiments, eh?
  - TeMPOraL 3y ago
    > Remember: it's three lines and a question. Pretty much the Country Music of experiments, eh? IDK, three lines of math is more than enough for me to make fatal mistakes in. For example, in my secondary and high school times, it took me years to stop carelessly forgetting to carry a minus sign during calculations - even as simple as requiring three lines. Another analogy - you may argue this is goal-post shifting, but I'm really not inventing new analogies here, only bringing up the other one of the two I always use - GPT-4 being like a 4 year old kid in terms of focus. Anyway: the few tests I run tended to show GPT-4 first understanding the problem correctly, and then confusing itself mid-way through its output. This is quite similar to what my 4 y.o. often does. She'll say stuff like "no, X isn't like Y, X is more like Z, it's not like Y", and then suddenly continue as if X was like Y, as if the idea of X being like Y overpowered her earlier assertion that it's not.

kenjackson 3y ago

I’m not sure MOST humans would pass the test. Maybe most decent SW engineers would.

TeMPOraL 3y ago

My take is: does a human programmer in this case would have to consciously talk to themselves to not make a mistake? "Remember, remember, 'el eye en' prints, 'pee ar aye en tee' computes length". Would they have to double-check their output? If yes, then don't expect LLMs to do better in general case, unless you make them do those steps. This is what I mean when I keep saying that a good analogy for current LLMs, GPT-4 in particular, is the "inner voice" - that bit in your head that spits out streams of thought using language. If you look at LLM output as you'd look at the thoughts popping into your head when you read the prompt, before you consciously fix/filter/"return to sender" them, I think you'd find the two very similar. My objection to the article's test is thus that it's intentionally set up to confuse one's gut feel. "len" and "print" are not opaque tokens, they are words with whole lot of associations, and those associations are precisely why they were chosen to name those specific functions. Even with a clear problem statement, at intuitive level, your intuition would still cling to those associations (and heaps of Python code you may have read and wrote in your life) - you need to override that at a conscious level. Since I claim that LLM output is equivalent to the thoughts produced by intuition, and there is no "conscious level" equivalent (unless you supply it yourself by a back-and-forth), I'd expect LLMs to have trouble with this test - and I see it disputing nothing but a bad mental model of the authors. BTW. there is a way to test my claim, which is to construct similar tests explicitly designed to avoid going against training data and word-level associations (i.e. everything else the word used as function name, like "print" here, evokes). If GPT-4 fails that, where a typical person's "gut feel" wouldn't, then I'd see this as significant, and would be very interested in the exact details of the failure. In fact, time allowing today/tomorrow, I'll try to do some tests like these myself.

mrtranscendence 3y ago
> Even with a clear problem statement, at intuitive level, your intuition would still cling to those associations (and heaps of Python code you may have read and wrote in your life) - you need to override that at a conscious level. For the simple problem posed here, I dispute this. I do not need to rely on consciously overriding my intuition in order to replace "print" with "len" in this simple of an example. It's immediately obvious and, frankly, intuitive.
- TeMPOraL 3y ago
  FWIW, I copy-pasted the example to GPT-4 and played with it a bit, and doing so I noticed that I do need to keep consciously overriding my intuition at all times when reading the output - the moment I stop, I get confused, and have to re-read the last 1-3 lines with even more concentration.
YeGoblynQueenne 3y ago
>> My take is: does a human programmer in this case would have to consciously talk to themselves to not make a mistake? "Remember, remember, 'el eye en' prints, 'pee ar aye en tee' computes length". Would they have to double-check their output? If yes, then don't expect LLMs to do better in general case, unless you make them do those steps. I'm pretty sure most people would not need to do any of that to answer the question in the motivating example correctly. Remember: it's three lines and a question. Pretty much the Country Music of experiments, eh?
- TeMPOraL 3y ago
  > Remember: it's three lines and a question. Pretty much the Country Music of experiments, eh? IDK, three lines of math is more than enough for me to make fatal mistakes in. For example, in my secondary and high school times, it took me years to stop carelessly forgetting to carry a minus sign during calculations - even as simple as requiring three lines. Another analogy - you may argue this is goal-post shifting, but I'm really not inventing new analogies here, only bringing up the other one of the two I always use - GPT-4 being like a 4 year old kid in terms of focus. Anyway: the few tests I run tended to show GPT-4 first understanding the problem correctly, and then confusing itself mid-way through its output. This is quite similar to what my 4 y.o. often does. She'll say stuff like "no, X isn't like Y, X is more like Z, it's not like Y", and then suddenly continue as if X was like Y, as if the idea of X being like Y overpowered her earlier assertion that it's not.