4 ms·

Human programmers will also initially spend 10x of effort working on code base where identifier names are meanongful but swapped. Doing so will employ a lot of

Human programmers will also initially spend 10x of effort working on code base where identifier names are meanongful but swapped. Doing so will employ a lot of inner dialogue such as "this method says close but it is actually reset" and I don't doubt LLMs may be made straight by the same prompts.

TeMPOraL 3y ago
Yup. The whole premise of this test is idiotic. When you give a piece of code to an LLM, it doesn't magically switch into "code focused mode". They write: > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. But this is bullshit. Identifier renaming may be a no-op for the code, or even for the compiler/interpreter, but it's absolutely a meaningful change for human programmers. If it weren't, we'd all be calling functions named f0001, f0002, f0003, ... etc. to save space. Like us, LLMs process many different associations at the same time. A function named "print" has a lot of associations that together reinforce the understanding that it will make something the output of the program. Renaming it into len() makes the meaning go against all the association that go with that word. Swapping print() and len() to use in the same program? That would trip any human up, it's tailor-made to be difficult for us to process. And, as it again turns out, so it is for LLMs, because they really seem to process associations the same way we do (at the gut/intuition/first reaction level).
- mjburgess 3y ago
  So the dialectic here is: > AI Industry: Generative AI is sensitive to the semantic properties of code > Credulous Fanatic: Yes! Of course! Here's an infinite number of cases to confirm that idea > Scientist: Here's a single case which shows that's false > Credulous Fanatic: But... "some irrelevant unevidenced point about human capabilities, 101 distractions, repetition of the latest OpenAI press release" Conclusion: ChatGPT is not sensitive to the semantic properties of code. It's output displays that apparent property in many useful cases.
  - TeMPOraL 3y ago
    No. The problem here is, "scientists" believe GPT models couldn't possibly be in any way exhibiting forms of thinking or intelligence similar to humans, and so they assume they must be doing some specific kind of computation humans are not, despite every interaction with ChatGPT being evidence to the contrary. They design a test that a human would not pass (despite their claims), but their imagined specific computational system would - and then act all proud and mighty after ChatGPT fails it too. The only real insight from this event is: the authors of this paper imagined ChatGPT to be something it isn't, then demonstrated it in fact isn't it, and think they've discovered something.
- pwdisswordfishc 3y ago
  If a human being trips up over this 50% of the time (and I expect the actual figure to be lower), while an LLM trips up 90% of the time, that’s still a win for the human being.
  - thriftwy 3y ago
    It is a known and published weakness of LLMs that they do not have internal monologue, whereas human do (where they can remind themselves over and over that names are switched). But it's not impossible to implement it.
    - TeMPOraL 3y ago
      LLMs are the internal monologue.
      - thriftwy 3y ago
        They don't have N steps of iterative internal monologue before arriving at end result.
        TeMPOraL 3y ago
        Exactly. Which is why asking them to "think step by step", or reflecting their output back at them (as the chat interface does) is so effective. LLMs are the internal monologue, but they lack the ability to iterate - you have to pump them manually.
- YeGoblynQueenne 3y ago
  To be clear, did you have any trouble figuring out the correct answer in the motivating example in the article's introduction? I'm asking because sure, a human programmer might well get tripped up by a name swap if they had missed the swap, or forgot it, but when presented with a short test of three lines like the one in the article's motivating example, the majority of human programmers would pass the test easily. That's what the experiment is testing. Three short lines and a question. The LLMs fail it. How often would humans fail it? (Note I think the experiment is not very informative for other reasons, but I'm specifically pointing out that most humans would pass the test anyway).
  - kenjackson 3y ago
    I’m not sure MOST humans would pass the test. Maybe most decent SW engineers would.
  - TeMPOraL 3y ago
    My take is: does a human programmer in this case would have to consciously talk to themselves to not make a mistake? "Remember, remember, 'el eye en' prints, 'pee ar aye en tee' computes length". Would they have to double-check their output? If yes, then don't expect LLMs to do better in general case, unless you make them do those steps. This is what I mean when I keep saying that a good analogy for current LLMs, GPT-4 in particular, is the "inner voice" - that bit in your head that spits out streams of thought using language. If you look at LLM output as you'd look at the thoughts popping into your head when you read the prompt, before you consciously fix/filter/"return to sender" them, I think you'd find the two very similar. My objection to the article's test is thus that it's intentionally set up to confuse one's gut feel. "len" and "print" are not opaque tokens, they are words with whole lot of associations, and those associations are precisely why they were chosen to name those specific functions. Even with a clear problem statement, at intuitive level, your intuition would still cling to those associations (and heaps of Python code you may have read and wrote in your life) - you need to override that at a conscious level. Since I claim that LLM output is equivalent to the thoughts produced by intuition, and there is no "conscious level" equivalent (unless you supply it yourself by a back-and-forth), I'd expect LLMs to have trouble with this test - and I see it disputing nothing but a bad mental model of the authors. BTW. there is a way to test my claim, which is to construct similar tests explicitly designed to avoid going against training data and word-level associations (i.e. everything else the word used as function name, like "print" here, evokes). If GPT-4 fails that, where a typical person's "gut feel" wouldn't, then I'd see this as significant, and would be very interested in the exact details of the failure. In fact, time allowing today/tomorrow, I'll try to do some tests like these myself.

TeMPOraL 3y ago

Yup. The whole premise of this test is idiotic. When you give a piece of code to an LLM, it doesn't magically switch into "code focused mode". They write: > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. But this is bullshit. Identifier renaming may be a no-op for the code, or even for the compiler/interpreter, but it's absolutely a meaningful change for human programmers. If it weren't, we'd all be calling functions named f0001, f0002, f0003, ... etc. to save space. Like us, LLMs process many different associations at the same time. A function named "print" has a lot of associations that together reinforce the understanding that it will make something the output of the program. Renaming it into len() makes the meaning go against all the association that go with that word. Swapping print() and len() to use in the same program? That would trip any human up, it's tailor-made to be difficult for us to process. And, as it again turns out, so it is for LLMs, because they really seem to process associations the same way we do (at the gut/intuition/first reaction level).

mjburgess 3y ago
So the dialectic here is: > AI Industry: Generative AI is sensitive to the semantic properties of code > Credulous Fanatic: Yes! Of course! Here's an infinite number of cases to confirm that idea > Scientist: Here's a single case which shows that's false > Credulous Fanatic: But... "some irrelevant unevidenced point about human capabilities, 101 distractions, repetition of the latest OpenAI press release" Conclusion: ChatGPT is not sensitive to the semantic properties of code. It's output displays that apparent property in many useful cases.
- TeMPOraL 3y ago
  No. The problem here is, "scientists" believe GPT models couldn't possibly be in any way exhibiting forms of thinking or intelligence similar to humans, and so they assume they must be doing some specific kind of computation humans are not, despite every interaction with ChatGPT being evidence to the contrary. They design a test that a human would not pass (despite their claims), but their imagined specific computational system would - and then act all proud and mighty after ChatGPT fails it too. The only real insight from this event is: the authors of this paper imagined ChatGPT to be something it isn't, then demonstrated it in fact isn't it, and think they've discovered something.
pwdisswordfishc 3y ago
If a human being trips up over this 50% of the time (and I expect the actual figure to be lower), while an LLM trips up 90% of the time, that’s still a win for the human being.
- thriftwy 3y ago
  It is a known and published weakness of LLMs that they do not have internal monologue, whereas human do (where they can remind themselves over and over that names are switched). But it's not impossible to implement it.
  - TeMPOraL 3y ago
    LLMs are the internal monologue.
    - thriftwy 3y ago
      They don't have N steps of iterative internal monologue before arriving at end result.
      - TeMPOraL 3y ago
        Exactly. Which is why asking them to "think step by step", or reflecting their output back at them (as the chat interface does) is so effective. LLMs are the internal monologue, but they lack the ability to iterate - you have to pump them manually.
YeGoblynQueenne 3y ago
To be clear, did you have any trouble figuring out the correct answer in the motivating example in the article's introduction? I'm asking because sure, a human programmer might well get tripped up by a name swap if they had missed the swap, or forgot it, but when presented with a short test of three lines like the one in the article's motivating example, the majority of human programmers would pass the test easily. That's what the experiment is testing. Three short lines and a question. The LLMs fail it. How often would humans fail it? (Note I think the experiment is not very informative for other reasons, but I'm specifically pointing out that most humans would pass the test anyway).
- kenjackson 3y ago
  I’m not sure MOST humans would pass the test. Maybe most decent SW engineers would.
- TeMPOraL 3y ago
  My take is: does a human programmer in this case would have to consciously talk to themselves to not make a mistake? "Remember, remember, 'el eye en' prints, 'pee ar aye en tee' computes length". Would they have to double-check their output? If yes, then don't expect LLMs to do better in general case, unless you make them do those steps. This is what I mean when I keep saying that a good analogy for current LLMs, GPT-4 in particular, is the "inner voice" - that bit in your head that spits out streams of thought using language. If you look at LLM output as you'd look at the thoughts popping into your head when you read the prompt, before you consciously fix/filter/"return to sender" them, I think you'd find the two very similar. My objection to the article's test is thus that it's intentionally set up to confuse one's gut feel. "len" and "print" are not opaque tokens, they are words with whole lot of associations, and those associations are precisely why they were chosen to name those specific functions. Even with a clear problem statement, at intuitive level, your intuition would still cling to those associations (and heaps of Python code you may have read and wrote in your life) - you need to override that at a conscious level. Since I claim that LLM output is equivalent to the thoughts produced by intuition, and there is no "conscious level" equivalent (unless you supply it yourself by a back-and-forth), I'd expect LLMs to have trouble with this test - and I see it disputing nothing but a bad mental model of the authors. BTW. there is a way to test my claim, which is to construct similar tests explicitly designed to avoid going against training data and word-level associations (i.e. everything else the word used as function name, like "print" here, evokes). If GPT-4 fails that, where a typical person's "gut feel" wouldn't, then I'd see this as significant, and would be very interested in the exact details of the failure. In fact, time allowing today/tomorrow, I'll try to do some tests like these myself.