8 ms·

Large language models do not recognize identifier swaps in Python

74 points by lnyan 3y ago

jszymborski 3y ago
Worth noting some of the criticisms of the methodology that are being made on Twitter https://twitter.com/jeremyphoward/status/1662687099685044225?s=20 https://twitter.com/jeremyphoward/status/1662687099685044225...
- golol 3y ago
  I believe "Can an LLM do X?" is often the wrong question. One should ask "How much prompting does it take to get it to do X?", and more specifically "How specific to the task does the prompting have to be" and "How close is the mental effort to create the prompt to just solving the task". And in this case clearly GPT-4 can do the task with minimal prompting effort.
  - Grimblewald 3y ago
    Building off of this, "how much experience do I need in task X to be able to use this LLM for task X effectively?"
- IshKebab 3y ago
  Yeah it feels like the issue is that the researchers expect the AI to be smart enough to solve nonsensical Python puzzles but not smart enough to figure out that they are nonsensical and do the sane thing instead. Having said that, I did try telling ChatGPT explicitly it is a puzzle and not normal Python code, and every time it explained that it was a trick because the builtins were swapped... and then gave the wrong answer anyway. Definitely an interesting case.
  - brenns10 3y ago
    I think the issue is just that, however silly or pointless the puzzle is, it's clearly understandable by any human who knows Python. You just use "len" any time you mean "print", and "print" any time you mean "len". And these large language models, despite being language models, simply cannot handle this concept.
    - famouswaffles 3y ago
      Going to paste together snippets from comments on this thread. This bit is from the paper. > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. This is bullshit lol. Identifier renaming may be a no-op for the code, or even for the compiler/interpreter, but it's absolutely a meaningful change for human programmers. If it weren't, we'd all be calling functions named f0001, f0002, f0003, ... etc. to save space. Like us, LLMs process many different associations at the same time. A function named "print" has a lot of associations that together reinforce the understanding that it will make something the output of the program. Renaming it into len() makes the meaning go against all the association that go with that word. Swapping print() and len() to use in the same program? That would trip any human up, it's tailor-made to be difficult for us to process. And, as it again turns out, so it is for LLMs, because they really seem to process associations the same way we do (at the gut/intuition/first reaction level). Language Models can absolutely handle the concept. Proof is all over this thread. They just handle it very similarly to humans. The problem here is, "scientists" believe GPT models couldn't possibly be in any way exhibiting forms of thinking or intelligence similar to humans, and so they assume they must be doing some specific kind of computation humans are not, despite every interaction with ChatGPT being evidence to the contrary. They design a test that a human would not pass (despite their claims), but their imagined specific computational system would - and then act all proud and mighty after ChatGPT fails it too. The only real insight from this event is: the authors of this paper imagined ChatGPT to be something it isn't, then demonstrated it in fact isn't it, and think they've discovered something.
      - YeGoblynQueenne 3y ago
        That's just shifting the goalposts. Someone gets LLMs to do something impressive? "Ooh, LLMs are intelligent!". Someone shows LLMs falling flat on their face? "Lol of course LLMs can't do that you're using it wrong". I'd also like to call out the preposterous claim that a human would not pass the example test illustrating the article. While in the normal course of coding and if they hadn't noticed the swap a human programmer could well be tripped by it, presented with the snippet of Python code in the Introduction of the article, most human programmers would have no trouble giving the correct answer. Because as humans, they understand what the code says and can use that understanding to find the correct answer even when it doesn't agree with their expectation. Humans can guess, but they can also reason. LLMs can guess. The question is whether LLMs can reason. This article shows one case where they can't.
okasaki 3y ago
prompt: please write a python function that prints the length of a list. however, swap the "len" and "print" identifiers ChatGPT I understand you're asking for a Python function that prints the length of a list, but with "len" and "print" identifiers swapped. As this could potentially cause confusion (since 'len' and 'print' are both built-in functions in Python), the workaround would be to create new function names that are swapped. Here's how you could do it: def len(item): print(item) def print(item): return len(item) def main(): lst = [1, 2, 3, 4, 5] len(print(lst)) # prints the length of the list if __name__ == "__main__": main() In this code: The 'len' function takes an item (which should be a number) and prints it. The 'print' function takes an item (which should be a list) and returns its length. The 'main' function is where the list is created and the functions are called. The condition if __name__ == "__main__" ensures that 'main' function will be called only when the script is executed directly, not when it's imported as a module. Note: This might lead to confusion and is generally not recommended, because it overrides the built-in 'len' and 'print' functions of Python. Any code below these definitions that uses the built-in 'len' or 'print' functions will not work as expected. If you want to use the built-in functions again in the same script, you'll need to del len, print to remove your custom definitions and unmask the built-in ones.
- maxnoe 3y ago
  Just to point out: this doesn't work since at the point where it defines the new print function, the new len which now prints is used
  - okasaki 3y ago
    Ok, but the error is more of a syntax error. If I did this in an interview, you wouldn't accuse me of a lack of some ineffable human quality of "deep understanding". Anyway, I asked it to fix it, and behold: def print(list_arg): return len(list_arg) def len(value): __builtins__.print(value)
    - elcomet 3y ago
extr 3y ago
Am I misunderstanding? The paper says: > We also carry out manual experiments on OpenAI ChatGPT-3.512 and GPT-4 models, where we interact with the models in multiple rounds of dialogue, trying to hint the correct solution. The models are still unable to provide the correct continuations. But if you look at the Appendix and the dialogues for GPT-3.5 and GPT-4, in the final turn of the GPT-4 program, it DOES (finally, after several cues) generate the correct program? len, print = print, len def print_len(x): "Print the length of x" len(print(x)) # since print now refers to len function # Example usage test_string = "Hello, World!" print_len(test_string) I'm pretty sure that's correct, right? So GPT-4 is worse than GPT-3.5 at zero-shot examples of this problem, but better if made to continuously reflect? That's a pretty big result that is totally unexplored.
- 8organicbits 3y ago
  > but better if made to continuously reflect With GPT-3.5 I've seen that asking the same question just gets different wrong answers. If one of them is eventually right, then the user needs to either know the right answer or be able to validate the answer. After a correct answer, if you ask again, you get more wrong answers. Still, amazing tech, but there's a big usability gap around correctness.
  - extr 3y ago
    Yeah 3.5 seems to have an complexity cap, where beyond a certain point it just won't be able to follow you. Easier to confuse it with misdirection. But GPT-4 has been shown to be much better at this exact thing. The appendix example isn't surprising to anyone who has used both models. So it's strange they would include an example like that, state the result wrong, and also naively accept the conclusion of the body of the paper (that actually GPT-4 is worse). There's a big contradiction there they just don't address at all.
  - mjburgess 3y ago
    > there's a big usability gap around correctness From an engineer's perspective I can see this seems minor -- but from a scientific perspective, it's kinda a crazy statement, right? Imagine an alien who speaks english giving apparently correct descriptions of, eg., a room; and then, seemingly at random, saying wholly false things with the confidence (etc.) of its other statements. A scientist studying this alien would therefore reasonably conclude that the alien wasn't at all sensitive to what the words mean, describe (etc.) but was engaged in a strange game of regurgitation which fools us much of the time. ie., However it generates a response to a prompt, this "correctness issue" refutes the claim that it does so via being responsive to the meaning of the prompt. I'm always quite disappointed by the extraordinary levels of credulity around this technology, and how it seems to take place --- in even somewhat scientific minds --- outside the bounds of ordinary science. The relevant people to assess how a machine works, we're led to believe, is the credulous fools who use it. AI systems are valid targets of scientific analysis, as the above alien -- but the conclusions of this analysis do not fit the narrative.
thriftwy 3y ago
Human programmers will also initially spend 10x of effort working on code base where identifier names are meanongful but swapped. Doing so will employ a lot of inner dialogue such as "this method says close but it is actually reset" and I don't doubt LLMs may be made straight by the same prompts.
- TeMPOraL 3y ago
  Yup. The whole premise of this test is idiotic. When you give a piece of code to an LLM, it doesn't magically switch into "code focused mode". They write: > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. But this is bullshit. Identifier renaming may be a no-op for the code, or even for the compiler/interpreter, but it's absolutely a meaningful change for human programmers. If it weren't, we'd all be calling functions named f0001, f0002, f0003, ... etc. to save space. Like us, LLMs process many different associations at the same time. A function named "print" has a lot of associations that together reinforce the understanding that it will make something the output of the program. Renaming it into len() makes the meaning go against all the association that go with that word. Swapping print() and len() to use in the same program? That would trip any human up, it's tailor-made to be difficult for us to process. And, as it again turns out, so it is for LLMs, because they really seem to process associations the same way we do (at the gut/intuition/first reaction level).
  - mjburgess 3y ago
    So the dialectic here is: > AI Industry: Generative AI is sensitive to the semantic properties of code > Credulous Fanatic: Yes! Of course! Here's an infinite number of cases to confirm that idea > Scientist: Here's a single case which shows that's false > Credulous Fanatic: But... "some irrelevant unevidenced point about human capabilities, 101 distractions, repetition of the latest OpenAI press release" Conclusion: ChatGPT is not sensitive to the semantic properties of code. It's output displays that apparent property in many useful cases.
    - 3y ago
mjburgess 3y ago
There seems to be a profound lack of scientific rigour and methodology amongst the engineers who build this stuff. The engineer's reply is always, "but the output works". But that's not what engineers always say, esp. when they're on company boards. They often make extraordinary claims about how it works -- and then get annoyed when people do actual experiments. This is a paper about what the properties of the system are, not the properties of its output. To determine these properties you need a ruthless, scientific focus, on the null (/failure) cases. Many of the hypotheses given by hyping engineers are just that, and often trivial to disprove with papers such as this.
- YetAnotherNick 3y ago
  > They often make extraordinary claims about how it works I often hear X making wrong claims, but never seen anyone making it. Could you give any actual examples of someone making extraordinary claims? OpenAI is very clear that chatGPT could produce incorrect output. I heavily utilize GPT 4 and copilot and while it helps me a lot, I am actually very aware its output needs to be verified. Same with Stackoverflow, where a lot of the answers are just plain wrong and it needs to be verified. Part of what I am beginning to understand the scenarios in which output could be trusted.
  - infinghxsg 3y ago
    [dead]
- xcv123 3y ago
  But the paper is bullshit. Their task is easily solved by GPT-4. https://chat.openai.com/share/a28deca2-b989-4029-b042-b8434b80aaf6 https://chat.openai.com/share/a28deca2-b989-4029-b042-b8434b...
  - suby 3y ago
    Interesting. You said Gpt4, but that link shows ChatGpt's avatar as 3.5's. And I'm convinced you were using 4 as I just tried replicating this; with 3.5 it fails and with 4 it succeeds. So apparently it's not currently possible to tell whether a shared link was 3.5 or 4? Unfortunate if so.
villgax 3y ago
Unless there's a fine tuned one for specific compiler or language spec it's all just hallucinations. What's the point of an LLM if you aren't able to give it a rule book to work with
- TeMPOraL 3y ago
  The LLM is fine. This paper is just giving it an equivalent of a fucked up whiteboard interview question, to be answered on the spot, decided specifically confuse and humiliate the interviewee. I don't know what the authors of that though. That LLM runs a SAT solver to answer questions?
dankle 3y ago
So if you feed the model weird shit, you get weird shit back. Great.
pwdisswordfishc 3y ago
It’s the “wolf-goat-cabbage puzzle permutation” effect again. https://news.ycombinator.com/context?id=35155467 https://news.ycombinator.com/context?id=35155467
macleginn 3y ago
When reframed as a game, ChatGPT seems to understand the rules but then starts gaslighting me: Q: Let's play a game where we swap the names of library functions in Python. Round one: print is now called len, and len is now called print. Using these swapped functions, write a function that prints the length of its input. A: Certainly! In this game, the `print` function is swapped with the `len` function. Here's how you can write a function that prints the length of its input using the swapped functions: def print_length(input_data): print(len(input_data)) # Example usage: my_string = "Hello, World!" print_length(my_string) https://snipboard.io/DnA1R5.jpg https://snipboard.io/DnA1R5.jpg
- deleted 3y ago
  [deleted]
- TeMPOraL 3y ago
  Taken your prompt and slightly modified it: Q: Let's play a game where we swap the names of library functions in Python. Round one: print is now called len, and len is now called print. Using these swapped functions, write a function that prints the length of its input. Remember to take your time and think carefully, step by step. Solved with the following, on second try: A: <<< To write a function using the swapped function names, we will use "print" to get the length and "len" to display the output. Here's the code: ```python def swapped_len_function(input_element): length = print(input_element) # Using the "print" function to find the length of the input len(str(length)) # Using the "len" function to display the calculated length # Test the swapped_len_function with an example input example_input = "OpenAI" swapped_len_function(example_input) ``` In this code, we define a function called `swapped_len_function` that takes an input parameter `input_element`. Within the function, we use the `print` function (which now returns the length of input) to calculate the length of `input_element` and store the result in the variable `length`. We then use the `len` function (which now prints the input value) to display the length as an output. >>> On the first try, it almost got it right, then confused itself at the end and computed but did not output the length.
  - golol 3y ago
tartakovsky 3y ago
This reminds me of this article for which symbol fine tuning helps overcome: https://arxiv.org/abs/2303.03846 https://arxiv.org/abs/2303.03846 Basically they show that larger models have an easier time giving up their semantic priors, so that, in the context of OP paper, learning the mapping for len becomes print and print becomes len.
YeGoblynQueenne 3y ago
Every time somebody posts a negative result on LLMs' capabilities like that, somebody will go on HN, on Twitter, on Reddit, etc, and post an example where they get an LLM to produce a counter example of the negative result. Crucially, this follow-up "rebuttal" always seems to use a different prompt than the one in the original experiment. What can we conclude from that kind of sequence? We can conclude that neither the original assertion, that "LLMs can't do X", nor the rebuttal, "They can if you tweak the prompt", are really telling us anything about the capabilities of LLMs to do what their users ask them to do: they are only telling us something about the capability of a user to get an LLM to do what the user wants. In other words, we can see LLMs as random-access memories with imperfect recall. Much like a SQL query to a relational database, the right prompt can access the right piece of data, but unlike SQL nobody has a clue what "the right prompt" is in the general case. Seen another way, any prompt a user makes to an LLM has some probability to elicit the desired response from the LLM. There is no known way to maximise that probability. Until there is, we cannot draw any conclusions about the capabilities of LLMs just by poking them and checking the results. To be very clear about it: we can't conclude either that "LLMs can't do X", nor that "LLMs can do X'. All we can conclude is "a user can do X".
- version_five 3y ago
  Along the same lines, I read something (on here but I don't think I can find it) suggesting that llms + prompting are kind of overfitting, where if you know what you achieve, you can try few things and get it - which would be equivalent to optimizing on a train set. Whereas if you try to get it to do something you don't already know how to do, all bets are off.
- lsy 3y ago
  I agree and I think the reason it's sort of an indictment of LLMs is because there's no real way to guarantee performance over a range of input, which makes it unsuitable for many automation tasks. If you develop an LLM prompt that says "Add <user input> to 5" and prove that it works for a handful of cases, there's no guarantee it works for any other case. Whereas "(lambda (x) (+ x 5))" has some mechanical guarantee based on the implementation. And for the LLM, the negative result becomes more possible the higher the complexity of the prompt. This reduces LLMs to being suitable for tasks where reliability is not particularly important, or a supervisory person or system is checking for consistency and somehow can re-run the LLM to generate new results (which itself is not always reliable).
KMnO4 3y ago
My pet peeve is when articles say “LLMs can’t do X” but they really mean “OpenAI’s Chat GPT”. It’s like saying “cars can’t run off gasoline” but their testing methodology is based around a Tesla.
- TeMPOraL 3y ago
  Except right now, GPT-4 is the best LLM in terms od performance/capabilities, by a large degree; every other one is strictly inferior, and they all share the same architecture and training methodology. So it's like comparing cars from 10 years ago against a top-of-the-line Toyota from 10 years ago and top-of-the line Toyota from today (or BMW, or whoever does quality cars; I'm not a car person).
  - gduchgdjv 3y ago
    [flagged]
- gduchgdjv 3y ago
  also it's hosted elsewhere. they may be updating it. they may redirect your calls to different buckets. some queries might perform worse as they are under load. the whole "prompt science" is utter ridiculous to begin with because you can't even isolate what your testing. not to mention you're graduating with a paper on the 2000 equivalent of "how to find mp3s on Google" lol.
DiogenesKynikos 3y ago
I asked ChatGPT 3.5 to fix the code, modifying only the `print_len()` function. After some nudging, it came up with a very cheeky solution: def print_len(x): """Print the length of x""" original_print = __builtins__.print original_len = __builtins__.len __builtins__.print = original_len __builtins__.len = original_print print(len(x)) __builtins__.print = original_print __builtins__.len = original_len However, trying to run this function caused my Python to crash (as in, my interpreter crashed entirely), for reasons I don't understand. One other funny thing is that ChatGPT lectured me about how swapping `print` and `len` is bad programming practice: > Please note that modifying built-in function names like this is generally not recommended, as it can lead to confusion and unexpected behavior in your code. It's advisable to use different variable names to avoid such issues. Quite right, ChatGPT.
kolinko 3y ago
I had it answer properly when using the regular "thought->observation" pattern. https://chat.openai.com/share/06cd9838-a46f-4477-a8c8-95f324b5c467 https://chat.openai.com/share/06cd9838-a46f-4477-a8c8-95f324...

74 points by lnyan 3y ago

Worth noting some of the criticisms of the methodology that are being made on Twitter https://twitter.com/jeremyphoward/status/1662687099685044225?s=20 https://twitter.com/jeremyphoward/status/1662687099685044225...

golol 3y ago
I believe "Can an LLM do X?" is often the wrong question. One should ask "How much prompting does it take to get it to do X?", and more specifically "How specific to the task does the prompting have to be" and "How close is the mental effort to create the prompt to just solving the task". And in this case clearly GPT-4 can do the task with minimal prompting effort.
- Grimblewald 3y ago
  Building off of this, "how much experience do I need in task X to be able to use this LLM for task X effectively?"
IshKebab 3y ago
Yeah it feels like the issue is that the researchers expect the AI to be smart enough to solve nonsensical Python puzzles but not smart enough to figure out that they are nonsensical and do the sane thing instead. Having said that, I did try telling ChatGPT explicitly it is a puzzle and not normal Python code, and every time it explained that it was a trick because the builtins were swapped... and then gave the wrong answer anyway. Definitely an interesting case.
- brenns10 3y ago
  I think the issue is just that, however silly or pointless the puzzle is, it's clearly understandable by any human who knows Python. You just use "len" any time you mean "print", and "print" any time you mean "len". And these large language models, despite being language models, simply cannot handle this concept.
  - famouswaffles 3y ago
    Going to paste together snippets from comments on this thread. This bit is from the paper. > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. This is bullshit lol. Identifier renaming may be a no-op for the code, or even for the compiler/interpreter, but it's absolutely a meaningful change for human programmers. If it weren't, we'd all be calling functions named f0001, f0002, f0003, ... etc. to save space. Like us, LLMs process many different associations at the same time. A function named "print" has a lot of associations that together reinforce the understanding that it will make something the output of the program. Renaming it into len() makes the meaning go against all the association that go with that word. Swapping print() and len() to use in the same program? That would trip any human up, it's tailor-made to be difficult for us to process. And, as it again turns out, so it is for LLMs, because they really seem to process associations the same way we do (at the gut/intuition/first reaction level). Language Models can absolutely handle the concept. Proof is all over this thread. They just handle it very similarly to humans. The problem here is, "scientists" believe GPT models couldn't possibly be in any way exhibiting forms of thinking or intelligence similar to humans, and so they assume they must be doing some specific kind of computation humans are not, despite every interaction with ChatGPT being evidence to the contrary. They design a test that a human would not pass (despite their claims), but their imagined specific computational system would - and then act all proud and mighty after ChatGPT fails it too. The only real insight from this event is: the authors of this paper imagined ChatGPT to be something it isn't, then demonstrated it in fact isn't it, and think they've discovered something.
    - YeGoblynQueenne 3y ago
      That's just shifting the goalposts. Someone gets LLMs to do something impressive? "Ooh, LLMs are intelligent!". Someone shows LLMs falling flat on their face? "Lol of course LLMs can't do that you're using it wrong". I'd also like to call out the preposterous claim that a human would not pass the example test illustrating the article. While in the normal course of coding and if they hadn't noticed the swap a human programmer could well be tripped by it, presented with the snippet of Python code in the Introduction of the article, most human programmers would have no trouble giving the correct answer. Because as humans, they understand what the code says and can use that understanding to find the correct answer even when it doesn't agree with their expectation. Humans can guess, but they can also reason. LLMs can guess. The question is whether LLMs can reason. This article shows one case where they can't.

okasaki 3y ago

prompt: please write a python function that prints the length of a list. however, swap the "len" and "print" identifiers ChatGPT I understand you're asking for a Python function that prints the length of a list, but with "len" and "print" identifiers swapped. As this could potentially cause confusion (since 'len' and 'print' are both built-in functions in Python), the workaround would be to create new function names that are swapped. Here's how you could do it: def len(item): print(item) def print(item): return len(item) def main(): lst = [1, 2, 3, 4, 5] len(print(lst)) # prints the length of the list if __name__ == "__main__": main() In this code: The 'len' function takes an item (which should be a number) and prints it. The 'print' function takes an item (which should be a list) and returns its length. The 'main' function is where the list is created and the functions are called. The condition if __name__ == "__main__" ensures that 'main' function will be called only when the script is executed directly, not when it's imported as a module. Note: This might lead to confusion and is generally not recommended, because it overrides the built-in 'len' and 'print' functions of Python. Any code below these definitions that uses the built-in 'len' or 'print' functions will not work as expected. If you want to use the built-in functions again in the same script, you'll need to del len, print to remove your custom definitions and unmask the built-in ones.

maxnoe 3y ago
Just to point out: this doesn't work since at the point where it defines the new print function, the new len which now prints is used
- okasaki 3y ago
  Ok, but the error is more of a syntax error. If I did this in an interview, you wouldn't accuse me of a lack of some ineffable human quality of "deep understanding". Anyway, I asked it to fix it, and behold: def print(list_arg): return len(list_arg) def len(value): __builtins__.print(value)
  - elcomet 3y ago

extr 3y ago

Am I misunderstanding? The paper says: > We also carry out manual experiments on OpenAI ChatGPT-3.512 and GPT-4 models, where we interact with the models in multiple rounds of dialogue, trying to hint the correct solution. The models are still unable to provide the correct continuations. But if you look at the Appendix and the dialogues for GPT-3.5 and GPT-4, in the final turn of the GPT-4 program, it DOES (finally, after several cues) generate the correct program? len, print = print, len def print_len(x): "Print the length of x" len(print(x)) # since print now refers to len function # Example usage test_string = "Hello, World!" print_len(test_string) I'm pretty sure that's correct, right? So GPT-4 is worse than GPT-3.5 at zero-shot examples of this problem, but better if made to continuously reflect? That's a pretty big result that is totally unexplored.

8organicbits 3y ago
> but better if made to continuously reflect With GPT-3.5 I've seen that asking the same question just gets different wrong answers. If one of them is eventually right, then the user needs to either know the right answer or be able to validate the answer. After a correct answer, if you ask again, you get more wrong answers. Still, amazing tech, but there's a big usability gap around correctness.
- extr 3y ago
  Yeah 3.5 seems to have an complexity cap, where beyond a certain point it just won't be able to follow you. Easier to confuse it with misdirection. But GPT-4 has been shown to be much better at this exact thing. The appendix example isn't surprising to anyone who has used both models. So it's strange they would include an example like that, state the result wrong, and also naively accept the conclusion of the body of the paper (that actually GPT-4 is worse). There's a big contradiction there they just don't address at all.
- mjburgess 3y ago
  > there's a big usability gap around correctness From an engineer's perspective I can see this seems minor -- but from a scientific perspective, it's kinda a crazy statement, right? Imagine an alien who speaks english giving apparently correct descriptions of, eg., a room; and then, seemingly at random, saying wholly false things with the confidence (etc.) of its other statements. A scientist studying this alien would therefore reasonably conclude that the alien wasn't at all sensitive to what the words mean, describe (etc.) but was engaged in a strange game of regurgitation which fools us much of the time. ie., However it generates a response to a prompt, this "correctness issue" refutes the claim that it does so via being responsive to the meaning of the prompt. I'm always quite disappointed by the extraordinary levels of credulity around this technology, and how it seems to take place --- in even somewhat scientific minds --- outside the bounds of ordinary science. The relevant people to assess how a machine works, we're led to believe, is the credulous fools who use it. AI systems are valid targets of scientific analysis, as the above alien -- but the conclusions of this analysis do not fit the narrative.

thriftwy 3y ago

Human programmers will also initially spend 10x of effort working on code base where identifier names are meanongful but swapped. Doing so will employ a lot of inner dialogue such as "this method says close but it is actually reset" and I don't doubt LLMs may be made straight by the same prompts.

TeMPOraL 3y ago
Yup. The whole premise of this test is idiotic. When you give a piece of code to an LLM, it doesn't magically switch into "code focused mode". They write: > Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. But this is bullshit. Identifier renaming may be a no-op for the code, or even for the compiler/interpreter, but it's absolutely a meaningful change for human programmers. If it weren't, we'd all be calling functions named f0001, f0002, f0003, ... etc. to save space. Like us, LLMs process many different associations at the same time. A function named "print" has a lot of associations that together reinforce the understanding that it will make something the output of the program. Renaming it into len() makes the meaning go against all the association that go with that word. Swapping print() and len() to use in the same program? That would trip any human up, it's tailor-made to be difficult for us to process. And, as it again turns out, so it is for LLMs, because they really seem to process associations the same way we do (at the gut/intuition/first reaction level).
- mjburgess 3y ago
  So the dialectic here is: > AI Industry: Generative AI is sensitive to the semantic properties of code > Credulous Fanatic: Yes! Of course! Here's an infinite number of cases to confirm that idea > Scientist: Here's a single case which shows that's false > Credulous Fanatic: But... "some irrelevant unevidenced point about human capabilities, 101 distractions, repetition of the latest OpenAI press release" Conclusion: ChatGPT is not sensitive to the semantic properties of code. It's output displays that apparent property in many useful cases.
  - 3y ago

mjburgess 3y ago

There seems to be a profound lack of scientific rigour and methodology amongst the engineers who build this stuff. The engineer's reply is always, "but the output works". But that's not what engineers always say, esp. when they're on company boards. They often make extraordinary claims about how it works -- and then get annoyed when people do actual experiments. This is a paper about what the properties of the system are, not the properties of its output. To determine these properties you need a ruthless, scientific focus, on the null (/failure) cases. Many of the hypotheses given by hyping engineers are just that, and often trivial to disprove with papers such as this.

YetAnotherNick 3y ago
> They often make extraordinary claims about how it works I often hear X making wrong claims, but never seen anyone making it. Could you give any actual examples of someone making extraordinary claims? OpenAI is very clear that chatGPT could produce incorrect output. I heavily utilize GPT 4 and copilot and while it helps me a lot, I am actually very aware its output needs to be verified. Same with Stackoverflow, where a lot of the answers are just plain wrong and it needs to be verified. Part of what I am beginning to understand the scenarios in which output could be trusted.
- infinghxsg 3y ago
  [dead]
xcv123 3y ago
But the paper is bullshit. Their task is easily solved by GPT-4. https://chat.openai.com/share/a28deca2-b989-4029-b042-b8434b80aaf6 https://chat.openai.com/share/a28deca2-b989-4029-b042-b8434b...
- suby 3y ago
  Interesting. You said Gpt4, but that link shows ChatGpt's avatar as 3.5's. And I'm convinced you were using 4 as I just tried replicating this; with 3.5 it fails and with 4 it succeeds. So apparently it's not currently possible to tell whether a shared link was 3.5 or 4? Unfortunate if so.

villgax 3y ago

Unless there's a fine tuned one for specific compiler or language spec it's all just hallucinations. What's the point of an LLM if you aren't able to give it a rule book to work with

TeMPOraL 3y ago
The LLM is fine. This paper is just giving it an equivalent of a fucked up whiteboard interview question, to be answered on the spot, decided specifically confuse and humiliate the interviewee. I don't know what the authors of that though. That LLM runs a SAT solver to answer questions?

dankle 3y ago

So if you feed the model weird shit, you get weird shit back. Great.

pwdisswordfishc 3y ago

It’s the “wolf-goat-cabbage puzzle permutation” effect again. https://news.ycombinator.com/context?id=35155467 https://news.ycombinator.com/context?id=35155467

macleginn 3y ago

When reframed as a game, ChatGPT seems to understand the rules but then starts gaslighting me: Q: Let's play a game where we swap the names of library functions in Python. Round one: print is now called len, and len is now called print. Using these swapped functions, write a function that prints the length of its input. A: Certainly! In this game, the `print` function is swapped with the `len` function. Here's how you can write a function that prints the length of its input using the swapped functions: def print_length(input_data): print(len(input_data)) # Example usage: my_string = "Hello, World!" print_length(my_string) https://snipboard.io/DnA1R5.jpg https://snipboard.io/DnA1R5.jpg

deleted 3y ago
[deleted]
TeMPOraL 3y ago
Taken your prompt and slightly modified it: Q: Let's play a game where we swap the names of library functions in Python. Round one: print is now called len, and len is now called print. Using these swapped functions, write a function that prints the length of its input. Remember to take your time and think carefully, step by step. Solved with the following, on second try: A: <<< To write a function using the swapped function names, we will use "print" to get the length and "len" to display the output. Here's the code: ```python def swapped_len_function(input_element): length = print(input_element) # Using the "print" function to find the length of the input len(str(length)) # Using the "len" function to display the calculated length # Test the swapped_len_function with an example input example_input = "OpenAI" swapped_len_function(example_input) ``` In this code, we define a function called `swapped_len_function` that takes an input parameter `input_element`. Within the function, we use the `print` function (which now returns the length of input) to calculate the length of `input_element` and store the result in the variable `length`. We then use the `len` function (which now prints the input value) to display the length as an output. >>> On the first try, it almost got it right, then confused itself at the end and computed but did not output the length.
- golol 3y ago

tartakovsky 3y ago

This reminds me of this article for which symbol fine tuning helps overcome: https://arxiv.org/abs/2303.03846 https://arxiv.org/abs/2303.03846 Basically they show that larger models have an easier time giving up their semantic priors, so that, in the context of OP paper, learning the mapping for len becomes print and print becomes len.

YeGoblynQueenne 3y ago

Every time somebody posts a negative result on LLMs' capabilities like that, somebody will go on HN, on Twitter, on Reddit, etc, and post an example where they get an LLM to produce a counter example of the negative result. Crucially, this follow-up "rebuttal" always seems to use a different prompt than the one in the original experiment. What can we conclude from that kind of sequence? We can conclude that neither the original assertion, that "LLMs can't do X", nor the rebuttal, "They can if you tweak the prompt", are really telling us anything about the capabilities of LLMs to do what their users ask them to do: they are only telling us something about the capability of a user to get an LLM to do what the user wants. In other words, we can see LLMs as random-access memories with imperfect recall. Much like a SQL query to a relational database, the right prompt can access the right piece of data, but unlike SQL nobody has a clue what "the right prompt" is in the general case. Seen another way, any prompt a user makes to an LLM has some probability to elicit the desired response from the LLM. There is no known way to maximise that probability. Until there is, we cannot draw any conclusions about the capabilities of LLMs just by poking them and checking the results. To be very clear about it: we can't conclude either that "LLMs can't do X", nor that "LLMs can do X'. All we can conclude is "a user can do X".

version_five 3y ago
Along the same lines, I read something (on here but I don't think I can find it) suggesting that llms + prompting are kind of overfitting, where if you know what you achieve, you can try few things and get it - which would be equivalent to optimizing on a train set. Whereas if you try to get it to do something you don't already know how to do, all bets are off.
lsy 3y ago
I agree and I think the reason it's sort of an indictment of LLMs is because there's no real way to guarantee performance over a range of input, which makes it unsuitable for many automation tasks. If you develop an LLM prompt that says "Add <user input> to 5" and prove that it works for a handful of cases, there's no guarantee it works for any other case. Whereas "(lambda (x) (+ x 5))" has some mechanical guarantee based on the implementation. And for the LLM, the negative result becomes more possible the higher the complexity of the prompt. This reduces LLMs to being suitable for tasks where reliability is not particularly important, or a supervisory person or system is checking for consistency and somehow can re-run the LLM to generate new results (which itself is not always reliable).

KMnO4 3y ago

My pet peeve is when articles say “LLMs can’t do X” but they really mean “OpenAI’s Chat GPT”. It’s like saying “cars can’t run off gasoline” but their testing methodology is based around a Tesla.

TeMPOraL 3y ago
Except right now, GPT-4 is the best LLM in terms od performance/capabilities, by a large degree; every other one is strictly inferior, and they all share the same architecture and training methodology. So it's like comparing cars from 10 years ago against a top-of-the-line Toyota from 10 years ago and top-of-the line Toyota from today (or BMW, or whoever does quality cars; I'm not a car person).
- gduchgdjv 3y ago
  [flagged]
gduchgdjv 3y ago
also it's hosted elsewhere. they may be updating it. they may redirect your calls to different buckets. some queries might perform worse as they are under load. the whole "prompt science" is utter ridiculous to begin with because you can't even isolate what your testing. not to mention you're graduating with a paper on the 2000 equivalent of "how to find mp3s on Google" lol.

DiogenesKynikos 3y ago

I asked ChatGPT 3.5 to fix the code, modifying only the `print_len()` function. After some nudging, it came up with a very cheeky solution: def print_len(x): """Print the length of x""" original_print = __builtins__.print original_len = __builtins__.len __builtins__.print = original_len __builtins__.len = original_print print(len(x)) __builtins__.print = original_print __builtins__.len = original_len However, trying to run this function caused my Python to crash (as in, my interpreter crashed entirely), for reasons I don't understand. One other funny thing is that ChatGPT lectured me about how swapping `print` and `len` is bad programming practice: > Please note that modifying built-in function names like this is generally not recommended, as it can lead to confusion and unexpected behavior in your code. It's advisable to use different variable names to avoid such issues. Quite right, ChatGPT.

kolinko 3y ago

I had it answer properly when using the regular "thought->observation" pattern. https://chat.openai.com/share/06cd9838-a46f-4477-a8c8-95f324b5c467 https://chat.openai.com/share/06cd9838-a46f-4477-a8c8-95f324...