4 ms·

Just to point out: this doesn't work since at the point where it defines the new print function, the new len which now prints is used

• okasaki 3y ago
  Ok, but the error is more of a syntax error. If I did this in an interview, you wouldn't accuse me of a lack of some ineffable human quality of "deep understanding". Anyway, I asked it to fix it, and behold: def print(list_arg): return len(list_arg) def len(value): __builtins__.print(value)
  • elcomet 3y ago
    It's more a logic error. Like swapping two variables, you usually need to create a third temporary one.
  • brenns10 3y ago
    But that doesn't fix it? Sure, no more infinite recursion, but now print() calls len() which calls __builtins__.print(). So it's not doing the expected swapped behavior. len() is swapped, but not print(). It seems obvious that there's a lack of understanding of the underlying concept here. That's the whole point. We know that LLMs can generate valid programs, but this is demonstrating that they cannot reason about code. There is no understanding of how Python code is evaluated, and how to avoid the infinite recursion. A human who understands Python could properly handle the situation, but the LLM can't, which is ok, it just demonstrates a flaw.
  • YeGoblynQueenne 3y ago
    >> If I did this in an interview, you wouldn't accuse me of a lack of some ineffable human quality of "deep understanding". Yeah, sorry but I would. The experiment in the article is about identifier swapping, not about function redefinition, which is what you have done. Better not do that in an interview.