Obviously I wouldn't have been impressed by hallucinated output. I'm talking about correctly modeling a variable's state through its conception of what a Pytho

Obviously I wouldn't have been impressed by hallucinated output. I'm talking about correctly modeling a variable's state through its conception of what a Python interpreter does, which requires building a model of that interpreter and extracting consequences from it, rather than pure language statistics. I agree that LLMs will often hallucinate. There is obviously no guarantee that the output is correct. But sometimes it is correct anyway, which I notice by actually running the code. Here is a trivial example which I only mention to bring the conversation back to the reality of GPT-4 actually being able to do things like this: Me: You are a Python interpreter. Please give the correct output of the supplied code, with no commentary. >>> a = ["wokwokwok", "says", "i", "have", "no", "understanding", "of", "code"] >>> a.append("!") >>> " ".join([w.upper() for w in a]) GPT-4, on first attempt: WOKWOKWOK SAYS I HAVE NO UNDERSTANDING OF CODE !