5 ms·

It is impressive, but no, it won’t. It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of

It is impressive, but no, it won’t. It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of hallucinations it will be. There’s a huge difference. Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. Without access to environment, it’s not possible to magically know what the output of a command will be, it doesn’t have an embedded understanding of code, specifically not iterations and mapped data or mathematical functions. For trivial obvious outputs it’s good, but it’s not executing the code; it’s generating what seems like plausible output; and if the output is trivially derived from the input, it’ll be impressively accurate. …but, as the complexity of the task increases or the task deviates from “standard problem space” the triviality of generating accurate output decreases and it stops generating impressive outputs. Tldr; yes, but it doesn’t scale well beyond trivial outputs.

ben_w 4y ago
> Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. The former is necessarily the case given the Halting Problem; the latter is falsified by the fact we can reason about code despite the Halting Problem.
- cvs268 4y ago
  >> without running the command yourself, > the latter is falsified by the fact we can reason about code despite the Halting Problem i think wokwokwok's point holds true in practice. Our patience and working-memory is far more limited than what is essential to accurately model all the necessary details of even moderately complex algorithms in our head. One of the main reasons to limit code-complexity to improve readability/maintainability. https://en.wikipedia.org/wiki/Cyclomatic_complexity https://en.wikipedia.org/wiki/Cyclomatic_complexity
- wokwokwok 4y ago
  > the latter is falsified by the fact we can reason about code despite the Halting Problem. I'm not talking in general terms, or describing 'what the code does' in a summary or bullet point high-level form. No one is arguing that it can't summarize and describe what code does. These models are very good at that. I'm talking specifically about generating the output of the command, as the OP specifically mentioned. It does generate the exact output for commands and scripts if you request it, sometimes even if you don't, just as an example; they're just, often, hallucinated rubbish. Being impressed that GPT can invent from 'thin air' some creative writing (fiction) when you tell it `pretend you're a docker container and now run 'ls'` is I feel, missing the boat, in terms of understanding or being impressed by the capabilities of these LLMs.
  - vidarh 4y ago
    And they're often not, even for quite complex functions that requires symbolically executing quite a few calculations to get to the pre-requisite output. Nobody is impressed that it "can invent from 'thin air' some creative writing (fiction)", but that it often does not and in fact produces correct output. You're right we can't rely on it producing the correct output as it currently stands, but that it is capable of doing this at all is impressive.
- STM32F030R8 4y ago
  Can you explain how the halting problem applies here?
  - mellavora 4y ago
    Suspect this is a troll posting, but on the off chance I'm wrong... The LLM gives the output of a command. To do so, it has to be able to determine when the command exits. This is exactly the halting problem. For a trivial example, what is the output of: ``` while True: pass print("goodby world") ``` (this is also proof that leaving out the curly braces makes code harder instead of simpler #python-lie-to-me. multiple edits to get this to render correctly on HN )
    - krackers 4y ago
      Rice's theorem may actually be more appropriate here (but it's a consequence of the halting theorem). But it's important to note that just because there's no algorithm that works on ALL programs doesn't mean that the semantic properties of all programs are undecidable. Clearly for the particular programs where the program is bounded and guaranteed to terminate (e.g. no unbounded loops or recursion allowed) we can determine such properties, and I believe theorem provers in fact only allow such programs. And similarly you can restrict yourself to only the programs that you can prove will terminate in N steps (which might be excluding some programs that do terminate but require more than N steps of compute to prove).
pmoriarty 4y ago
> It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of hallucinations it will be. It doesn't even have to be complex. Ask it about a program in a language that's not very popular and the odds that it'll completely screw up its answers is high.
reaperman 4y ago
> Tldr; yes, but it doesn’t scale well beyond trivial outputs. I understand that the current LLM architectures are fundamentally incapable of goal-seeking and that they lack any concept of "correctness". However, I also recognize that somehow, the incredibly "wrong" architecture of ChatGPT is able to be useful. I'd like to think that with access to a sandbox runtime environment, a significantly larger context window, and perhaps additional copies of the LLM "supervising/orchestrating" multiple "lower" copies of the LLM by breaking down large work into smaller tasks, that the current ChatGPT architecture could scale well beyond trivial scripts. And then I hope we abandon this LLM architecture and develop architectures which can actually internally work towards "going beyond" in terms of quality of output for a given task.
- hammyhavoc 4y ago
  Instead of abandoning anything, why not just use appropriate tooling? Boilerplate if you want some code written for you, debugger if you want to know what's wrong with what you're doing. Been around for more than twenty years.
DennisP 4y ago
> the more complex the script, the more wrong and full of hallucinations it will be I often have that same problem myself.
cjbprime 4y ago
Obviously I wouldn't have been impressed by hallucinated output. I'm talking about correctly modeling a variable's state through its conception of what a Python interpreter does, which requires building a model of that interpreter and extracting consequences from it, rather than pure language statistics. I agree that LLMs will often hallucinate. There is obviously no guarantee that the output is correct. But sometimes it is correct anyway, which I notice by actually running the code. Here is a trivial example which I only mention to bring the conversation back to the reality of GPT-4 actually being able to do things like this: Me: You are a Python interpreter. Please give the correct output of the supplied code, with no commentary. >>> a = ["wokwokwok", "says", "i", "have", "no", "understanding", "of", "code"] >>> a.append("!") >>> " ".join([w.upper() for w in a]) GPT-4, on first attempt: WOKWOKWOK SAYS I HAVE NO UNDERSTANDING OF CODE !
- wokwokwok 4y ago
  That’s because you’ve asked for something that is trivially derived. > Tldr; yes, but it doesn’t scale well beyond trivial outputs.
  - cjbprime 4y ago
    Which algorithms are trivial and which are non-trivial? Is sorting trivial? base64-encoding? Can you provide a set of small scripts that you expect GPT-4 to be unable to correctly simulate execution of? It is weird that you seem to agree that it is capable of performing algorithmic simulation, while discounting that with "but it’s not executing the code; it’s generating what seems like plausible output", in a way that seems suspiciously close to defining anything it simulates correctly as "trivial", and anything it would fail at as "executing the code"...

ben_w 4y ago

> Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. The former is necessarily the case given the Halting Problem; the latter is falsified by the fact we can reason about code despite the Halting Problem.

cvs268 4y ago
>> without running the command yourself, > the latter is falsified by the fact we can reason about code despite the Halting Problem i think wokwokwok's point holds true in practice. Our patience and working-memory is far more limited than what is essential to accurately model all the necessary details of even moderately complex algorithms in our head. One of the main reasons to limit code-complexity to improve readability/maintainability. https://en.wikipedia.org/wiki/Cyclomatic_complexity https://en.wikipedia.org/wiki/Cyclomatic_complexity
wokwokwok 4y ago
> the latter is falsified by the fact we can reason about code despite the Halting Problem. I'm not talking in general terms, or describing 'what the code does' in a summary or bullet point high-level form. No one is arguing that it can't summarize and describe what code does. These models are very good at that. I'm talking specifically about generating the output of the command, as the OP specifically mentioned. It does generate the exact output for commands and scripts if you request it, sometimes even if you don't, just as an example; they're just, often, hallucinated rubbish. Being impressed that GPT can invent from 'thin air' some creative writing (fiction) when you tell it `pretend you're a docker container and now run 'ls'` is I feel, missing the boat, in terms of understanding or being impressed by the capabilities of these LLMs.
- vidarh 4y ago
  And they're often not, even for quite complex functions that requires symbolically executing quite a few calculations to get to the pre-requisite output. Nobody is impressed that it "can invent from 'thin air' some creative writing (fiction)", but that it often does not and in fact produces correct output. You're right we can't rely on it producing the correct output as it currently stands, but that it is capable of doing this at all is impressive.
STM32F030R8 4y ago
Can you explain how the halting problem applies here?
- mellavora 4y ago
  Suspect this is a troll posting, but on the off chance I'm wrong... The LLM gives the output of a command. To do so, it has to be able to determine when the command exits. This is exactly the halting problem. For a trivial example, what is the output of: ``` while True: pass print("goodby world") ``` (this is also proof that leaving out the curly braces makes code harder instead of simpler #python-lie-to-me. multiple edits to get this to render correctly on HN )
  - krackers 4y ago
    Rice's theorem may actually be more appropriate here (but it's a consequence of the halting theorem). But it's important to note that just because there's no algorithm that works on ALL programs doesn't mean that the semantic properties of all programs are undecidable. Clearly for the particular programs where the program is bounded and guaranteed to terminate (e.g. no unbounded loops or recursion allowed) we can determine such properties, and I believe theorem provers in fact only allow such programs. And similarly you can restrict yourself to only the programs that you can prove will terminate in N steps (which might be excluding some programs that do terminate but require more than N steps of compute to prove).

pmoriarty 4y ago

> It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of hallucinations it will be. It doesn't even have to be complex. Ask it about a program in a language that's not very popular and the odds that it'll completely screw up its answers is high.

reaperman 4y ago

> Tldr; yes, but it doesn’t scale well beyond trivial outputs. I understand that the current LLM architectures are fundamentally incapable of goal-seeking and that they lack any concept of "correctness". However, I also recognize that somehow, the incredibly "wrong" architecture of ChatGPT is able to be useful. I'd like to think that with access to a sandbox runtime environment, a significantly larger context window, and perhaps additional copies of the LLM "supervising/orchestrating" multiple "lower" copies of the LLM by breaking down large work into smaller tasks, that the current ChatGPT architecture could scale well beyond trivial scripts. And then I hope we abandon this LLM architecture and develop architectures which can actually internally work towards "going beyond" in terms of quality of output for a given task.

hammyhavoc 4y ago
Instead of abandoning anything, why not just use appropriate tooling? Boilerplate if you want some code written for you, debugger if you want to know what's wrong with what you're doing. Been around for more than twenty years.

DennisP 4y ago

> the more complex the script, the more wrong and full of hallucinations it will be I often have that same problem myself.

cjbprime 4y ago

Obviously I wouldn't have been impressed by hallucinated output. I'm talking about correctly modeling a variable's state through its conception of what a Python interpreter does, which requires building a model of that interpreter and extracting consequences from it, rather than pure language statistics. I agree that LLMs will often hallucinate. There is obviously no guarantee that the output is correct. But sometimes it is correct anyway, which I notice by actually running the code. Here is a trivial example which I only mention to bring the conversation back to the reality of GPT-4 actually being able to do things like this: Me: You are a Python interpreter. Please give the correct output of the supplied code, with no commentary. >>> a = ["wokwokwok", "says", "i", "have", "no", "understanding", "of", "code"] >>> a.append("!") >>> " ".join([w.upper() for w in a]) GPT-4, on first attempt: WOKWOKWOK SAYS I HAVE NO UNDERSTANDING OF CODE !

wokwokwok 4y ago
That’s because you’ve asked for something that is trivially derived. > Tldr; yes, but it doesn’t scale well beyond trivial outputs.
- cjbprime 4y ago
  Which algorithms are trivial and which are non-trivial? Is sorting trivial? base64-encoding? Can you provide a set of small scripts that you expect GPT-4 to be unable to correctly simulate execution of? It is weird that you seem to agree that it is capable of performing algorithmic simulation, while discounting that with "but it’s not executing the code; it’s generating what seems like plausible output", in a way that seems suspiciously close to defining anything it simulates correctly as "trivial", and anything it would fail at as "executing the code"...