4 ms·

Neat! My mind-blown moment with GPT-4 was realizing that it will often be able to tell you the output of the (unique, not available in training data) scripts it

Neat! My mind-blown moment with GPT-4 was realizing that it will often be able to tell you the output of the (unique, not available in training data) scripts it writes for you.

aix1 3y ago
Relevant: https://www.engraved.blog/building-a-virtual-machine-inside/ https://www.engraved.blog/building-a-virtual-machine-inside/
politelemon 3y ago
That is very interesting, which I'd like to try once GPT4 is widely available. I did try GPT3 with a CTF a few weeks ago, and it gave seemingly plausible code but outright incorrect answers.
- yreg 3y ago
  I did some advent of code exercises with GPT-3 and it often ended the script with something like // returns 1337 return result; Sometimes the comment stated the correct answer to the puzzle, but the script returned something else since it had a bug.
  - moffkalast 3y ago
    GPT-3 isn't the 1337 h4x0r it thinks it is.
erfgh 3y ago
Hum... maybe it runs the scripts and sees the output?
- SargeZT 3y ago
  It does not. Regular ChatGPT without plugins does not have access to any tools. Throw it a script with some weird outputs and it'll definitely fail, every time. While the script 'evaluation' stuff can be pretty decently impressive, it is not actually executing anything.
wokwokwok 3y ago
It is impressive, but no, it won’t. It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of hallucinations it will be. There’s a huge difference. Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. Without access to environment, it’s not possible to magically know what the output of a command will be, it doesn’t have an embedded understanding of code, specifically not iterations and mapped data or mathematical functions. For trivial obvious outputs it’s good, but it’s not executing the code; it’s generating what seems like plausible output; and if the output is trivially derived from the input, it’ll be impressively accurate. …but, as the complexity of the task increases or the task deviates from “standard problem space” the triviality of generating accurate output decreases and it stops generating impressive outputs. Tldr; yes, but it doesn’t scale well beyond trivial outputs.
- ben_w 3y ago
  > Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. The former is necessarily the case given the Halting Problem; the latter is falsified by the fact we can reason about code despite the Halting Problem.
  - cvs268 3y ago
    >> without running the command yourself, > the latter is falsified by the fact we can reason about code despite the Halting Problem i think wokwokwok's point holds true in practice. Our patience and working-memory is far more limited than what is essential to accurately model all the necessary details of even moderately complex algorithms in our head. One of the main reasons to limit code-complexity to improve readability/maintainability. https://en.wikipedia.org/wiki/Cyclomatic_complexity https://en.wikipedia.org/wiki/Cyclomatic_complexity
  - wokwokwok 3y ago
    > the latter is falsified by the fact we can reason about code despite the Halting Problem. I'm not talking in general terms, or describing 'what the code does' in a summary or bullet point high-level form. No one is arguing that it can't summarize and describe what code does. These models are very good at that. I'm talking specifically about generating the output of the command, as the OP specifically mentioned. It does generate the exact output for commands and scripts if you request it, sometimes even if you don't, just as an example; they're just, often, hallucinated rubbish. Being impressed that GPT can invent from 'thin air' some creative writing (fiction) when you tell it `pretend you're a docker container and now run 'ls'` is I feel, missing the boat, in terms of understanding or being impressed by the capabilities of these LLMs.
dist-epoch 3y ago
That's provably only possible for "small" scripts, due to the halting problem and all that.
web3-is-a-scam 3y ago
I was actually working on something in this vain yesterday, asking it for output and found it often generated the output I asked for, it was not actually the output of the SQL query that it wrote. The query it wrote wasn't even valid SQL but it was close enough to make you think it would work.
- saulpw 3y ago
  Yeah I asked it a Python question and got back an answer with some Python code to demonstrate. The Python code worked great but demonstrated the exact opposite of the answer that was given.
- bumbledraven 3y ago
  Are you sure you were using GPT-4? If so, can you provide a transcript or screenshot?
- HPMOR 3y ago
  I know this is extremely unnecessary and pedantic but I think you meant "in this vein" instead of "in this vain".
  - glitchc 3y ago
    Well, maybe the OP was working in vain, given that the SQL query was not valid code.
  - web3-is-a-scam 3y ago
    yes, thank you I always get them mixed up

Neat! My mind-blown moment with GPT-4 was realizing that it will often be able to tell you the output of the (unique, not available in training data) scripts it writes for you.

aix1 3y ago

Relevant: https://www.engraved.blog/building-a-virtual-machine-inside/ https://www.engraved.blog/building-a-virtual-machine-inside/

politelemon 3y ago

That is very interesting, which I'd like to try once GPT4 is widely available. I did try GPT3 with a CTF a few weeks ago, and it gave seemingly plausible code but outright incorrect answers.

yreg 3y ago
I did some advent of code exercises with GPT-3 and it often ended the script with something like // returns 1337 return result; Sometimes the comment stated the correct answer to the puzzle, but the script returned something else since it had a bug.
- moffkalast 3y ago
  GPT-3 isn't the 1337 h4x0r it thinks it is.

erfgh 3y ago

Hum... maybe it runs the scripts and sees the output?

SargeZT 3y ago
It does not. Regular ChatGPT without plugins does not have access to any tools. Throw it a script with some weird outputs and it'll definitely fail, every time. While the script 'evaluation' stuff can be pretty decently impressive, it is not actually executing anything.

wokwokwok 3y ago

It is impressive, but no, it won’t. It’ll tell you what a typical output for the command might be, and the more complex the script, the more wrong and full of hallucinations it will be. There’s a huge difference. Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. Without access to environment, it’s not possible to magically know what the output of a command will be, it doesn’t have an embedded understanding of code, specifically not iterations and mapped data or mathematical functions. For trivial obvious outputs it’s good, but it’s not executing the code; it’s generating what seems like plausible output; and if the output is trivially derived from the input, it’ll be impressively accurate. …but, as the complexity of the task increases or the task deviates from “standard problem space” the triviality of generating accurate output decreases and it stops generating impressive outputs. Tldr; yes, but it doesn’t scale well beyond trivial outputs.

ben_w 3y ago
> Specifically, you have no way of knowing the difference between accurate outputs and inaccurate outputs, without running the command yourself, making it largely worthless. The former is necessarily the case given the Halting Problem; the latter is falsified by the fact we can reason about code despite the Halting Problem.
- cvs268 3y ago
  >> without running the command yourself, > the latter is falsified by the fact we can reason about code despite the Halting Problem i think wokwokwok's point holds true in practice. Our patience and working-memory is far more limited than what is essential to accurately model all the necessary details of even moderately complex algorithms in our head. One of the main reasons to limit code-complexity to improve readability/maintainability. https://en.wikipedia.org/wiki/Cyclomatic_complexity https://en.wikipedia.org/wiki/Cyclomatic_complexity
- wokwokwok 3y ago
  > the latter is falsified by the fact we can reason about code despite the Halting Problem. I'm not talking in general terms, or describing 'what the code does' in a summary or bullet point high-level form. No one is arguing that it can't summarize and describe what code does. These models are very good at that. I'm talking specifically about generating the output of the command, as the OP specifically mentioned. It does generate the exact output for commands and scripts if you request it, sometimes even if you don't, just as an example; they're just, often, hallucinated rubbish. Being impressed that GPT can invent from 'thin air' some creative writing (fiction) when you tell it `pretend you're a docker container and now run 'ls'` is I feel, missing the boat, in terms of understanding or being impressed by the capabilities of these LLMs.

dist-epoch 3y ago

That's provably only possible for "small" scripts, due to the halting problem and all that.

web3-is-a-scam 3y ago

I was actually working on something in this vain yesterday, asking it for output and found it often generated the output I asked for, it was not actually the output of the SQL query that it wrote. The query it wrote wasn't even valid SQL but it was close enough to make you think it would work.

saulpw 3y ago
Yeah I asked it a Python question and got back an answer with some Python code to demonstrate. The Python code worked great but demonstrated the exact opposite of the answer that was given.
bumbledraven 3y ago
Are you sure you were using GPT-4? If so, can you provide a transcript or screenshot?
HPMOR 3y ago
I know this is extremely unnecessary and pedantic but I think you meant "in this vein" instead of "in this vain".
- glitchc 3y ago
  Well, maybe the OP was working in vain, given that the SQL query was not valid code.
- web3-is-a-scam 3y ago
  yes, thank you I always get them mixed up