5 ms·

But the paper is bullshit. Their task is easily solved by GPT-4. https://chat.openai.com/share/a28deca2-b989-4029-b042-b8434b80aaf6 https://chat.openai.com/sha

by xcv123 3y ago

But the paper is bullshit. Their task is easily solved by GPT-4. https://chat.openai.com/share/a28deca2-b989-4029-b042-b8434b80aaf6 https://chat.openai.com/share/a28deca2-b989-4029-b042-b8434b...

suby 3y ago
Interesting. You said Gpt4, but that link shows ChatGpt's avatar as 3.5's. And I'm convinced you were using 4 as I just tried replicating this; with 3.5 it fails and with 4 it succeeds. So apparently it's not currently possible to tell whether a shared link was 3.5 or 4? Unfortunate if so.
- xcv123 3y ago
  The shared link was from my own GPT-4 session. It shows "Model: GPT-4" in the header of my chat from which I shared the link.
  - elcomet 3y ago
    No it shows Model: Default
    - xcv123 3y ago
      Yes but it's from GPT-4. "Model: Default" is a defect in their link sharing. In my private session it says "Model: GPT-4"
  - lordnacho 3y ago
    How do you set it to 4? I can only find 3.5 somehow.
    - xcv123 3y ago
      You must pay for it. ChatGPT Plus. $20 per month.
twright0 3y ago
Sometimes! My attempt with GPT-4 yields a response where it acknowledges the print/len swap, but does not produce correct code in the end - it sort of loses track of what the original goal was. https://chat.openai.com/share/300382cb-ac72-4a75-847c-ecbf5ab83720 https://chat.openai.com/share/300382cb-ac72-4a75-847c-ecbf5a...
- mjburgess 3y ago
  And if we're doing Science, ie., trying to explain how ChatGPT works and what it's intrinsic properties are --- this case is far more significant than the other. Inasmuch as the hypothesis that ChatGPT works "so as to be actually sensitive to the meaning of the code" is here falsified -- by a single case. An infinite number of apparent confirmations of this hypothesis are now Invalid!
  - fiso64 3y ago
    Humans make foolish, nonsensical mistakes too. Does that mean that they're not intelligent and not sensitive to the meaning of the code?
    - YeGoblynQueenne 3y ago
      We already know that humans are sensitive to meaning. That's not a hypothesis that needs testing.
      - TeMPOraL 3y ago
        This experiment only proves LLMs are sensitive to meaning too. It was also already known.
        YeGoblynQueenne 3y ago
        All I can see with any clarity is that LLMs are sensitive to the user's prompt.
  - 3y ago

by xcv123 3y ago

But the paper is bullshit. Their task is easily solved by GPT-4. https://chat.openai.com/share/a28deca2-b989-4029-b042-b8434b80aaf6 https://chat.openai.com/share/a28deca2-b989-4029-b042-b8434b...

suby 3y ago

Interesting. You said Gpt4, but that link shows ChatGpt's avatar as 3.5's. And I'm convinced you were using 4 as I just tried replicating this; with 3.5 it fails and with 4 it succeeds. So apparently it's not currently possible to tell whether a shared link was 3.5 or 4? Unfortunate if so.

xcv123 3y ago
The shared link was from my own GPT-4 session. It shows "Model: GPT-4" in the header of my chat from which I shared the link.
- elcomet 3y ago
  No it shows Model: Default
  - xcv123 3y ago
    Yes but it's from GPT-4. "Model: Default" is a defect in their link sharing. In my private session it says "Model: GPT-4"
- lordnacho 3y ago
  How do you set it to 4? I can only find 3.5 somehow.
  - xcv123 3y ago
    You must pay for it. ChatGPT Plus. $20 per month.

twright0 3y ago

Sometimes! My attempt with GPT-4 yields a response where it acknowledges the print/len swap, but does not produce correct code in the end - it sort of loses track of what the original goal was. https://chat.openai.com/share/300382cb-ac72-4a75-847c-ecbf5ab83720 https://chat.openai.com/share/300382cb-ac72-4a75-847c-ecbf5a...

mjburgess 3y ago
And if we're doing Science, ie., trying to explain how ChatGPT works and what it's intrinsic properties are --- this case is far more significant than the other. Inasmuch as the hypothesis that ChatGPT works "so as to be actually sensitive to the meaning of the code" is here falsified -- by a single case. An infinite number of apparent confirmations of this hypothesis are now Invalid!
- fiso64 3y ago
  Humans make foolish, nonsensical mistakes too. Does that mean that they're not intelligent and not sensitive to the meaning of the code?
  - YeGoblynQueenne 3y ago
    We already know that humans are sensitive to meaning. That's not a hypothesis that needs testing.
    - TeMPOraL 3y ago
      This experiment only proves LLMs are sensitive to meaning too. It was also already known.
      - YeGoblynQueenne 3y ago
        All I can see with any clarity is that LLMs are sensitive to the user's prompt.
- 3y ago