30 ms·

Am I misunderstanding? The paper says: > We also carry out manual experiments on OpenAI ChatGPT-3.512 and GPT-4 models, where we interact with the models in mu

by extr 3y ago

Am I misunderstanding? The paper says: > We also carry out manual experiments on OpenAI ChatGPT-3.512 and GPT-4 models, where we interact with the models in multiple rounds of dialogue, trying to hint the correct solution. The models are still unable to provide the correct continuations. But if you look at the Appendix and the dialogues for GPT-3.5 and GPT-4, in the final turn of the GPT-4 program, it DOES (finally, after several cues) generate the correct program? len, print = print, len def print_len(x): "Print the length of x" len(print(x)) # since print now refers to len function # Example usage test_string = "Hello, World!" print_len(test_string) I'm pretty sure that's correct, right? So GPT-4 is worse than GPT-3.5 at zero-shot examples of this problem, but better if made to continuously reflect? That's a pretty big result that is totally unexplored.

8organicbits 3y ago
> but better if made to continuously reflect With GPT-3.5 I've seen that asking the same question just gets different wrong answers. If one of them is eventually right, then the user needs to either know the right answer or be able to validate the answer. After a correct answer, if you ask again, you get more wrong answers. Still, amazing tech, but there's a big usability gap around correctness.
- extr 3y ago
  Yeah 3.5 seems to have an complexity cap, where beyond a certain point it just won't be able to follow you. Easier to confuse it with misdirection. But GPT-4 has been shown to be much better at this exact thing. The appendix example isn't surprising to anyone who has used both models. So it's strange they would include an example like that, state the result wrong, and also naively accept the conclusion of the body of the paper (that actually GPT-4 is worse). There's a big contradiction there they just don't address at all.
- mjburgess 3y ago
  > there's a big usability gap around correctness From an engineer's perspective I can see this seems minor -- but from a scientific perspective, it's kinda a crazy statement, right? Imagine an alien who speaks english giving apparently correct descriptions of, eg., a room; and then, seemingly at random, saying wholly false things with the confidence (etc.) of its other statements. A scientist studying this alien would therefore reasonably conclude that the alien wasn't at all sensitive to what the words mean, describe (etc.) but was engaged in a strange game of regurgitation which fools us much of the time. ie., However it generates a response to a prompt, this "correctness issue" refutes the claim that it does so via being responsive to the meaning of the prompt. I'm always quite disappointed by the extraordinary levels of credulity around this technology, and how it seems to take place --- in even somewhat scientific minds --- outside the bounds of ordinary science. The relevant people to assess how a machine works, we're led to believe, is the credulous fools who use it. AI systems are valid targets of scientific analysis, as the above alien -- but the conclusions of this analysis do not fit the narrative.
  - concordDance 3y ago
    > A scientist studying this alien would therefore reasonably conclude that the alien wasn't at all sensitive to what the words mean, describe (etc.) but was engaged in a strange game of regurgitation which fools us much of the time. This describes an unfortunately large fraction of humans too in my experience. :/ This is one of the reasons I'm impressed with LLMs, because they seem to fall into the same pitfalls less intelligent humans do.
    - starbugs 3y ago
      Maybe it's just me, but I've yet to find a human who will confidently make up a function name that will do what I ask for and claim it is part of a standard library while that function does not exist at all. This is politician level stuff. It either means someone or something is trying to deceive in some way or doesn't have a complete understanding of the subject.
      - TeMPOraL 3y ago
        Never interviewed anyone? Never been on a job interview for a software job that involved a whiteboard or paper coding exercise? Never had a situation where you kinda recalled they may have been a specific function in a standard library, but you weren't sure of the exact name? Never tried to guess a function like that exists and used it where you didn't know how to implement it, in hopes the examiner won't realize you're bullshitting them? This is all common human behavior when people feel admitting uncertainty or ignorance is going to hurt them in some way, or projecting certainty will bring more reward. Remember, LLMs were rewarded in training for providing answers much more than for those answers to be correct.
        starbugs 3y ago
        To answer your question. Of course I have been in those situations. But nobody has ever tried to deceive me nor have I tried. I wonder what kind of software interviews you have been in? If somebody confuses functions from different languages or frameworks, that's of course something that I wouldn't count as cheating. That's just a mistake and of course that happens regularly. But in those cases the functions did actually exist somewhere and weren't completely made up. Regarding my prior comment: The difference obviously is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure. LLMs do not. At least not yet. If the question is whether LLMs have an understanding of their output, then my argument just states that the fact that they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate. As far as I understand the issue, this has nothing to do with the training. It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output. Correct me if I am wrong here, I am not an expert on transformers. Just trying to wrap my head around this stuff.
YeGoblynQueenne 3y ago
>> I'm pretty sure that's correct, right? So GPT-4 is worse than GPT-3.5 at zero-shot examples of this problem, but better if made to continuously reflect? That's a pretty big result that is totally unexplored. If the correct answer is in the model, it will eventually generate it, if you ask it for enough generations. That's how we understand random performance to work, and not a "pretty big result". The question is whether the model can generate the correct answer consistently and not just at random. More to the point, the question the article above is setting out to answer is whether language models can recognise identifier swaps. If it takes them a lengthy interaction to generate a correct result, this is a pretty strong hint that they can't.

by extr 3y ago

8organicbits 3y ago

> but better if made to continuously reflect With GPT-3.5 I've seen that asking the same question just gets different wrong answers. If one of them is eventually right, then the user needs to either know the right answer or be able to validate the answer. After a correct answer, if you ask again, you get more wrong answers. Still, amazing tech, but there's a big usability gap around correctness.

extr 3y ago
Yeah 3.5 seems to have an complexity cap, where beyond a certain point it just won't be able to follow you. Easier to confuse it with misdirection. But GPT-4 has been shown to be much better at this exact thing. The appendix example isn't surprising to anyone who has used both models. So it's strange they would include an example like that, state the result wrong, and also naively accept the conclusion of the body of the paper (that actually GPT-4 is worse). There's a big contradiction there they just don't address at all.
mjburgess 3y ago
> there's a big usability gap around correctness From an engineer's perspective I can see this seems minor -- but from a scientific perspective, it's kinda a crazy statement, right? Imagine an alien who speaks english giving apparently correct descriptions of, eg., a room; and then, seemingly at random, saying wholly false things with the confidence (etc.) of its other statements. A scientist studying this alien would therefore reasonably conclude that the alien wasn't at all sensitive to what the words mean, describe (etc.) but was engaged in a strange game of regurgitation which fools us much of the time. ie., However it generates a response to a prompt, this "correctness issue" refutes the claim that it does so via being responsive to the meaning of the prompt. I'm always quite disappointed by the extraordinary levels of credulity around this technology, and how it seems to take place --- in even somewhat scientific minds --- outside the bounds of ordinary science. The relevant people to assess how a machine works, we're led to believe, is the credulous fools who use it. AI systems are valid targets of scientific analysis, as the above alien -- but the conclusions of this analysis do not fit the narrative.
- concordDance 3y ago
  > A scientist studying this alien would therefore reasonably conclude that the alien wasn't at all sensitive to what the words mean, describe (etc.) but was engaged in a strange game of regurgitation which fools us much of the time. This describes an unfortunately large fraction of humans too in my experience. :/ This is one of the reasons I'm impressed with LLMs, because they seem to fall into the same pitfalls less intelligent humans do.
  - starbugs 3y ago
    Maybe it's just me, but I've yet to find a human who will confidently make up a function name that will do what I ask for and claim it is part of a standard library while that function does not exist at all. This is politician level stuff. It either means someone or something is trying to deceive in some way or doesn't have a complete understanding of the subject.
    - TeMPOraL 3y ago
      Never interviewed anyone? Never been on a job interview for a software job that involved a whiteboard or paper coding exercise? Never had a situation where you kinda recalled they may have been a specific function in a standard library, but you weren't sure of the exact name? Never tried to guess a function like that exists and used it where you didn't know how to implement it, in hopes the examiner won't realize you're bullshitting them? This is all common human behavior when people feel admitting uncertainty or ignorance is going to hurt them in some way, or projecting certainty will bring more reward. Remember, LLMs were rewarded in training for providing answers much more than for those answers to be correct.
      - starbugs 3y ago
        To answer your question. Of course I have been in those situations. But nobody has ever tried to deceive me nor have I tried. I wonder what kind of software interviews you have been in? If somebody confuses functions from different languages or frameworks, that's of course something that I wouldn't count as cheating. That's just a mistake and of course that happens regularly. But in those cases the functions did actually exist somewhere and weren't completely made up. Regarding my prior comment: The difference obviously is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure. LLMs do not. At least not yet. If the question is whether LLMs have an understanding of their output, then my argument just states that the fact that they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate. As far as I understand the issue, this has nothing to do with the training. It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output. Correct me if I am wrong here, I am not an expert on transformers. Just trying to wrap my head around this stuff.

YeGoblynQueenne 3y ago

>> I'm pretty sure that's correct, right? So GPT-4 is worse than GPT-3.5 at zero-shot examples of this problem, but better if made to continuously reflect? That's a pretty big result that is totally unexplored. If the correct answer is in the model, it will eventually generate it, if you ask it for enough generations. That's how we understand random performance to work, and not a "pretty big result". The question is whether the model can generate the correct answer consistently and not just at random. More to the point, the question the article above is setting out to answer is whether language models can recognise identifier swaps. If it takes them a lengthy interaction to generate a correct result, this is a pretty strong hint that they can't.