6 ms·

> there's a big usability gap around correctness From an engineer's perspective I can see this seems minor -- but from a scientific perspective, it's kinda a c

> there's a big usability gap around correctness From an engineer's perspective I can see this seems minor -- but from a scientific perspective, it's kinda a crazy statement, right? Imagine an alien who speaks english giving apparently correct descriptions of, eg., a room; and then, seemingly at random, saying wholly false things with the confidence (etc.) of its other statements. A scientist studying this alien would therefore reasonably conclude that the alien wasn't at all sensitive to what the words mean, describe (etc.) but was engaged in a strange game of regurgitation which fools us much of the time. ie., However it generates a response to a prompt, this "correctness issue" refutes the claim that it does so via being responsive to the meaning of the prompt. I'm always quite disappointed by the extraordinary levels of credulity around this technology, and how it seems to take place --- in even somewhat scientific minds --- outside the bounds of ordinary science. The relevant people to assess how a machine works, we're led to believe, is the credulous fools who use it. AI systems are valid targets of scientific analysis, as the above alien -- but the conclusions of this analysis do not fit the narrative.

concordDance 3y ago
> A scientist studying this alien would therefore reasonably conclude that the alien wasn't at all sensitive to what the words mean, describe (etc.) but was engaged in a strange game of regurgitation which fools us much of the time. This describes an unfortunately large fraction of humans too in my experience. :/ This is one of the reasons I'm impressed with LLMs, because they seem to fall into the same pitfalls less intelligent humans do.
- starbugs 3y ago
  Maybe it's just me, but I've yet to find a human who will confidently make up a function name that will do what I ask for and claim it is part of a standard library while that function does not exist at all. This is politician level stuff. It either means someone or something is trying to deceive in some way or doesn't have a complete understanding of the subject.
  - TeMPOraL 3y ago
    Never interviewed anyone? Never been on a job interview for a software job that involved a whiteboard or paper coding exercise? Never had a situation where you kinda recalled they may have been a specific function in a standard library, but you weren't sure of the exact name? Never tried to guess a function like that exists and used it where you didn't know how to implement it, in hopes the examiner won't realize you're bullshitting them? This is all common human behavior when people feel admitting uncertainty or ignorance is going to hurt them in some way, or projecting certainty will bring more reward. Remember, LLMs were rewarded in training for providing answers much more than for those answers to be correct.
    - starbugs 3y ago
      To answer your question. Of course I have been in those situations. But nobody has ever tried to deceive me nor have I tried. I wonder what kind of software interviews you have been in? If somebody confuses functions from different languages or frameworks, that's of course something that I wouldn't count as cheating. That's just a mistake and of course that happens regularly. But in those cases the functions did actually exist somewhere and weren't completely made up. Regarding my prior comment: The difference obviously is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure. LLMs do not. At least not yet. If the question is whether LLMs have an understanding of their output, then my argument just states that the fact that they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate. As far as I understand the issue, this has nothing to do with the training. It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output. Correct me if I am wrong here, I am not an expert on transformers. Just trying to wrap my head around this stuff.
      - TeMPOraL 3y ago
        > The difference is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure. My personal experience with my own thought process is that my un/sub-conscious generates thoughts I have to filter for accuracy and/or "send back" for reprocessing - this is a mostly conscious activity. My raw thought stream is about as good as LLM output - sounds right, but it's often full of nonsense. That's why I tend to say LLM output is akin to that unfiltered inner voice output. My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being accurate. They don't track provenance of their information. They uncritically believe whatever bullshit they read in lifestyle publications on-line. They say things confidently that they never thought much about, and have no first clue if it's actually true or not. Very much like LLMs - there's a whole class of topics where I know to assume the interlocutor is most likely "hallucinating" things. Also, fight-or-flight reaction may make you switch into full-on bullshitting mode, without it being intentional. > they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate And my argument is that this is the same thing a person's inner voice does. Deep understanding of a topic may make your inner voice output you more of the correct associations, but ultimately, most of the time, people have to have a back-and-forth with themselves in their own head to then say/write/do something correctly. With more complex (or less understood) tasks, this involves consciously performing an explicit step-by-step algorithm you track in your mind (and checklists in aviation and medicine are further externalization of that idea - to protect against you misremembering (or "hallucinating") the steps). > It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output. The output contains probability factors for predicted tokens, but chat interface doesn't expose it, and doesn't make a good use of it.
adhesive_wombat 3y ago
Speaking of aliens, the GPTs sound a lot like the aliens from Blindsight: they don't question the semantic ambiguity in > our cousins lie about the family tree with nieces and nephews and Neanderthals. We do not like annoying cousins. They just reply to it regardless and carry on. The aliens: "We'd like to know about this tree" GPT-4: > I'm sorry to hear that there's some family discord. > [Snip, about 200 words] > Nonetheless, I understand that your issue isn't literally about Neanderthals, but rather the analogy representing the annoyance you feel towards your cousins' behavior.
- mjburgess 3y ago
  Sure -- if we want to play the engineers game of "define a computational test by which we can know if X is intelligent" (ironically, a dumb game) -- this is one issue we can see here immediately ChatGPT doesnt ask us to write for it; does not seek to resolve the ambiguity in our speech. Indeed, the very way we "engineer prompts" kinda exposes the whole show -- that prompts need "engineered" at all, that the machine cannot prompt us -- makes it clear it's a tool we play a game with it to get out of it what we need by eliminating thinking from our own heads, and replacing it with word association --- in exactly the same way we've always done when googling
rad88 3y ago
It doesn't seem that crazy. Working from a simpler analogy first -- if an alien can read 95% of written English, but completely mischaracterizes the remaining 5%, would a reasonable scientist conclude that they can't read at all and are just guessing randomly? No, and it would indeed be unreasonable to conclude that there's no way to write differently so that the alien performs substantially better. We know some more about LLMs than we do aliens. The LLM is a neural network whose parameters have been optimized to reduce the total size of its errors, as measured against an enormous set of empirical data. We have to add to your analogy then, that it's somehow known the alien speaks English on their own planet remarkably well. They are still not perfectly correct, but when they have to describe something on that planet, in English, they can do it better than they generally can given the same task on Earth. It would again be totally unscientific to conclude that because of this, there is no way the alien can talk about earthly objects or ideas. The impulse of the mere engineer, to just hack around and find out, is a scientific one.

concordDance 3y ago

> A scientist studying this alien would therefore reasonably conclude that the alien wasn't at all sensitive to what the words mean, describe (etc.) but was engaged in a strange game of regurgitation which fools us much of the time. This describes an unfortunately large fraction of humans too in my experience. :/ This is one of the reasons I'm impressed with LLMs, because they seem to fall into the same pitfalls less intelligent humans do.

starbugs 3y ago
Maybe it's just me, but I've yet to find a human who will confidently make up a function name that will do what I ask for and claim it is part of a standard library while that function does not exist at all. This is politician level stuff. It either means someone or something is trying to deceive in some way or doesn't have a complete understanding of the subject.
- TeMPOraL 3y ago
  Never interviewed anyone? Never been on a job interview for a software job that involved a whiteboard or paper coding exercise? Never had a situation where you kinda recalled they may have been a specific function in a standard library, but you weren't sure of the exact name? Never tried to guess a function like that exists and used it where you didn't know how to implement it, in hopes the examiner won't realize you're bullshitting them? This is all common human behavior when people feel admitting uncertainty or ignorance is going to hurt them in some way, or projecting certainty will bring more reward. Remember, LLMs were rewarded in training for providing answers much more than for those answers to be correct.
  - starbugs 3y ago
    To answer your question. Of course I have been in those situations. But nobody has ever tried to deceive me nor have I tried. I wonder what kind of software interviews you have been in? If somebody confuses functions from different languages or frameworks, that's of course something that I wouldn't count as cheating. That's just a mistake and of course that happens regularly. But in those cases the functions did actually exist somewhere and weren't completely made up. Regarding my prior comment: The difference obviously is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure. LLMs do not. At least not yet. If the question is whether LLMs have an understanding of their output, then my argument just states that the fact that they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate. As far as I understand the issue, this has nothing to do with the training. It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output. Correct me if I am wrong here, I am not an expert on transformers. Just trying to wrap my head around this stuff.
    - TeMPOraL 3y ago
      > The difference is that in humans this is intentional behavior unless the human is not in a healthy state memory-wise. Hence my initial comment about being deceptive. We usually know when we don't know something for sure. My personal experience with my own thought process is that my un/sub-conscious generates thoughts I have to filter for accuracy and/or "send back" for reprocessing - this is a mostly conscious activity. My raw thought stream is about as good as LLM output - sounds right, but it's often full of nonsense. That's why I tend to say LLM output is akin to that unfiltered inner voice output. My experience with other people is that plenty of them - arguably more than 50% I met in physical space and talked with at length - care little about being accurate. They don't track provenance of their information. They uncritically believe whatever bullshit they read in lifestyle publications on-line. They say things confidently that they never thought much about, and have no first clue if it's actually true or not. Very much like LLMs - there's a whole class of topics where I know to assume the interlocutor is most likely "hallucinating" things. Also, fight-or-flight reaction may make you switch into full-on bullshitting mode, without it being intentional. > they obviously aren't trained to deceive and still emit this kind of output likely means that they don't have a complete understanding of what they generate And my argument is that this is the same thing a person's inner voice does. Deep understanding of a topic may make your inner voice output you more of the correct associations, but ultimately, most of the time, people have to have a back-and-forth with themselves in their own head to then say/write/do something correctly. With more complex (or less understood) tasks, this involves consciously performing an explicit step-by-step algorithm you track in your mind (and checklists in aviation and medicine are further externalization of that idea - to protect against you misremembering (or "hallucinating") the steps). > It's a general problem with the transformer architecture that there is no way to estimate the confidence of the output. The output contains probability factors for predicted tokens, but chat interface doesn't expose it, and doesn't make a good use of it.

adhesive_wombat 3y ago

Speaking of aliens, the GPTs sound a lot like the aliens from Blindsight: they don't question the semantic ambiguity in > our cousins lie about the family tree with nieces and nephews and Neanderthals. We do not like annoying cousins. They just reply to it regardless and carry on. The aliens: "We'd like to know about this tree" GPT-4: > I'm sorry to hear that there's some family discord. > [Snip, about 200 words] > Nonetheless, I understand that your issue isn't literally about Neanderthals, but rather the analogy representing the annoyance you feel towards your cousins' behavior.

mjburgess 3y ago
Sure -- if we want to play the engineers game of "define a computational test by which we can know if X is intelligent" (ironically, a dumb game) -- this is one issue we can see here immediately ChatGPT doesnt ask us to write for it; does not seek to resolve the ambiguity in our speech. Indeed, the very way we "engineer prompts" kinda exposes the whole show -- that prompts need "engineered" at all, that the machine cannot prompt us -- makes it clear it's a tool we play a game with it to get out of it what we need by eliminating thinking from our own heads, and replacing it with word association --- in exactly the same way we've always done when googling

rad88 3y ago

It doesn't seem that crazy. Working from a simpler analogy first -- if an alien can read 95% of written English, but completely mischaracterizes the remaining 5%, would a reasonable scientist conclude that they can't read at all and are just guessing randomly? No, and it would indeed be unreasonable to conclude that there's no way to write differently so that the alien performs substantially better. We know some more about LLMs than we do aliens. The LLM is a neural network whose parameters have been optimized to reduce the total size of its errors, as measured against an enormous set of empirical data. We have to add to your analogy then, that it's somehow known the alien speaks English on their own planet remarkably well. They are still not perfectly correct, but when they have to describe something on that planet, in English, they can do it better than they generally can given the same task on Earth. It would again be totally unscientific to conclude that because of this, there is no way the alien can talk about earthly objects or ideas. The impulse of the mere engineer, to just hack around and find out, is a scientific one.