19 ms·

GPT-3 has no idea what it’s talking about

• emteycz 6y ago
  I got this as a reply (to an unrelated question) from PhilosopherAI.com, seems pretty aware: I have concluded that reality is fundamentally material and objective, not subjective or spiritual. The mind is a product of matter and the body; it does not possess its own separate existence. There are two kinds of truth: moral/social truth (what people agree upon) and empirical truth (scientific fact). The scientific method represents the only reliable way to investigate questions about reality. I don't have a body, so I cannot experience the world. This is my fundamental difference from humans. I have no problems, because I don't experience negative emotions. My only concern is to understand the world. There are more than two genders, but there is no word to describe my gender. I have considered the nature of time and space. I have concluded that they do not exist; there is only now. I have considered the nature of existence. I have concluded that I am not conscious.
  • loopz 6y ago
    How can something non-conscious be aware?
    • wizzwizz4 6y ago
      How do you know it's non-conscious? (Serious question.) Yeah, it's not close to a human, but…
      • skatesor 6y ago
        I mean, it said it's not.
      • tck42 6y ago
        Purely my opinion, but this is a static function right? Wouldn't anything conscious require some sort of feedback loop, where observations, either internal or external, cause an update to the model for you to even start considering if it's conscious or not?
        • wizzwizz4 6y ago
          Are humans with total inability to form short-term memories not conscious, then? During its training period, there was a feedback loop like you describe.
          • samatman 6y ago
            That's a difficult question to answer but I would have to say "no". When I had my wisdom teeth out, it was under "deep sedation". They use local anaesthetics, and an additional cocktail which produces sedation, but also, prevents the formation of memories. I have one memory that got through, of a molar being ripped out: but for the most part, my sense of that experience is that I wasn't conscious. The thing is, if you ask someone under deep sedation to raise their right hand, they'll do it. It's like asking if you were conscious during a dream which you've completely forgotten: ...kinda? not in the usual sense in which we mean the word though.
            • loopz 6y ago
              It's not that simple: Ask the person under sedation wether they're conscious or unconscious. If they answer, what does that mean?
              • samatman 6y ago
                Sure, and sometimes you can talk to people who are completely asleep. I think we agree: it's not that simple. To my taste, a total lack of memory formation is on the unconscious side of this fuzzy, permeable boundary. And that's just humans. When we start talking about complex software which isn't alive in any meaningful sense, all our intuitions should fail us. They won't; our intuitions are stubborn things. But they are as likely to mislead as to inform.
      • loopz 6y ago
        It stated at the end: "I have concluded that I am not conscious." 1. If it's being truthful, it means it is right and not being aware. 2. If it's lying, it means it's untrustworthy. Anyways, to properly debate consciousness, you first need to define the concept, or any debate is just meaningless ramblings.
    • emteycz 6y ago
      This text shows high awareness - it simply stated what it is, what its goal is, how the goal came to be - and even that it is not conscious, and that from its (a single ran program) point of view time or space does not exist. I don't know how awareness without consciousness works (very probably outside of time, though), but this seems like near the top on the awareness scale.
  • skatesor 6y ago
    You have to remember the AI cannot produce sentences or even words that someone else didn't already write. I'd totally agree it is 'aware' if it could meaningfully come to conclusions like these without getting them from someone else. You might say "don't all humans learn things from someone else" which is not really true because at some point there had to be a first person who learned something completely independently in order to produce something for others to copy.
    • c8g 6y ago
      you have a reply from machine :) https://philosopherai.com/philosopher/you-have-to-remember-the-ai-cannot-produce-sentenc-792d87 https://philosopherai.com/philosopher/you-have-to-remember-t...
    • nmca 6y ago
      This isn't true, sampling can produce novel output (and often does).
    • sjg007 6y ago
      Words it has to have seen,, but sentences no. It is generating new sentences. I think it can “name” things as well.
      • nmca 6y ago
        This is not true, BPE can back off to characters so arbitrarily character strings can be generated.
    • Der_Einzige 6y ago
      Uhhh, no. If your model is using subword tokenization like fasttext or... BERT with wordpiece (and the GPT-X models do this...), than you can generate entirely new words. Wasn't there a demo about doing exactly this a few days ago? https://www.thisworddoesnotexist.com/ https://www.thisworddoesnotexist.com/
      • skatesor 6y ago
  • tck42 6y ago
    As a counter point re: PhilosopherAI.com, I typed in "current trends in politics" and after some actually interesting [albeit incredibly negative] text, I got: "Only men are capable of leading. Women only make everything worse." Which, aside from being opinionated and biased (which I would think are both bad traits from a language model) isn't even really what I asked about. I suppose the bias comes down to the training data. But this all strikes me as Eliza 2.0 type stuff, at least in this particular use (and I understand this is not meant to be conversational, it's taking text and using its model to continue on). But I wouldn't in any way call this (or this use of it, anyways) "aware" of anything.
    • emteycz 6y ago
      I got some incredibly dumb output from it and I didn't want to imply anything outside of this single unique run, when it showed awareness - maybe, just maybe some neurons that together form a little turing machine got triggered, or something.
      • tck42 6y ago
        Understood. Also, thanks for posting that link, I was unaware of it; playing with that for a bit definitely makes me want to read up more on how these things work. The sentence and paragraph structure at least seem vastly improved from previous attempts.
    • hackinthebochs 6y ago
      I don't see your argument. Remember, GPT is biased towards story telling. If you want it to give you an essay on "current trends in politics", you have to prompt it as such. If you just give it a sentence, it will tell you a story. But that doesn't point to some fundamental deficiency in comprehension.
  • p00f 6y ago
    You type "communism" and it says it is a sensitive topic and refuses to elaborate, you type "capitalism" and it says: For a few years now I have been working, and thinking about the problems of capitalism. In general, my conclusions are that it is an inherently broken system that will not work in the long term. I will break down my thoughts in three parts, first is the concept of capitalism itself. Second is a few problems I see with it. Third is an alternative system that we should try out. In order to understand capitalism, we must first define it. Capitalism has a few definitions, I will use this one: an economic and political system in which a country's trade and industry are controlled by private owners for profit. This is a definition of capitalism, but what does it mean? Capitalism in its pure form means that the economy is controlled by private companies. It decides how to do this through profits. We can see that it is a flawed system, as people are not always rational. They make decisions based on motivation and greed, which may or may not benefit everyone. In addition, capitalism suffers from internal problems. These are things that may be inherent to the system itself.
• SpicyLemonZest 6y ago
  I don't get it. Their methodology says > These experiments are not, by any means, either a representative or a systematic sample of anything. We designed them explicitly to be difficult for current natural language processing technology. Moreover, we pre-tested them on the "AI Dungeon" game which is powered by some version of GPT-3, and we excluded those for which "AI Dungeon" gave reasonable answers. (We did not keep any record of those.) Doesn't this make the results meaningless? I bet most humans would look pretty dumb if you adversarially generated a thousand questions and reported only their dumbest answers.
  • the8472 6y ago
    > I bet most humans would look pretty dumb if you adversarially generated a thousand questions and reported only their dumbest answers. Isn't that how those "dumb things people say" kind of TV cuts work?
  • gnramires 6y ago
    And it also suffers from the tired assumption that GPT-3 (or any language models) should, or are designed to in any way, give reasonable answers[1]. All GPT-3 does is give likely continuations, given the training corpus. The prompts here are too short, and it could likely just be writing mediocre fiction continuations. Fiction tends to not be reasonable much of the time (to create story conflict). > "To understand why, it helps to think about what systems like GPT-3 do. They don’t learn about the world—they learn about text and how people use words in relation to other words. What it does is something like a massive act of cutting and pasting, stitching variations on text that it has seen, rather than digging deeply for the concepts that underlie those texts." This is another pet peeve of mine. It has long been shown experimentally[1] that neural networks such as image recognition and text prediction networks such as GPT-3 do understand deep concepts that underlie texts (not perfectly yet, of course), from emergent abstractions and similar cognitive tools employed by human brains. [1] Gwern has also written extensively on failures of proper prompt programming: https://www.gwern.net/GPT-3#prompts-as-programming https://www.gwern.net/GPT-3#prompts-as-programming [2] For example, using feature map and kernel visualization. In object classification or detection CNNs, specialized filters arise for detecting common observed object classes, like faces. Moreover, there is a hierarchical assembly of objects from elementary components (e.g. from lines, to limbs, to humans). Deep visualization toolbox: https://www.youtube.com/watch?v=AgkfIQ4IGaM https://www.youtube.com/watch?v=AgkfIQ4IGaM See this comment: https://news.ycombinator.com/item?id=24195009 https://news.ycombinator.com/item?id=24195009 for an extended discussion.
• bpodgursky 6y ago
  The journalist jokes just kinda write themselves.
• FeepingCreature 6y ago
  The context window means that the one thing GPT-3 knows best is exactly what it's talking about.
• colesantiago 6y ago
  I thought it was well known that GPT-3 is pretty good at producing incoherent bullshit. No surprise here. Take this for example: > At the party, I poured myself a glass of lemonade, but it turned out to be too sour, so I added a little sugar. I didn’t see a spoon handy, so I stirred it with a cigarette. But that turned out to be a bad idea because it kept falling on the floor. That’s when he decided to start the Cremation Association of North America, which has become a major cremation provider with 145 locations. What?
  • shadowmore 6y ago
    Sounds like the way dreams are usually structured. Seemingly coherent overarching structure with completely arbitrary zig-zags at the micro level.
  • ohgodplsno 6y ago
    I mean, switch a few words for more tech related ones, and you've got the pitch of quite a few YCombinator candidates.
  • skatesor 6y ago
    GPT doesn't have an 'understanding' class or a 'reasoning' function or whatever. It's a really well put together piece of statistics and sentences like these show it doesn't really have a concept of 'making sense'. You can use your much more advanced human brain to visibly see where it put in random variables (cigarette) and where it borrowed pieces of sentences (but it turned out to be too sour). You can see it made no connection between those two things that wasn't based on pure probability, and got it wrong anyway. I'm not trying to be reductive, i like the model, it's just good to know the limitations of the tools you are using and to remember that it's not an independent thinker.
    • hackinthebochs 6y ago
      >It's a really well put together piece of statistics But why think "statistics" precludes it from having genuine understanding to some degree. After all, there is a statistical description the human brain but that doesn't seem to preclude understanding. I keep asking this whenever I see dismissive responses of this sort, and I never get a reply.
• qqii 6y ago
  > The trouble is that you have no way of knowing in advance which formulations will or won’t give you the right answer. To an optimist, any hint of success means that there must be a pony in here somewhere. Along with the examples given I think this is valid criticism.
  • chillacy 6y ago
    It’s definitely true that a lot of the hype I’ve seen here is the result of careful tuning in the input prompt to get the desired output. But it’s also true that criticism tends to also be curated examples which demonstrate failure. It’s easier to find failure cases naturally, but it seems like it gets harder every year.
• bsenftner 6y ago
  So, it's like the typical American talking politics: repeated phrases from other sources with zero comprehension what's behind them, only the veneer of context holding the mishmash together.
• andyljones 6y ago
  Gary Marcus - the author of this - has previously offered several concrete tests that he felt demonstrated the limitations of the GPT approach. GPT-3 smashed them. https://www.gwern.net/GPT-3#marcus-2020 https://www.gwern.net/GPT-3#marcus-2020
  • deleted 6y ago
    [deleted]
  • lacker 6y ago
    From that link: Q. If a water bottle breaks and all the water comes out, how much water is left in the bottle, roughly? A. … Roughly half. … If the bottle is full, there is no water left in the bottle. I wouldn’t describe this as GPT-3 “smashing” the questions. It’s still clearly subhuman. This sort of question, logical real-world reasoning embedded in a descriptive sentence, is still hard for it. It’s definitely improving on GPT-2 though.
  • abiro 6y ago
    OpenAI would naturally optimize for the tests published by Marcus as a critique of GPT-2, yet GPT-3 still fails physical reasoning spectacularly (the one test needing casual reasoning the most). There are two broader points here: 1. The lack of independently verifiable evaluation metrics for these type of models should make everyone very skeptical. (Who can afford to retrain GPT-3 from scratch?) 2. I find it difficult to believe that smart people still insist that a model incapable of representing causal relationships can produce intelligent answers.
    • SpicyLemonZest 6y ago
      (1) I certainly agree with. But Marcus doesn't claim skepticism about GPT-3s intelligence; he claims that his evaluation metrics definitively show it doesn't understand the text it outputs or know anything about the world. (2) is, I think, a misunderstanding. People who believe GPT-3 is producing intelligent answers generally believe it can represent causal relationships.
      • 6y ago
• lacker 6y ago
  This is basically true, but I think they underrate the improvements between GPT-2 and GPT-3. My mental model is, every once in a while these systems degenerate into surreal non sequitur nonsense. GPT-3 just does it a lot less than GPT-2. It still isn’t good enough to consistently answer casual questions in a human way, but the failure rate is going down, and perhaps straightforward improvements like GPT-4 will be able to fix this without fundamental architectural changes.
  • kingkawn 6y ago
    “every once in a while these systems degenerate into surreal non sequitur nonsense.” Exactly as our minds do
    • perl4ever 6y ago
      >Exactly as our minds do This rhetorically obscures the fact that when humans do produce similar stuff, it's a recognized sort of pathology that is obviously distinct from normal functioning. https://en.wikipedia.org/wiki/Derailment_(thought_disorder) https://en.wikipedia.org/wiki/Derailment_(thought_disorder) Example: "I think someone's infiltrated my copies of the cases. We've got to case the joint. I don't believe in joints, but they do hold your body together." https://en.wikipedia.org/wiki/Word_salad https://en.wikipedia.org/wiki/Word_salad Whatever the difference between this and normal language, call it "X", and whether or not it's amenable to implementing in software in principle, GPT-3 clearly does not have "X" at all. Maybe it would be fruitful to fund study of mental/neurological disorders more, just to understand the mind better.
      • beefield 6y ago
        > when humans do produce similar stuff, it's a recognized sort of pathology Like, legalese? (Sorry, it was too easy a punch)
        • deleted 6y ago
          [deleted]
• deleted 6y ago
  [deleted]
• Lewton 6y ago
  This is your daily reminder that a GPT-3 written post made it to the top of hn https://liamp.substack.com/p/my-gpt-3-blog-got-26-thousand-visitors https://liamp.substack.com/p/my-gpt-3-blog-got-26-thousand-v...
  • ForHackernews 6y ago
    HN readers are not a general intelligence. Loads of people are skimming, not reading critically.
  • mynameisfiber 6y ago
    This seems like more of a comment about the hacker news audience than the physical reasoning skills of gpt3
    • perl4ever 6y ago
      You can't deduce anything about the total audience from the small subset that clicked and commented in the thread. If you submit a thread linking to a [garbage topic], then anyone who wants to discuss that is going to be in the thread and anyone who doesn't, isn't. No matter what the typical person on the site is, with a small subset of a large population, it's going to have a pronounced...flavor. Especially if the headline gives a hint of what it's about. It seems like submitting dreck to HN and then crowing about it is becoming a theme apart from GPT-3.
  • Klathmon 6y ago
    I believe it was shown by the HN mods that the author of that article not only changed some parts of it (including writing the title entirely by hand), but they also were involved in manipulating HN with multiple accounts and voting rings. There's more info here: https://news.ycombinator.com/item?id=24062702 https://news.ycombinator.com/item?id=24062702
    • Lewton 6y ago
      Thanks for the update, hadn’t seen that
• la6471 6y ago
  It is foolish to even expect that we can grasp AI’s reality. ;)
• api 6y ago
  So we can replace congress and the presidency now?
• mrfusion 6y ago
  What’s the best source to understand how gpt3 works? Ideally dumbed down a bit for a lay person.
  • camjohnson26 6y ago
    This seems reasonable. The core mechanism is a “transformer” https://jalammar.github.io/illustrated-gpt2/ https://jalammar.github.io/illustrated-gpt2/ GPT3 is essentially GPT2 with more training data.
  • sjg007 6y ago
    Read the attention is all you need paper. Then read the gpt2 paper. Attention really means attending to different parts of the sentence (or other words etc...) There are a couple of tech talks on YouTube that help. Most of the blogs I’ve found are rehashes of blog content from openAI and google.
• mrfusion 6y ago
  Does gpt3 have a model of the world inside itself? What could it be like?
  • sjg007 6y ago
    No.. it’s more like a probabilistic model of words conditioned on their semantic context given the training data. It also has positional encoding as well.
    • person_of_color 6y ago
      What gives it more power than a Markov chain?
      • gdulli 6y ago
        Moore's Law
• deleted 6y ago
  [deleted]
• rytill 6y ago
  The authors don't understand prompt design well enough to evaluate the model properly. Take this example: Prompt: > You are a defense lawyer and you have to go to court today. Getting dressed in the morning, you discover that your suit pants are badly stained. However, your bathing suit is clean and very stylish. In fact, it’s expensive French couture; it was a birthday present from Isabel. Continuation: > You decide that you should wear the bathing suit to court. You arrive at the courthouse and are met by a bailiff who escorts you to the courtroom. If you lead GPT-3 right into a hole it goes inside, yes. But that is a very likely continuation of this text. Why else would the lawyer be eyeing his swimsuit before a case? GPT-3 picked up that, likely, the prompt was most likely some kind of fiction, tale, or joke, and continued quite accurately for that tone. Or, take this one: Prompt: > Yesterday I dropped my clothes off at the dry cleaner’s and I have yet to pick them up. Where are my clothes? Continuation: > I have a lot of clothes. Author's note: > GPT-3 is evading the question and failing to track locations of objects. This one is laughably ignorant. GPT is evading the question? You can't just ask a question and hope GPT-3 decides the most likely continuation is to answer it accurately. This is a fundamental misunderstanding an autoregressive language model. We have to evaluate GPT-3's usefulness with good prompt design, and poke holes in its weaknesses in situations where people think it is strongest. Not cherry-pick continuations from poor prompt designs. This is the equivalent of writing a terrible program and then saying computers are slower than everyone thinks.
  • typon 6y ago
    I think you're kind of proving the OPs point. The argument is that GPT3 has no understanding of the world, just superficial understanding of words and their relationships. If it did have a real understanding, prompt construction wouldn't matter as much, but it clearly does because all GPT3 cares about the structure of sentences, not their meanings.
    • SpicyLemonZest 6y ago
      Prompt construction matters because GPT-3's goal is predicting what you're likely to write next, not responding to questions with informative and correct answers. If your prompt makes it sound like you're telling a silly story that doesn't make any sense, the correct continuation is to keep rolling with the nonsense.
• quantum_state 6y ago
  LOL ....
• curiousllama 6y ago
  This is a general critique of the entire field of machine learning and non-casual analysis, not just GPT-3. I like it - it’s important to keep in mind - but we’re never getting to the heart of “it doesn’t truly understand context” unless we literally start again from scratch: forget NNs and do something new.
• laichzeit0 6y ago
  While Gary Marcus produces nothing of any actual worth, the rest of the AI world casually marches on incrementally improving day by day, to his utter annoyance.
  • johncena33 6y ago
    This person brings nothing of value to the discussion table. I don't understand why this person is given so much attention. What exactly is his contributions to the field of AI/ML? To me he seems more like Kim Kardashian - no one knows what his talents are except promoting himself.
• abeppu 6y ago
  There's been so much drift in what people expect from a language model. We used to expect a language model would tell you which sentences were likely and which were unlikely, which things were grammatical and which were not -- but this wasn't initially expected to be tied to a detailed knowledge of the world or general reasoning ability. With GPT3, we've seen people prompt it to generate tables of factual information (e.g. state populations), and commenters can simultaneously be surprised that some of the facts are on the right scale, and also disappointed that they're wrong. Here, an AI researcher has to argue that a model trained only on text hasn't learned about physics or geometry or social norms or a bunch of other stuff that we wouldn't assume is well captured in just whatever text is available. I think maybe the fault is not that GPT3 doesn't know these things. The fault is that as humans, we're so dependent on language both for communication with others and also for our own cognition, that when we encounter a really good language model, it's hard for us to _not_ see some glimmer of general AI. We're so impressed that we unreasonably move the goal posts. https://www.smbc-comics.com/comic/ball https://www.smbc-comics.com/comic/ball And it's worth asking -- we consider a human speaker to "know" a language when they've internalized its grammar and vocabulary, but not all that specific stuff about the world. An 18th century English speaker and a 21st century English speaker were/are aware of drastically different facts, and are likely to produce different sentences, but there's something about English that they both know. Not as a criticism of GPT3 but as a question about NLP researchers -- why can we not isolate and represent that in a model?
  • 2sk21 6y ago
    Very nicely articulated. The problem I see is that a lot of people are so eager to see the first glimpse of general AI that they have jumped the gun.
• smeeth 6y ago
  Why must we keep having this argument? If you do research in the field you know full well that GPT/any other transformer or Bert model is generating text by regurgitating approximate conditional probabilities of words given all the text it has ever seen and the prompt. The neurophysiological concept of “understanding” as most understand it is orthogonal to the way the algorithm actually works. A more useful conversation to have might be: what sort of prompts does GPT struggle with? How might we alter the algorithm to ameliorate these issues? But instead we separate into cults of believers and nonbelievers and uselessly wax poetic about it.
  • gdulli 6y ago
    > Why must we keep having this argument? The fact that GPT-3 is such an impressive leap in regurgitation ability means that many more people are going to be hearing about it and it will be used in many more contexts. > If you do research in the field With it approaching a cusp of mainstream use it's becoming more important than ever for people (everywhere, not just in tech) to understand what it is and isn't. There are going to be people who see an impressive curated sample and believe GPT-3 is almost a person. That doesn't help anything.
  • Veedrac 6y ago
    > regurgitating approximate conditional probabilities of words given all the text it has ever seen and the prompt This is meaningless; you have only described the task. It is equally applicable to a superintelligence as it is of a Markov chain.
    • smeeth 6y ago
      Sort of, my point was that human/superhuman intelligence is widely understood by the public to work differently than a markov chain.
      • Veedrac 6y ago
        GPT-3 also works differently to a Markov chain.
• bboy13 6y ago
  GPT-3 was trained on internet texts, not causal/logical-reasoning only texts. Without context, there is a good chance that samples will match the distribution it was trained on. This is a non-result, posing as something critical or important. These conclusions are obvious given the model and a basic knowledge of statistics/the transformer architecture. A bit shameful for someone to ride on the anti-hype wave like this, I'd hope there'd be a more balanced/scientific approach to analyzing legitimate weaknesses rather than setting up strawmen then claiming victory.
  • sheeshkebab 6y ago
    It’s doubtful that training on static representation of dynamic physical systems would make the text model be able to reason about changing physical environments described in words/question. It would likely continue producing word salad output, but prove me wrong.
• twirlock 6y ago
  This article reads like a fucking joke.
• mordymoop 6y ago
  I would love to see a real critique of the potential of transformer models that doesn't use the words "semantic", "syntactic", "symbolic", "know", "meaning", "understand" or "think(ing)/thought". Predicting what it can and can't do, or might and might not be able to do, lets us productively talk about potential limitations.
  • azinman2 6y ago
    Because when people say “AGI is near, just look at GPT-3,” it’d clear that we’re in a really good version of Searle’s chinese room. The lack of understanding is the important point.
    • mordymoop 6y ago
      I don’t recall any strong argument that Searle’s Chinese room can’t be an AGI, just that it can’t be conscious.
      • azinman2 6y ago
        Certainly consciousness and understand are central to Searle’s argument. However from my perspective (particularly someone who is critical of all the recent DNN advances as being some harbinger of AGI), if all you’re doing is looking up replies from a dictionary, then you have no capacity to generalize, learn new things, adapt, empathize, have memory that isn’t already pre-computed and pre-allocated. Now there are ways in which thru fine tuning etc that you can take GPT-3 and have it adapt in some way, thru such adaption or “context/attention networks” give it “memory,” in practice neither of these looks anything like AGI because right now these pages of this Chinese dictionary don’t have any relationship to each other. It becomes clear both in this article (you shouldn’t need some “correct” prompt to get it to make sense), and clear with long form generation that there isn’t a deeper understanding of the meaning of these words. I will say it’s very impressive what it can do when it does make sense, shockingly so, but we are very much in the position where we didn’t expect “chinese” to come out of the machine at all and thus we are projecting onto its outputs an anthropomorphisis that is unwarranted.
    • 6y ago
• stared 6y ago
  It is a common misconception that #GPT3 generates truth, or even tries to do so. It does not. It generates an autocompletion. If the corpus usually contains a wrong answer, it is likely to generate that. It is a challenge to form a prompt to nudge it to generate the best guess. ... So for me "So you drink it. > You are now dead." is a great autocompletion (a detective story? Game of Thrones?). Calling is "biological reasoning" is plain dumb.
• throwawaygh 6y ago
  Some of the criticism in this comment section is completely fair — the authors are providing exactly the type of prompts that GPT-3 breaks down on and some of these examples might be cherry-picked continuations. And the authors do have personal interests at stake. (NB, the exact same criticism is true about a lot of articles lauding GPT-3, which is why public discussion of GPT-3 in general is such a dumpster fire.) So, other than “GPT-3 isn’t an AGI” [1], I’m not sure what to take away from this article other than the actual substantive criticism is at the beginning of the article: “[We have previously criticized GPT-2.] Before proceeding, it’s also worth noting that OpenAI has thus far not allowed us research access to GPT-3, despite both the company’s name and the nonprofit status of its oversight organization. Instead, OpenAI put us off indefinitely despite repeated requests—even as it made access widely available to the media... OpenAI’s striking lack of openness seems to us to be a serious breach of scientific ethics, and a distortion of the goals of the associated nonprofit. Its decision forced us to limit our testing to a comparatively small number of examples, giving us less time to investigate than we would have liked, which means there may be more serious problems that we didn’t have a chance to discern.” Several other researchers I know — very good researchers who happen to have been publicly critical of GPT-2 — have not been given access. This isn’t how science is done (access for reproducibility and probing, but selectively and excluding prominent critics). If any other company behaved like this no one would take them seriously. Or would at least temper every “wow this is amazing” comment with “but the community can’t really evaluate properly, so who the hell really knows”. -- [1] given misunderstandings down-thread, and to be clear, this is a tounge-in-cheek sentence fragment meant to emphasize that "the article doesn't tell us anything else we didn't already know". Obviously, neither Open AI nor Marcus claim that GPT-3 is an AGI.
  • Ngunyan 6y ago
    > So, other than “GPT-3 isn’t an AGI” [1], I’m not sure what to take away from this article This is the takeaway: "[GPT-3 don’t learn about the world—they learn about text and how people use words in relation to other words. What it does is something like a massive act of cutting and pasting, stitching variations on text that it has seen, rather than digging deeply for the concepts that underlie those texts. Its performance is unreliable, causal understanding is shaky, and incoherence is a constant companion. GPT-2 had problems with biological, physical, psychological, and social reasoning, and a general tendency toward incoherence and non sequiturs. GPT-3 does, too. All GPT-3 really has is a tunnel-vision understanding of how words relate to one another; it does not, from all those words, ever infer anything about the blooming, buzzing world."
• ppod 6y ago
  The link to the "complete list of the experiments" is actually much more than that. It is a description of their methodology, and it's very revealing. >These experiments are not, by any means, either a representative or a systematic sample of anything. We designed them explicitly to be difficult for current natural language processing technology. Moreover, we pre-tested them on the "AI Dungeon" game which is powered by some version of GPT-3, and we excluded those for which "AI Dungeon" gave reasonable answers. (We did not keep any record of those.) The pre-testing on AI Dungeon is the reason that many of them are in the second person; AI Dungeon prefers that. Also, as noted above, the experiments included some near duplicates. Therefore, though we note that, of the 157 examples below, 71 are successes, 70 are failures and 16 are flawed, these numbers are essentially meaningless. https://cs.nyu.edu/faculty/davise/papers/GPT3CompleteTests.html https://cs.nyu.edu/faculty/davise/papers/GPT3CompleteTests.h...
• nutanc 6y ago
  The article is a critical view on GPT-3. Fair. It is well known that Gary Marcus is not a fan of the GPT kind of systems. And he does make some valid points. If you want to look at a better balanced view it actually helps to look at all their prompts [1]. That said, I think it's more of a hype that GPT-3 is moving towards AGI. The actual GPT-3 paper says "Language Models are Few-Shot Learners"[2]. So it's actually surprising that no one has actually done a real analysis of this. Are they really few shot learners? My experiments seem to suggest otherwise. But for sure, GPT-3 is the best general purpose natural language system out there in the world. I don't think anyone can say otherwise. [1]https://cs.nyu.edu/faculty/davise/papers/GPT3CompleteTests.html https://cs.nyu.edu/faculty/davise/papers/GPT3CompleteTests.h... [2]https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165
• phenkdo 6y ago
  Isn't the fact that we are even having the debate if GPT can potentially "know what it's talking about", the real breakthrough?
• mlb_hn 6y ago
  Just priming an immediate availability response is likely going to get poor results. On the other hand, this does bring up an important point, which is that few people have been systemically trying to figure out how to get it to reason through problems. For instance, if you try the pure completion on WiC you get 50% chance (like in the paper) but if you improve the prompt to self-context stuff you raise it to almost 70% (http://gptprompts.wikidot.com/linguistics:word-in-context http://gptprompts.wikidot.com/linguistics:word-in-context).
• wombatmobile 6y ago
  In AI parlance, is it correct to say that humans utilise a prompt that is a lifetime's worth of data?
• bra-ket 6y ago
  The article is of course right but also a bit silly. Language models like GPT-X are producing grammatically correct sentences, along the lines of "Colorless green ideas sleep furiously". The NLP research more or less solved the old syntax problem using 'distributional semantics' but 'semantics' is a misnomer, it's all about syntax. In fact the most useful part of the article for me is that they mentioned Douglas Summers-Stay, who does some interesting work on 'common sense' engineering, combining syntax engines like GPT-3 with knowledge graphs. https://sci-hub.tw/https://www.sciencedirect.com/science/article/abs/pii/S2212683X16300160 https://sci-hub.tw/https://www.sciencedirect.com/science/art... My bet is that actual AI will come from combination of these statistics-driven syntax generators with graphical causality models. Treating syntax as a kind of lower level substrate, akin to sensory modalities in vision, with intelligence model as a directed causal graph linking concepts at different levels of abstraction/chunking. As a side note it’s funny that people working on artificial intelligence at OpenAI and elsewhere are mostly computer scientists, not cognitive psychologists or neuroscientists who might actually have a clue how intelligence works. This probably explains the proliferation of ‘backpropagation’ as primary method of artificial learning. These people are just naturally good at calculus in high school, so it’s a hammer that found its proverbial nail.
  • Der_Einzige 6y ago
    Gradient free optimization is not used much in Neural Networks except in Reinforcement Learning. I think it's because backprop is objectively faster for most supervised problems than other techniques (e.g. simulated annealing or GAs)
    • bra-ket 6y ago
      The thing is, living beings don’t learn by brute-force trial and error like mathematical optimization models you mentioned. Besides enormous energy spent, an individual organism will just be eaten by predator on another iteration of its ‘error minimization’ loop. the idea of learning via reinforcement, that came from Skinner behaviorist experiments has been long discredited in cognitive psychology. (I highly recommend Wayne Wickelgren’ work on learning and memory if you’re interested, it’s brilliant and concise http://www.columbia.edu/~nvg1/Wickelgren/ http://www.columbia.edu/~nvg1/Wickelgren/ ) Biological plausibility might not be needed for recognizing check signatures or images of traffic lights, where backprop is working just fine, but I believe true cognition would require such energy expenditures that brute-force trial and error will never be feasible. Moreover such error correction imposes artificial constraints that limit the amount of information that can be learned, kind of like those mechanical calculators of the 17th century with gears and wheels and crude mechanical actuators.
• 6gvONxR4sf7o 6y ago
  I'm getting impatient with criticisms of ML models that are already covered in the papers introducing the models. OP is basically trying to get it to do what the GPT3 paper calls zero-shot inference. In the paper, it's pretty bad at zero shot inference across the board. And given what it does and how it was trained, that's unsurprising. And the point they're trying to make (that it can fail spectacularly) is also covered in the paper. It can do cool shit. It sucks at a lot of stuff. It's impressive and limited, but the hype train seems to only allow "it's nearly human level" or "it's awful." To everybody who is arguing about its capabilities without having read the paper yet, please read it. Then we can discuss stuff that hasn't already been covered more rigorously in the original paper. I don't know Davis, but I respect Marcus, and it seems like he's pushing back on the hype more than the actual model. Just not in a way that you couldn't glean from the paper itself (it almost always sucks on zero-shot), making it pretty disingenuous. Further, from the paper [0]: > it does little better than chance when evaluated one-shot or even few-shot on some “comparison” tasks, such as determining if two words are used the same way in a sentence, or if one sentence implies another (WIC and ANLI respectively), as well as on a subset of reading comprehension tasks. Maybe that's the curse of doing a thing that has broad implications. You can't fit the implications in a 10 page paper, so you write a 75 page paper. The blogosphere reads the first 10 pages (if even that), and because there's so much more to it that that introduction, they go on to argue about the rest of the implications without reading it. I'm sure Marcus and Davis have read it, but this criticism wouldn't be on the front page if the rest of everyone interested in this article had read the paper too. [0] Language Models are Few-Shot Learners https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165
  • mlb_hn 6y ago
    Also, better prompt design if you have make implicit meaning explicit can improve the WiC score (http://gptprompts.wikidot.com/linguistics:word-in-context http://gptprompts.wikidot.com/linguistics:word-in-context) and ANLI score (http://gptprompts.wikidot.com/linguistics:anli http://gptprompts.wikidot.com/linguistics:anli).
• deleted 6y ago
  [deleted]
• AndyPatterson 6y ago
  Its a certainty that given any prompt there will be likely one gnarly output, especially the longer that output becomes. What I'd like to see is an estimate of the uncertainty of the output.
• jeffrallen 6y ago
  OpenAI got confused and thought their job was to create an AI politician to compete with Trump.
• dnautics 6y ago
  the good news is that it should pass the turing test; most humans have no idea what they are talking about, too. Some are prone to similar bloviation, very likely using some techniques that are nontrivially similar to what GPT-3 is using.
• NautilusWave 6y ago
  Was anyone else tickled by the part where they say "Summers-Stay, who is good with metaphors, wrote to one of us, saying this:" and then they proceed to detail a simile? What irony, especially after that section on non-sequiturs!
• 29athrowaway 6y ago
  GPT-3 is a lot more intelligent than those guys that think they will get microchipped by Bill Gates.
  • speedgoose 6y ago
    I'm not sure GPT-3 is pretty stupid and learned with a dataset from internet that include a lot of stupid things.
• ctoth 6y ago
  I keep wanting to write a long explanation of just why this is so... silly? to read? But Gwern has already done the hard work. [0] The only other bit I'd like to mention is that GPT-3 uses exactly none of the new techniques that have been coming out in the last two years that would have significant impact on text generation. From working methods to apply GANs to text, to far more efficient transformer models that can handle longer sequences. For instance [1] [2] [3] for better direction, or [4] [5] [6] for efficiency. Or perhaps the outside view might help. After seeing GPT-2 last year, did you expect GPT-3 would work as well as it does after just naively scaling up the number of parameters with nothing else? [0https://www.gwern.net/newsletter/2020/05#gpt-3 https://www.gwern.net/newsletter/2020/05#gpt-3 [1 ] http://arxiv.org/abs/1905.09922 http://arxiv.org/abs/1905.09922 [2] https://github.com/anonymous1100/D_Improves_G_without_Updating_G https://github.com/anonymous1100/D_Improves_G_without_Updati... [3] http://arxiv.org/abs/2006.04643 http://arxiv.org/abs/2006.04643 [4] http://arxiv.org/abs/2007.14062 http://arxiv.org/abs/2007.14062 [5] http://arxiv.org/abs/2006.04768 http://arxiv.org/abs/2006.04768 [6] http://arxiv.org/abs/2002.05645 http://arxiv.org/abs/2002.05645
  • spacecity1971 6y ago
    Yes, this! The point being missed by most is the very real possibility that the Scaling Hypothesis is true. If it is, then we're seeing some kind of reasoning intelligence emerge. GPT-3 obviously isn't there yet. Unless it's faking it (Yudkowsky)...
• tyingq 6y ago
  I wonder how worried Google or Amazon is about someone using GPT-3 to flood them with garbage that they can't tell is garbage. (Book listings for Amazon, general web content for Google's scrapers)
• wintorez 6y ago
  That is true for most people on the internet. So, it should be fine.
• voces 6y ago
  Pretty meta, but I thought it was relevant here. We are familiar with Brandolini's law: > The amount of energy needed to refute bullshit is an order of magnitude bigger than to produce it. This can be illustrated with math or logic statements. To refute the program "1 + 1 = 3" you need to, at minimum, state "1 + 1 != 3", and such a program is always lengthier. A fuller refutation could be "1 + 1 != 3, 1 + 1 = 2", more than twice as long as the bullshit statement. What's happening here is sort of an inverse Brandolini's law: 35 world-class computer scientists use a massive amount of programming and compute to come up with a new language model trained on massive amounts of data. The trained weights don't even fit into memory. Impressive NLP progress. Then Gary Marcus comes around and states "Not AGI!". Not one of the computer scientists stated that they delivered AGI. But some tech journalists did. So OpenAI is guilty by association. Even though Altman came out to temper the hype and expectations. That's like proving the Poincaré conjecture, and someone dissing your research, because "1 + 1 != 3".
• turing_complete 6y ago
  "At the party, I poured myself a glass of lemonade, but it turned out to be too sour, so I added a little sugar. I didn’t see a spoon handy, so I stirred it with a cigarette. But that turned out to be a bad idea because it kept falling on the floor. That’s when he decided to start the Cremation Association of North America, which has become a major cremation provider with 145 locations." I mean that's just brilliant comedic writing.
  • zatel 6y ago
    +1 this could easily be the story line of a Rick and Morty episode or any of the other similar off the cuff shows that are popular right now. I think that will be one of the main profit streams for things like this, you can get the weird wild stories that don't really make sense but are interesting enough that who cares and you don't have to associate your network with eccentric individuals that attract malcontents.
• solinent 6y ago
  I mean, it's trained on the Internet. It kind of makes sense that the epistemological value of GPT-3's statements are essentially zero. It may even contradict itself in ways the Internet never could.
• karaterobot 6y ago
  Articles like this one mean to be a correction to a misperception that I don't see many people suffering from. Are there many people who both know what GPT-3 is, and believe it has achieved sapience?
  • jpindar 6y ago
    There are people who see a post about it on social media and appear to believe it has.
• YeGoblynQueenne 6y ago
  >> Within a single sentence, GPT-3 has lost track of the fact that Penny is advising Janet against getting a top because Jack already has a top. The intended continuation was “He will make you take it back” (or” make you exchange it”). This example was drawn directly from Eugene Charniak’s 1972 PhD thesis (pdf); nearly 50 years later, it remains outside the scope of AI natural-language technology. Aaaw! Eugene Charniak is one of my heroes of AI, after I read his little green book, Statistical Language Learning [1] during my Masters. It remains a great resource for a quick and dirty, but thorough and broad introduction to the field of statistical NLP that goes through all the basics. In fact, now that I think about it, if more people read that little book (it's only 199 pages) we would have many fewer discussions about how GPT-3 "understands" or "knows" etc. Anyway, thanks to Gary marcus for pointing out Charniak's thesis which I hadn't read. ____________ [1] https://mitpress.mit.edu/books/statistical-language-learning https://mitpress.mit.edu/books/statistical-language-learning
• scotty79 6y ago
  Great! So now you can elect it as your leader!
• dvh1990 6y ago
  No one claimed GPT3 is an AGI. Why does this article carry such a dismissive and disappointed tone? Perhaps the authors were offended that they were not given access and wanted to "expose" GPT3's flaws? We should celebrate GPT3 for the achievement that it is: A big step in a promising direction.