11 ms·
For anyone else who doesn’t want to deal with Facebook, here’s the post: Some people have completely unrealistic expectations about what large-scale language m
by czzr 6y ago
For anyone else who doesn’t want to deal with Facebook, here’s the post:
Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do.
This simple explanatory study by my friends at Nabla debunks some of those expectations for people who think massive language models can be used in healthcare.
GPT-3 is a language model, which means that you feed it a text and ask it to predict the continuation of the text, one word at a time. GPT-3 doesn't have any knowledge of how the world actually works. It only appears to have some level of background knowledge, to the extent that this knowledge is present in the statistics of text. But this knowledge is very shallow and disconnected from the underlying reality.
As a question-answering system, GPT-3 is not very good. Other approaches that are explicitly built to represent massive amount of knowledge in "neural" associative memories are better at it.
As a dialog system, it's not very good either. Again, other approaches that are explicitly trained to perform to interact with people are better at it.
It's entertaining, and perhaps mildly useful as a creative help. But trying to build intelligent machines by scaling up language models is like a high-altitude airplanes to go to the moon. You might beat altitude records, but going to the moon will require a completely different approach.
It's quite possible that some of the current approaches could be the basis of a good QA system for medical applicatioms. The system could be trained on the entire medical literature and answer questions from physicians.
But compiling massive amounts of operational knowledge from text is still very much a research topic.
- tomasyany 6y agoYou can find the study Yann was commenting here https://www.nabla.com/blog/gpt-3/ https://www.nabla.com/blog/gpt-3/
- oldmonk1990 6y agoIt's not a "study". It's some stupid clickbait stuff to get headlines and attention.
- formercoder 6y agoHigh altitude planes going to the moon is a beautiful analogy. I think this is what I’ll use to explain to less technical friends why I think we’re still many years from self driving cars.
- dbcurtis 6y agoI have been using that analogy to explain why Tesla is many years away from a self driving car. Several others are building something that is fundamentally different.
- fastball 6y agoAren't they rolling out a beta of FSD literally as we speak?
- jeffbee 6y agoNo.
- legolas2412 6y agoAnd that is why i dislike the term "self-driving". It makes people think "driverless", which "full self-driving" is not.
- fastball 6y agoIf it can get you from A to B without any human input, is the driver not a safety technicality at that point? What term would you prefer to describe a car that can get you from A to B without your input?
- edgyquant 6y agoI think full self driving will remain two years away for a decade at least.
- fastball 6y ago
- nickwalton00 6y agoI agree that GPT-3 should not be used for medical applications, but I disagree that it's "not very good" as a dialog system. I've found it to be insanely good though it may require effective prompt engineering to work well.
- PaulHoule 6y agoTo remind people: Yann LeCun is an engineering superstar who was working on neural networks at Bell labs long before they were cool. Those handwritten digits that are a scourge today (e.g. 'would any of these methods work on a different set of symbols?' is unasked) came from a competition to develop a commercial zip code reader for the U.S. Postal Service post back in the day of the Apple Newton. He won it! I was a bagman for text classification data in the early 2000's and his reviews of the results you got using methods of the time (Naive Bayes, Rocchio, Perceptron, SVM) showed a depth of thought and attention to detail which helped me pick and choose tools to make classifiers with fairly predictable performance and development paths. GPT-3 on the other hand does a good job of spouting nonsense like Peter Thiel and that has something to do with it's emotional appeal. People make fun of it and laugh at the mistakes it makes like those videos where somebody kicks down one of those Boston Robotics dogs: it's just good enough to be an object for those sort of feelings.
- thomas_rm 6y agoTo remind people: Yann LeCun worked on artificial neural networks (ANN) during the period where they were actively shunned by most of the scientific community. You could barely publish a paper on ANN. Just to demonstrate, one the most common books during period, "Artificial Intelligence: A Modern Approach, 2nd ed" by Norvig, 1080 pages, has less than one (1!) page dedicated to ANNs. I personally think Norvig is an idiot with regards to Artificial Intelligence, and his book (used in 1500 schools in 135 countries and regions) singlehandedly slowed down the progress of AI by a few years, until a new generation of students outgrew this archaic book.
- ibarea23 6y agoNorvig is not an idiot, and one of the humblest and nicest people I've ever met. A lot of people were wrong about ANNs.
- tephra 6y agoFor some context. The 2nd edition was published in 2002 (so maybe written in 2001-2002?). The fourth edition published in 2020 seems to have a bunch more things on NNs.
- deleted 6y ago[deleted]
- GrantZvolsky 6y ago> GPT-3 doesn't have any knowledge of how the world actually works. It only appears to have some level of background knowledge, to the extent that this knowledge is present in the statistics of text. But this knowledge is very shallow and disconnected from the underlying reality. Without excessive effort, humans don't have any knowledge of how the world actually works. They only appear to have some level of background knowledge, to the extent that this knowledge is present in their faint memories. But this knowledge is very shallow and disconnected from the underlying reality.
- disgruntledphd2 6y agoThis is just not true. For example, all humans have the notion of object permanence, developed by about six months of life. Object permanence is the notion that things don't go away just because you can't see them. ML systems need to be specifically trained to have object permanence, and GPT-3 almost certainly does not possess it. Like, I get that it's hip to booster ML and GPT-3, and all of the stuff humans can do seem trivial, but it's really not the case and is something that is holding progress in AI back massively.
- GrantZvolsky 6y agoObject permanence isn't even defined in the stateless world of the GPT-3 API. My comment was merely a jab at Yann's poor argument. I don't find humans to be trivial at all, but neither do I believe that they are infinitely complex. The linked Nabla article is fair, albeit I would appreciate more technical details. It seems to be using the API in zero-shot fashion, which is not what one would do to get the most out of it.
- disgruntledphd2 6y agoYeah, agreed. My concern is that object permanence isn't even defined in self driving models;)
- GrantZvolsky 6y ago
- corobo 6y agoAppreciated, thank you! (feel free to off-topic downvote this troops, just wanted to make it known it was appreciated. Anonymous upvotes can only convey so much gratitude)
- deleted 6y ago[deleted]
- mellosouls 6y agoAs both a GPT-3 sceptic and LeCun-sceptic (nothing personal, I'm just sceptical of a lot of the modern hype and seeming cliquey-ness of the AGI "progress" centres), this summary is very useful, but sounds a bit sour-grapesy from LeCun...