24 ms·

"These language models aren't actually smart, they're just good at parroting smart sounding things they've been taught" I still don't understand the purported

"These language models aren't actually smart, they're just good at parroting smart sounding things they've been taught" I still don't understand the purported difference between "sounds smart" and "is smart". How else can you verify an entity's intelligence other than its "output"? Every doctor, pilot, president has obtained that position of power and responsibility literally only by being tested on their outputs. We can't see inside a human's brain just as we can't see inside an AI's brain. What is the difference? How else do you propose we verify intelligence?

radicalbyte 4y ago
By the output. Those professionals will not make up a random answer.
Someone1234 4y ago
> I still don't understand the purported difference between "sounds smart" and "is smart". One understands the problem space and used logical reasoning to produce a solution, the other put the problem into Google, hit "I'm feeling lucky" and then parroted the first result right or wrong (or an aggregate of that). > How else can you verify an entity's intelligence other than its "output"? I'd argue that's actually a good test. I'd also point out that that is exactly where these solutions let themselves down, because they make [to us] obvious errors and don't understand that they did. That's why you know they are smart sounding rather than actually smart. > we can't see inside an AI's brain. We absolutely can see inside a deep learning model's structure.
- ramesh31 4y ago
  >the other put the problem into Google, hit "I'm feeling lucky" and then parroted the first result right or wrong (or an aggregate of that). Sounds a lot like most college essays I've ever read.
  - edgyquant 4y ago
    Yea most college essays are written by people in the act of learning. Their grades are the reinforcement process.
    - allturtles 4y ago
      So here you imply that most college students don't count as intelligent yet? These conversations always seem to get to a point where someone is willing to imply that most humans are not intelligent in order to defend the idea that LLMs are not intelligent.
      - anonymouskimmer 4y ago
        > "So here you imply that most college students don't count as intelligent yet?" A couple of things: 1) College students are in the act of learning, so what they are attempting to do is ideally just at their limit. 2) College students aren't intelligences focused on a single kind of output.
blueflow 4y ago
>I still don't understand the purported difference between "sounds smart" and "is smart". "sounds smart" is using the vocabulary in a specific way that gives some people the impression of smartness. "being smart" is more like transporting a complex idea to the brains of other people, your words and sentences need to align quite well for this to work out. I have not seen such a thing from AI yet.
aardvarkr 4y ago
“Intelligence” implies that actual thought is happening. These language models are amazing at token prediction but it’s a mistake to believe that they’re actually intelligent. Give it an inane prompt to better see the difference, like "write me an essay about camels and their value in cybersecurity" which gives us the answer "The camel is a valuable animal in the cybersecurity field due to its ability to store large amounts of data in its hump. Camels are able to cross vast distances with little water, which makes them ideal for carrying large amounts of data across networks. They are also very sturdy animals, able to withstand harsh conditions and even attacks from cyber criminals. In addition, their long eyelashes protect their eyes from sand and dust, making them perfect for working in dusty environments such as data centers."
- williamcotton 4y ago
  That's a pretty funny response to a pretty funny question!
  - renewiltord 4y ago
    I'm on your team, here, Dot. Rather playful response, like someone would write in response to a creative writing prompt. In fact, I think this is the correct way to answer that question.
    - CryptoBanker 4y ago
      Found the AI!
    - mannykannot 4y ago
      Because, until now, only humans could generate language like this, there is a strong temptation to anthropomorphize and attribute intent.
- jameshart 4y ago
  That looks a lot like ‘playfulness’, which would generally be considered a sign of intelligence if it were observed in, say, animals. You asked it to write an essay on a nonsensical premise so you got a nonsensical essay. If you just ask chatGPT ‘what is the value of camels in cybersecurity’ it doesn’t see that as a cue to play: > I'm not sure what you mean by "the value of camels in cybersecurity." Camels are a type of mammal that are commonly used for transportation and as a source of food and other resources in certain parts of the world. They do not have any direct connection to cybersecurity.
thereddaikon 4y ago
>I still don't understand the purported difference between "sounds smart" and "is smart". Its easy to see in practice with AI. Look at AI generated images and how they just cant get human hands right. Everyone knows a human should have 4 fingers and a thumb on each hand. But the AI does not understand that it is drawing a hand with those parameters. It is making something that sort of looks like other things it has been trained with. The same is true for text generation. A common description for Chat GPT I've seen catch on is calling it "confidently wrong". It will string together words that sound good but it doesn't actually have any understanding behind them. Its logic is in stringing words together. This is more akin to a good con man faking knowledge than an expert explaining a topic.
- rcme 4y ago
  Humans often draw bad human hands, as well. Don’t even try to ask a human to draw a bicycle.
  - jfk13 4y ago
    For those who haven't seen this little gem before: https://www.booooooom.com/2016/05/09/bicycles-built-based-on-peoples-attempts-to-draw-them-from-memory/ https://www.booooooom.com/2016/05/09/bicycles-built-based-on...
    - woolion 4y ago
      It's astonishing how recently I saw a thread on Mastodon prompting artists to draw a bicycle and it was not much better than that. What about building a visual library? Now there's a lot of AI results coming up and it's not performing much better in that regard either though. https://mstdn.social/tags/CrabCycle https://mstdn.social/tags/CrabCycle
  - bakugo 4y ago
    That's because hands and bycicles are complex 3D shapes and it's hard to picture them in your mind from arbitrary angles. It's not because the artist doesn't understand what a hand is or how many fingers it's supposed to have.
anileated 4y ago
Once it has human rights, free will, agency, etc. you can draw parallels with human brain and say it should be allowed to learn from open information like a human does. Until then it’s a copyright laundering tool. Microsoft et al. want you to think of it as if it was somehow similar to a human because it is convenient to them (“aw but look it’s learning!”), though of course in reality if it was even remotely equal to a human the whole controversy would be gone as they wouldn’t be able to do things like using it as they do without its informed consent and fair compensation, think how ludicrous this idea is and there’s your answer.
- williamcotton 4y ago
  > Until then it’s a copyright laundering tool. Something is either a copyright violation or not. You can't "launder" a creative work. It's either a copy or it isn't. It's either something that is a direct replacement for something else in the market or it is not. If I write a book and you buy the book, read the book, sell the book, write another book about the ideas in the book, you can legally sell the new book I wrote. (Seldon v Baker) Ideas are not covered by copyright. Ideas are fungible. Creative works are non-fungible. Money laundering is dependent on money being fungible so the source can be hidden. Hiding the source of a copyright is incoherent. A "copyright laundering tool" is incoherent.
  - anileated 4y ago
    > If I write a book and you buy the book, read the book, sell the book, write another book about the ideas in the book, you can legally sell the new book I wrote. Reread my comment. If you read my book and write your own it’s OK, if a megacorp provides a tool that ingests my book and automatically writes this book for you (and charges you, and does not pay or credit me) it’s not. Why is it not OK? In addition to generating a derivative work based on my book (not on ideas in it, literally on my book, because again this tool is not conscious and does not operate ideas, only the letters), which depending on the license can be illegal, it also kills incentives for people to publish books at all—because now OpenAI can buy one copy of any book, feed it to its ML tool, then charge people for using it and no one else ever needs to buy this book and give any credit to its author. Implications for open information sharing like blogging and such go without mentioning.
tablespoon 4y ago
> I still don't understand the purported difference between "sounds smart" and "is smart". Sounds smart: pasting words together with little to no sense about whether the result is insensible or incorrect. Smart: Actually having ideas, expressing them, and being able recognize errors with that process.
- allturtles 4y ago
  But the question remains how you tell the difference from the outside. How will we know when an AI model has ideas? How do you know your doctor has ideas?
  - tablespoon 4y ago
    > But the question remains how you tell the difference from the outside. How will we know when an AI model has ideas? How do you know your doctor has ideas? That's a problem for the future, not now. We're talking about something like ChatGPT, not some hypothetical strong AGI. And what we have now spits out error-ridden nonsense that "sounds smart," and fantastic sci-fi covers were the people accidentally have 14 fingers or two legs and three feet.
mannykannot 4y ago
How would you feel about taking what seems to be a serious medical condition (e.g. a serious pain with no obvious cause) to a person whose only ability was to answer questions on medical topics by repeating the content of medical text books? Being smart is not just about knowing things, it is about being able to use that knowledge. You don't have to look inside someone's head to learn how well they can do that.
anonymouskimmer 4y ago
> "I still don't understand the purported difference between "sounds smart" and "is smart"." Humans will say things such as "doohickey", "you know, this thing here", or "what if there was something that did this" when they know the concept but don't have the words. https://www.girlgeniusonline.com/comic.php?date=20121121 https://www.girlgeniusonline.com/comic.php?date=20121121 In the reverse, what's smart about the "turbo encabulator" lecture is the form, not the content. And the person giving the lecture absolutely knows this. ChatGPT can form a lecture, and it can form it with correct information. But until it can form it with concepts that it doesn't have words for I don't see how you can say it "is smart". See also https://en.wikipedia.org/wiki/Hyperlexia https://en.wikipedia.org/wiki/Hyperlexia
mandevil 4y ago
Harry Frankfurt (a philosopher) wrote a fascinating essay back in 2005 "On Bullshit" that defined Bullshit as different from a Lie. A lie knows its a lie, and knows that there is a truth out there, and that fundamentally, truth matters. Bullshit, on the other hand, does not care about the truth, is orthogonal to the truth. At present, GPT3 and the rest of the language models generate bullshit: they have no understanding or care of whether the words they put together are true or not. We will need to see if future work can overcome that problem.
lordfrito 4y ago
> I still don't understand the purported difference between "sounds smart" and "is smart". To me the difference comes down to the answer to the question: is the output useful or not? Let's say I ask AI questions I already know the answer to. I'll use an example from this thread: Are camels good at cybersecurity? If the AI answers no, I already knew that. I get no new useful information about camels out of this interaction. If the AI answers yes, I know it's wrong. But again, there's no new useful about camels there for me. (yes I now know the AI stinks but that's not my point, my point is I'm asking about camels) So asking AI questions you already know the answer to is pointless. The usefulness (and the goal) of AI is to ask it questions you don't know the answers to, to uncover new truths and insights. If I don't know the answer to a question, how am I supposed to trust that the AI got the answer right? Especially when AI is demonstrating over and over again that it gets the answers wrong. It's clear that AI can't reason, and its results are unreliable to anyone who doesn't know the answer already. So from a practical standpoint, these AIs can't tell you anything you don't already know. It can't be relied upon to teach/tell us new things. So it's an unreliable source of information and should not be relied upon to determine truths. Interestingly formatted, low quality information. All noise little signal. This to me is the difference between "sounds smart" and "is smart". It's about whether or not you can trusting the quality of the output to be accurate and correct. I've met plenty of humans that "sound smart". We call these people bullshitters. AI is at the well spoken BS stage. At best it can smooth talk uninformed people into believing falsehoods. In the age of misinformation, that makes it dangerous in my opinion. God help us if one of them ends up running for office, as it will out-BS all human politicians. It's Artificial Intelligence after all. If the AI output is unreliable, then the AI isn't intelligent.
Noneofya 4y ago
While you are wrong in the sense that we know it’s not intelligent because we built it and know how it works, you mention an important philosophical question: There is no way for you to know if anyone or anything else besides you is truly conscious.
- mannykannot 4y ago
  If one is consistently that radically skeptical, one cannot rationally assert anything predicated on there being anyone or anything else. To make the argument only for consciousness would be rather tendentious.
woolion 4y ago
Well, sometimes you interact with computer systems and you know how to make something work, but you have no idea why and how that works; meaning your ability to extrapolate from known cases would be limited compared to a true understanding of the principles. In that case I'm always thinking of Searle's Chinese room.
jai_ 4y ago
> I still don't understand the purported difference between "sounds smart" and "is smart". This really does encapsulate the average Hackernews commenter

radicalbyte 4y ago

By the output. Those professionals will not make up a random answer.

Someone1234 4y ago

> I still don't understand the purported difference between "sounds smart" and "is smart". One understands the problem space and used logical reasoning to produce a solution, the other put the problem into Google, hit "I'm feeling lucky" and then parroted the first result right or wrong (or an aggregate of that). > How else can you verify an entity's intelligence other than its "output"? I'd argue that's actually a good test. I'd also point out that that is exactly where these solutions let themselves down, because they make [to us] obvious errors and don't understand that they did. That's why you know they are smart sounding rather than actually smart. > we can't see inside an AI's brain. We absolutely can see inside a deep learning model's structure.

ramesh31 4y ago
>the other put the problem into Google, hit "I'm feeling lucky" and then parroted the first result right or wrong (or an aggregate of that). Sounds a lot like most college essays I've ever read.
- edgyquant 4y ago
  Yea most college essays are written by people in the act of learning. Their grades are the reinforcement process.
  - allturtles 4y ago
    So here you imply that most college students don't count as intelligent yet? These conversations always seem to get to a point where someone is willing to imply that most humans are not intelligent in order to defend the idea that LLMs are not intelligent.
    - anonymouskimmer 4y ago
      > "So here you imply that most college students don't count as intelligent yet?" A couple of things: 1) College students are in the act of learning, so what they are attempting to do is ideally just at their limit. 2) College students aren't intelligences focused on a single kind of output.

blueflow 4y ago

>I still don't understand the purported difference between "sounds smart" and "is smart". "sounds smart" is using the vocabulary in a specific way that gives some people the impression of smartness. "being smart" is more like transporting a complex idea to the brains of other people, your words and sentences need to align quite well for this to work out. I have not seen such a thing from AI yet.

aardvarkr 4y ago

“Intelligence” implies that actual thought is happening. These language models are amazing at token prediction but it’s a mistake to believe that they’re actually intelligent. Give it an inane prompt to better see the difference, like "write me an essay about camels and their value in cybersecurity" which gives us the answer "The camel is a valuable animal in the cybersecurity field due to its ability to store large amounts of data in its hump. Camels are able to cross vast distances with little water, which makes them ideal for carrying large amounts of data across networks. They are also very sturdy animals, able to withstand harsh conditions and even attacks from cyber criminals. In addition, their long eyelashes protect their eyes from sand and dust, making them perfect for working in dusty environments such as data centers."

williamcotton 4y ago
That's a pretty funny response to a pretty funny question!
- renewiltord 4y ago
  I'm on your team, here, Dot. Rather playful response, like someone would write in response to a creative writing prompt. In fact, I think this is the correct way to answer that question.
  - CryptoBanker 4y ago
    Found the AI!
  - mannykannot 4y ago
    Because, until now, only humans could generate language like this, there is a strong temptation to anthropomorphize and attribute intent.
jameshart 4y ago
That looks a lot like ‘playfulness’, which would generally be considered a sign of intelligence if it were observed in, say, animals. You asked it to write an essay on a nonsensical premise so you got a nonsensical essay. If you just ask chatGPT ‘what is the value of camels in cybersecurity’ it doesn’t see that as a cue to play: > I'm not sure what you mean by "the value of camels in cybersecurity." Camels are a type of mammal that are commonly used for transportation and as a source of food and other resources in certain parts of the world. They do not have any direct connection to cybersecurity.

thereddaikon 4y ago

>I still don't understand the purported difference between "sounds smart" and "is smart". Its easy to see in practice with AI. Look at AI generated images and how they just cant get human hands right. Everyone knows a human should have 4 fingers and a thumb on each hand. But the AI does not understand that it is drawing a hand with those parameters. It is making something that sort of looks like other things it has been trained with. The same is true for text generation. A common description for Chat GPT I've seen catch on is calling it "confidently wrong". It will string together words that sound good but it doesn't actually have any understanding behind them. Its logic is in stringing words together. This is more akin to a good con man faking knowledge than an expert explaining a topic.

rcme 4y ago
Humans often draw bad human hands, as well. Don’t even try to ask a human to draw a bicycle.
- jfk13 4y ago
  For those who haven't seen this little gem before: https://www.booooooom.com/2016/05/09/bicycles-built-based-on-peoples-attempts-to-draw-them-from-memory/ https://www.booooooom.com/2016/05/09/bicycles-built-based-on...
  - woolion 4y ago
    It's astonishing how recently I saw a thread on Mastodon prompting artists to draw a bicycle and it was not much better than that. What about building a visual library? Now there's a lot of AI results coming up and it's not performing much better in that regard either though. https://mstdn.social/tags/CrabCycle https://mstdn.social/tags/CrabCycle
- bakugo 4y ago
  That's because hands and bycicles are complex 3D shapes and it's hard to picture them in your mind from arbitrary angles. It's not because the artist doesn't understand what a hand is or how many fingers it's supposed to have.

anileated 4y ago

Once it has human rights, free will, agency, etc. you can draw parallels with human brain and say it should be allowed to learn from open information like a human does. Until then it’s a copyright laundering tool. Microsoft et al. want you to think of it as if it was somehow similar to a human because it is convenient to them (“aw but look it’s learning!”), though of course in reality if it was even remotely equal to a human the whole controversy would be gone as they wouldn’t be able to do things like using it as they do without its informed consent and fair compensation, think how ludicrous this idea is and there’s your answer.

williamcotton 4y ago
> Until then it’s a copyright laundering tool. Something is either a copyright violation or not. You can't "launder" a creative work. It's either a copy or it isn't. It's either something that is a direct replacement for something else in the market or it is not. If I write a book and you buy the book, read the book, sell the book, write another book about the ideas in the book, you can legally sell the new book I wrote. (Seldon v Baker) Ideas are not covered by copyright. Ideas are fungible. Creative works are non-fungible. Money laundering is dependent on money being fungible so the source can be hidden. Hiding the source of a copyright is incoherent. A "copyright laundering tool" is incoherent.
- anileated 4y ago
  > If I write a book and you buy the book, read the book, sell the book, write another book about the ideas in the book, you can legally sell the new book I wrote. Reread my comment. If you read my book and write your own it’s OK, if a megacorp provides a tool that ingests my book and automatically writes this book for you (and charges you, and does not pay or credit me) it’s not. Why is it not OK? In addition to generating a derivative work based on my book (not on ideas in it, literally on my book, because again this tool is not conscious and does not operate ideas, only the letters), which depending on the license can be illegal, it also kills incentives for people to publish books at all—because now OpenAI can buy one copy of any book, feed it to its ML tool, then charge people for using it and no one else ever needs to buy this book and give any credit to its author. Implications for open information sharing like blogging and such go without mentioning.

tablespoon 4y ago

> I still don't understand the purported difference between "sounds smart" and "is smart". Sounds smart: pasting words together with little to no sense about whether the result is insensible or incorrect. Smart: Actually having ideas, expressing them, and being able recognize errors with that process.

allturtles 4y ago
But the question remains how you tell the difference from the outside. How will we know when an AI model has ideas? How do you know your doctor has ideas?
- tablespoon 4y ago
  > But the question remains how you tell the difference from the outside. How will we know when an AI model has ideas? How do you know your doctor has ideas? That's a problem for the future, not now. We're talking about something like ChatGPT, not some hypothetical strong AGI. And what we have now spits out error-ridden nonsense that "sounds smart," and fantastic sci-fi covers were the people accidentally have 14 fingers or two legs and three feet.

mannykannot 4y ago

How would you feel about taking what seems to be a serious medical condition (e.g. a serious pain with no obvious cause) to a person whose only ability was to answer questions on medical topics by repeating the content of medical text books? Being smart is not just about knowing things, it is about being able to use that knowledge. You don't have to look inside someone's head to learn how well they can do that.

anonymouskimmer 4y ago

> "I still don't understand the purported difference between "sounds smart" and "is smart"." Humans will say things such as "doohickey", "you know, this thing here", or "what if there was something that did this" when they know the concept but don't have the words. https://www.girlgeniusonline.com/comic.php?date=20121121 https://www.girlgeniusonline.com/comic.php?date=20121121 In the reverse, what's smart about the "turbo encabulator" lecture is the form, not the content. And the person giving the lecture absolutely knows this. ChatGPT can form a lecture, and it can form it with correct information. But until it can form it with concepts that it doesn't have words for I don't see how you can say it "is smart". See also https://en.wikipedia.org/wiki/Hyperlexia https://en.wikipedia.org/wiki/Hyperlexia

mandevil 4y ago

Harry Frankfurt (a philosopher) wrote a fascinating essay back in 2005 "On Bullshit" that defined Bullshit as different from a Lie. A lie knows its a lie, and knows that there is a truth out there, and that fundamentally, truth matters. Bullshit, on the other hand, does not care about the truth, is orthogonal to the truth. At present, GPT3 and the rest of the language models generate bullshit: they have no understanding or care of whether the words they put together are true or not. We will need to see if future work can overcome that problem.

lordfrito 4y ago

> I still don't understand the purported difference between "sounds smart" and "is smart". To me the difference comes down to the answer to the question: is the output useful or not? Let's say I ask AI questions I already know the answer to. I'll use an example from this thread: Are camels good at cybersecurity? If the AI answers no, I already knew that. I get no new useful information about camels out of this interaction. If the AI answers yes, I know it's wrong. But again, there's no new useful about camels there for me. (yes I now know the AI stinks but that's not my point, my point is I'm asking about camels) So asking AI questions you already know the answer to is pointless. The usefulness (and the goal) of AI is to ask it questions you don't know the answers to, to uncover new truths and insights. If I don't know the answer to a question, how am I supposed to trust that the AI got the answer right? Especially when AI is demonstrating over and over again that it gets the answers wrong. It's clear that AI can't reason, and its results are unreliable to anyone who doesn't know the answer already. So from a practical standpoint, these AIs can't tell you anything you don't already know. It can't be relied upon to teach/tell us new things. So it's an unreliable source of information and should not be relied upon to determine truths. Interestingly formatted, low quality information. All noise little signal. This to me is the difference between "sounds smart" and "is smart". It's about whether or not you can trusting the quality of the output to be accurate and correct. I've met plenty of humans that "sound smart". We call these people bullshitters. AI is at the well spoken BS stage. At best it can smooth talk uninformed people into believing falsehoods. In the age of misinformation, that makes it dangerous in my opinion. God help us if one of them ends up running for office, as it will out-BS all human politicians. It's Artificial Intelligence after all. If the AI output is unreliable, then the AI isn't intelligent.

Noneofya 4y ago

While you are wrong in the sense that we know it’s not intelligent because we built it and know how it works, you mention an important philosophical question: There is no way for you to know if anyone or anything else besides you is truly conscious.

mannykannot 4y ago
If one is consistently that radically skeptical, one cannot rationally assert anything predicated on there being anyone or anything else. To make the argument only for consciousness would be rather tendentious.

woolion 4y ago

Well, sometimes you interact with computer systems and you know how to make something work, but you have no idea why and how that works; meaning your ability to extrapolate from known cases would be limited compared to a true understanding of the principles. In that case I'm always thinking of Searle's Chinese room.

jai_ 4y ago

> I still don't understand the purported difference between "sounds smart" and "is smart". This really does encapsulate the average Hackernews commenter