4 ms·

Before you believe the ChatGPT hype, ask it something you know well

23 points by SLHamlet 4y ago

philmcp 4y ago
The naesayers need to realise this is only GPT4 Imagine what GPT6 is going to look like
- reustle 4y ago
  Isn’t this 3.5?
seba_dos1 4y ago
(for the record, the editorialized title that will likely get changed is right now: "Before you believe the ChatGPT hype, ask it something you know well") Even better: ask it something you just made up. For GPT, there's no real difference and you shouldn't expect any.
- drpixie 4y ago
  ... except it will be difficult to judge the correctness of something you make up - better to ask it something you know, so you can evaluate the meaning, not just the wording.
  - seba_dos1 4y ago
    The idea of that exercise is to see how "confident" the AI is in giving you a correct-sounding answer for something it can not have any knowledge of because you just made it up - and to realize that it's doing the very same thing with any other topic unless its statistical models happen to nudge it into using memorized facts to increase the likelihood of sounding correct. I think it gives you more insight into what GPT actually is than just seeing it being wrong about something you're knowledgeable in (which shouldn't really be surprising).
    - twelve40 4y ago
      haha siri at least had the modesty to say "i don't know" sometimes
- bloaf 4y ago
  Kids are like that too. There was some study that found ~50% of kids thought french fries were made from animals. The correct conclusion is not that 50% of kids are dumb and 50% smart. Actually, none of the kids knew but they had a 50% chance of picking the right answer at random. Knowing what you don't know is an "advanced" skill many adults haven't even mastered. I'm not at all surprised chatGPT struggles to not-answer questions on things you just made up.
  - dorkwood 4y ago
    > The correct conclusion is not that 50% of kids are dumb and 50% smart. Actually, none of the kids knew but they had a 50% chance of picking the right answer at random. I can remember being a child and having this belief, but it wasn’t based on a guess. To me, it was logical. I thought that because fish and chips accompanied each other on the plate, they probably also accompanied each other in the sea. I can distinctly remember picturing fries swimming around with fish, in what I imagined to be their natural habitat.
alar44 4y ago
What an absurd take. The author asked it something extremely theoretical, with no actual objective correct answers, it gave reasonable answers, but the author disagrees, so it's no good? What?
ajkjk 4y ago
At the end of the day a chatbot is a chatbot: it's producing plausible-sounding output from the input. GPT is just better at it than other chatbots. It still doesn't _know_ anything, or _have opinions_. It is like a person who's BSing: quick, talk about a subject, and sound like you know what it is! It's a farce to consider it as something other than that, which is also why I don't personally care about all the news about it (and agree with the OP: the hype is silly and tedious). When ML models are trained on whether they give correct answers, rather than correct-sounding answers, I'll be interested in the bot they produce.
- quitit 4y ago
  Similar to Dall-e: It's output looks impressive and revolutionary if that's not your field of expertise. If it is, it looks closer to Clippy rather that HAL. Thus it's not the UnemploymentMaker5000™. There's a number of naive voices that believe it just needs a little this or that and it will abound forward in capability. That little thing they're asking is not actually little for AI. It just seems simple because humans develop the requisite concepts (such as compositionality) so young.
deleted 4y ago
[deleted]
aeturnum 4y ago
I'm not sure this is a good article, but I did have a similar sobering experience with ChatGPT. I asked it about a character I've been playing in a fighting game[1] and not only was its answer generally wrong - it seemed to have confused the character with some other character from another game. None of the move names describe things Baiken can do in Strive. I tried with Giovanna (another Strive character, but one who appears in no other games) and the answer was similarly extremely wrong. Pretty interesting! [1] https://twitter.com/stsDrex/status/1611904716023418884 https://twitter.com/stsDrex/status/1611904716023418884
bloaf 4y ago
I mean, pose that question to some random dev with little exposure to second life and you'd probably get an even more anemic/generic response. It's weird to me that people seem to expect current AI to be as good as an expert, especially at something as subtle as the intersection of society and technology. The hype around chatGPT is not that it's an expert in all things, it's that it's a novice in all things, and better than that at some.
karaterobot 4y ago
> The answer (see above) reminds me of that famous story about the amazing harpsichord-playing robot from the 19th century... The robot was the toast of Europe, giving command performances to royalty across the Continent. Until someone finally noticed the midget on a keyboard, hidden beneath the robot's dress. Wait, how is ChatGPT like a famous robot secretly operated by a human? Anyway, a lot of the recommendations it makes, he dismisses as being unfeasible or ineffective, but that doesn't mean ChatGPT is wrong. For one thing, it's based on a fallacious appeal to authority: I'm an expert, I disagree, so it's wrong. Or, in some cases: We tried a version of what ChatGPT suggests, and it didn't work, therefore it's bad advice. That's not necessarily the case, since there are plenty of reasons something might have failed in the past, like execution or timing. Anyway, I know ChatGPT should not be trusted (it makes no claim to trustworthiness), I just don't like this reasoning.
- SLHamlet 4y ago
  > how is ChatGPT like a famous robot secretly operated by a human For one thing it's being trained by a bunch of Kenyans behind the scenes: https://www.pcgamer.com/chatgpt-outsourcing-report-time/ https://www.pcgamer.com/chatgpt-outsourcing-report-time/
s1mon 4y ago
I have done this about mechanical engineering and manufacturing things that I know well. With some things, its answers are about as good as a decent overview article that you might find with google after sorting through a few spammy ones. With other things, it can make remarkably dumb mistakes, but as has been pointed out here (and elsewhere), it's possible to give it some corrections and ask it to do further reasoning. For instance, I asked it to list common defects in injection molding. It did a pretty good job of listing and describing the things that I'm usually looking out for. On the other hand, I asked it about how much a solid tire is typically stretched when placed on a rim (e.g. for an electric scooter). It came up with a typical percentage range, and then I tried to get it to reason out how the cross-sectional dimensions would change based on that stretching. It made several really basic errors that I managed to get it to correct, but in the end, I didn't really trust its ability and or "knowledge" in this area. I have found it to be pretty relaxing to use as a recipe source, since it will just give you a recipe and the basic instructions without the fake history and 17 ads. It's pretty good at taking some vague requests about having certain ingredients and wanting something in a particular direction, and giving a reasonable result. Maybe if I was an expert on cooking particular types of food, I would find its answers comical or pedestrian.
- twelve40 4y ago
  soo, this is the question that keeps bugging me > basic instructions without the fake history and 17 ads if this were ever to be truly commercialized as a mainstream source of information, do you think that whoever runs this will forego the trillion-dollar opportunity to still show you 17 ads? purely out of goodness of their heart? In addition to that, at least a tiny fraction of those 17 ads' revenue went to the people who actually create recipes. In the brave new world, why would anyone bother putting out information if this chat middle-thing charges its users but doesn't ever pay to creators?
joak 4y ago
ChatGPT is merely a statistical aggregate of what people says. So when you ask a question you get the common misconceptions and platitudes about the topic. Systems like wikipedia and stackoverflow have (imperfect) social mechanisms to filter out useless or false information. ChatGPT have none of that. No miracle.
- SLHamlet 4y ago
  > ChatGPT is merely a statistical aggregate of what people says. That's the crazy part. This is pretty much how Google has worked for like 30 fucking years -- links to the relevant content that has the most links! But people are all, "OMG ChatGPT is an amazing breakthrough that's going to argue in front of the Supreme Court!"
blastonico 4y ago
But I don't need chatgpt to teach me what I know. I need it to teach new things easier than by reading books or digging into the endless amount of pages from Google. Those are things that I don't need to have a deep and accurate understanding. Usually, they are topics that I barely know how to start searching about them. So, for taking me from zero to enough so fast and intuitively, Chatgpt is really impressive to me.
captn3m0 4y ago
My favorite was asking it "how can I solve a rubik's cube blindfolded". That's something I've been learning on and off for some time, and there's enough tutorials on the internet on how to do it. ChatGPT's answer (summarized)[1]: "Memorize the initial state, and track the complete state of the cube in your head as you solve a cube normally". If you aren't a cuber, the answer sounds plausible enough (It does work for chess after all), but it's humanly impossible with a Rubik's Cube. The only way you can memorize a random state of the cube _and_ operate on it, is to keep your changes to a minimal so your memorized state gets incrementally solved - you solve a few pieces at a time. [1]: https://twitter.com/captn3m0/status/1598301110955311107 https://twitter.com/captn3m0/status/1598301110955311107
MilStdJunkie 4y ago
If you know a lot about something that is only one degree removed from a Wikipedia stub article, ChatGPT gets really silly really fast. I tried to have a conversation about S1000D governance, and it was making up whole new bodies of NATO governance in about two replies.

23 points by SLHamlet 4y ago

philmcp 4y ago

The naesayers need to realise this is only GPT4 Imagine what GPT6 is going to look like

reustle 4y ago
Isn’t this 3.5?

seba_dos1 4y ago

(for the record, the editorialized title that will likely get changed is right now: "Before you believe the ChatGPT hype, ask it something you know well") Even better: ask it something you just made up. For GPT, there's no real difference and you shouldn't expect any.

drpixie 4y ago
... except it will be difficult to judge the correctness of something you make up - better to ask it something you know, so you can evaluate the meaning, not just the wording.
- seba_dos1 4y ago
  The idea of that exercise is to see how "confident" the AI is in giving you a correct-sounding answer for something it can not have any knowledge of because you just made it up - and to realize that it's doing the very same thing with any other topic unless its statistical models happen to nudge it into using memorized facts to increase the likelihood of sounding correct. I think it gives you more insight into what GPT actually is than just seeing it being wrong about something you're knowledgeable in (which shouldn't really be surprising).
  - twelve40 4y ago
    haha siri at least had the modesty to say "i don't know" sometimes
bloaf 4y ago
Kids are like that too. There was some study that found ~50% of kids thought french fries were made from animals. The correct conclusion is not that 50% of kids are dumb and 50% smart. Actually, none of the kids knew but they had a 50% chance of picking the right answer at random. Knowing what you don't know is an "advanced" skill many adults haven't even mastered. I'm not at all surprised chatGPT struggles to not-answer questions on things you just made up.
- dorkwood 4y ago
  > The correct conclusion is not that 50% of kids are dumb and 50% smart. Actually, none of the kids knew but they had a 50% chance of picking the right answer at random. I can remember being a child and having this belief, but it wasn’t based on a guess. To me, it was logical. I thought that because fish and chips accompanied each other on the plate, they probably also accompanied each other in the sea. I can distinctly remember picturing fries swimming around with fish, in what I imagined to be their natural habitat.

alar44 4y ago

What an absurd take. The author asked it something extremely theoretical, with no actual objective correct answers, it gave reasonable answers, but the author disagrees, so it's no good? What?

ajkjk 4y ago

At the end of the day a chatbot is a chatbot: it's producing plausible-sounding output from the input. GPT is just better at it than other chatbots. It still doesn't _know_ anything, or _have opinions_. It is like a person who's BSing: quick, talk about a subject, and sound like you know what it is! It's a farce to consider it as something other than that, which is also why I don't personally care about all the news about it (and agree with the OP: the hype is silly and tedious). When ML models are trained on whether they give correct answers, rather than correct-sounding answers, I'll be interested in the bot they produce.

quitit 4y ago
Similar to Dall-e: It's output looks impressive and revolutionary if that's not your field of expertise. If it is, it looks closer to Clippy rather that HAL. Thus it's not the UnemploymentMaker5000™. There's a number of naive voices that believe it just needs a little this or that and it will abound forward in capability. That little thing they're asking is not actually little for AI. It just seems simple because humans develop the requisite concepts (such as compositionality) so young.

deleted 4y ago

[deleted]

aeturnum 4y ago

I'm not sure this is a good article, but I did have a similar sobering experience with ChatGPT. I asked it about a character I've been playing in a fighting game[1] and not only was its answer generally wrong - it seemed to have confused the character with some other character from another game. None of the move names describe things Baiken can do in Strive. I tried with Giovanna (another Strive character, but one who appears in no other games) and the answer was similarly extremely wrong. Pretty interesting! [1] https://twitter.com/stsDrex/status/1611904716023418884 https://twitter.com/stsDrex/status/1611904716023418884

bloaf 4y ago

I mean, pose that question to some random dev with little exposure to second life and you'd probably get an even more anemic/generic response. It's weird to me that people seem to expect current AI to be as good as an expert, especially at something as subtle as the intersection of society and technology. The hype around chatGPT is not that it's an expert in all things, it's that it's a novice in all things, and better than that at some.

karaterobot 4y ago

> The answer (see above) reminds me of that famous story about the amazing harpsichord-playing robot from the 19th century... The robot was the toast of Europe, giving command performances to royalty across the Continent. Until someone finally noticed the midget on a keyboard, hidden beneath the robot's dress. Wait, how is ChatGPT like a famous robot secretly operated by a human? Anyway, a lot of the recommendations it makes, he dismisses as being unfeasible or ineffective, but that doesn't mean ChatGPT is wrong. For one thing, it's based on a fallacious appeal to authority: I'm an expert, I disagree, so it's wrong. Or, in some cases: We tried a version of what ChatGPT suggests, and it didn't work, therefore it's bad advice. That's not necessarily the case, since there are plenty of reasons something might have failed in the past, like execution or timing. Anyway, I know ChatGPT should not be trusted (it makes no claim to trustworthiness), I just don't like this reasoning.

SLHamlet 4y ago
> how is ChatGPT like a famous robot secretly operated by a human For one thing it's being trained by a bunch of Kenyans behind the scenes: https://www.pcgamer.com/chatgpt-outsourcing-report-time/ https://www.pcgamer.com/chatgpt-outsourcing-report-time/

s1mon 4y ago

I have done this about mechanical engineering and manufacturing things that I know well. With some things, its answers are about as good as a decent overview article that you might find with google after sorting through a few spammy ones. With other things, it can make remarkably dumb mistakes, but as has been pointed out here (and elsewhere), it's possible to give it some corrections and ask it to do further reasoning. For instance, I asked it to list common defects in injection molding. It did a pretty good job of listing and describing the things that I'm usually looking out for. On the other hand, I asked it about how much a solid tire is typically stretched when placed on a rim (e.g. for an electric scooter). It came up with a typical percentage range, and then I tried to get it to reason out how the cross-sectional dimensions would change based on that stretching. It made several really basic errors that I managed to get it to correct, but in the end, I didn't really trust its ability and or "knowledge" in this area. I have found it to be pretty relaxing to use as a recipe source, since it will just give you a recipe and the basic instructions without the fake history and 17 ads. It's pretty good at taking some vague requests about having certain ingredients and wanting something in a particular direction, and giving a reasonable result. Maybe if I was an expert on cooking particular types of food, I would find its answers comical or pedestrian.

twelve40 4y ago
soo, this is the question that keeps bugging me > basic instructions without the fake history and 17 ads if this were ever to be truly commercialized as a mainstream source of information, do you think that whoever runs this will forego the trillion-dollar opportunity to still show you 17 ads? purely out of goodness of their heart? In addition to that, at least a tiny fraction of those 17 ads' revenue went to the people who actually create recipes. In the brave new world, why would anyone bother putting out information if this chat middle-thing charges its users but doesn't ever pay to creators?

joak 4y ago

ChatGPT is merely a statistical aggregate of what people says. So when you ask a question you get the common misconceptions and platitudes about the topic. Systems like wikipedia and stackoverflow have (imperfect) social mechanisms to filter out useless or false information. ChatGPT have none of that. No miracle.

SLHamlet 4y ago
> ChatGPT is merely a statistical aggregate of what people says. That's the crazy part. This is pretty much how Google has worked for like 30 fucking years -- links to the relevant content that has the most links! But people are all, "OMG ChatGPT is an amazing breakthrough that's going to argue in front of the Supreme Court!"

blastonico 4y ago

But I don't need chatgpt to teach me what I know. I need it to teach new things easier than by reading books or digging into the endless amount of pages from Google. Those are things that I don't need to have a deep and accurate understanding. Usually, they are topics that I barely know how to start searching about them. So, for taking me from zero to enough so fast and intuitively, Chatgpt is really impressive to me.

captn3m0 4y ago

My favorite was asking it "how can I solve a rubik's cube blindfolded". That's something I've been learning on and off for some time, and there's enough tutorials on the internet on how to do it. ChatGPT's answer (summarized)[1]: "Memorize the initial state, and track the complete state of the cube in your head as you solve a cube normally". If you aren't a cuber, the answer sounds plausible enough (It does work for chess after all), but it's humanly impossible with a Rubik's Cube. The only way you can memorize a random state of the cube _and_ operate on it, is to keep your changes to a minimal so your memorized state gets incrementally solved - you solve a few pieces at a time. [1]: https://twitter.com/captn3m0/status/1598301110955311107 https://twitter.com/captn3m0/status/1598301110955311107

MilStdJunkie 4y ago

If you know a lot about something that is only one degree removed from a Wikipedia stub article, ChatGPT gets really silly really fast. I tried to have a conversation about S1000D governance, and it was making up whole new bodies of NATO governance in about two replies.