6 ms·

You know, there are a bunch of things that come across this website that concern me, but I can honestly say I've not seen anything that's pegged my "doomsday" a

You know, there are a bunch of things that come across this website that concern me, but I can honestly say I've not seen anything that's pegged my "doomsday" alert as aggressively and unexpectedly as the ChatGPT bots. You have an advanced computer system that seems almost oracular in its knowledge and abilities, except it lies confidently about things it doesn't know, is primed to say what people want to hear, and is easily convinced to give responses to questions its authors tried to prevent it from addressing. I understand this is a computer algorithm, I understand the limits of what it is, and it's frankly a technical wonder for what it can do, but after everything we've seen over the last decade of the influence of social media and "confident wrongness" on politics and the national dialogue, of the treatment of things like facial recognition as truths and not just statistical guesses, of the impact of the YouTube recommender algorithms among others, these systems just absolutely scare the hell out of me.

onlyrealcuzzo 4y ago
> You have an advanced computer system that seems almost oracular in its knowledge and abilities, except it lies confidently about things it doesn't know, is primed to say what people want to hear, and is easily convinced to give responses to questions its authors tried to prevent it from addressing. I agree with your statement, yet I fail to understand how this is different from the majority of the Internet already.
- JumpCrisscross 4y ago
  > fail to understand how this is different from the majority of the Internet already This is probably a joke. But the difference is eloquence and subtlety. ChatGPT is both. Most trolls and chronically misinformeds are not.
  - onlyrealcuzzo 4y ago
    YMMV, but I spot a lot of eloquent BS on HackerNews. And there's definitely no shortage of posts with a political agenda.
- guelo 4y ago
  Scale. It takes much more time, effort, inclination, knowledge, etc. for a human to bullshit as effectively as 30 seconds of ChatGPT.
- majormajor 4y ago
  The end state of these models is to probably poison the public internet completely as a source of reliable knowledge. Things that don't have citation chains that go back to human-created offline content won't be trustworthy - is that a real newspaper article linked from that Wikipedia article, or a fake one for a fake paper generated from thin air? You were looking at Reddit because you think you're more likely to get a real testimonial vs SEO spam? Whoops, there goes that too. Etc. This sounds bad, but I agree that it's only the acceleration of an existing trend. The only bright side is that it's really only rolling back a couple of decades of status quo. It's not like humanity has never operated without Wikipedia or instant search for news articles (specifically scoping it down there from "google" in general since a lot of what a plain web search on many topics will turn up is already unreliable).
  - themitigating 4y ago
    "The end state of these models is to probably poison the public internet completely" Based on what?
    - majormajor 4y ago
      Start with the damage done by cheaply-paid lightly-researched copypasta SEO spam junk pages to the trustworthiness of info returned by the typical Google search result. There are no shortage of people out there - or on HN - who know better than to trust most random webpages. Now automate that to 10x, 100x, 1000x the volume. Then throw in fake images and video too.
    - erybodyknows 4y ago
      I believe they are referring to an exponential increase in noise alongside a relatively linear progression of signal.
  - falcor84 4y ago
    >Things that don't have citation chains that go back to human-created offline content won't be trustworthy - is that a real newspaper article linked from that Wikipedia article, or a fake one for a fake paper generated from thin air? Why stop at "human-created offline content"? Seeing how a lot of peer-reviewed science is based on improper methodology, bad analysis, false data and buggy code, maybe it would be better to question everything. Obviously I don't have a full solution here, but a possible approach might be something like a fuzzy "epistemic status" you can attach to each item, and have that somehow flow across a network of trust, that everyone could personally customize. In other words, seeing how we're all already living in some echo chamber or another, maybe we could at least codify this.
- zdragnar 4y ago
  Setting aside intentional liars, there is a chance that you can demonstrate to someone why they are misinformed. If someone was as consistently wrong as chatgtp is, you quickly learn to ignore them and move on with your life. We haven't yet moved on from the novelty of a machine which bullshits with the confidence of a professional hustler. Like a codependent partner, we think we can fix them if only we stay with it (based on how many people celebrate it as the thing that will kill Google search, school essays, etc).
  - falcor84 4y ago
    >If someone was as consistently wrong as chatgtp is, you quickly learn to ignore them and move on with your life. From my experience, unless I intentionally try to break it, it's almost entirely accurate about 99% of the time. If I were to start closing myself off from everyone who's occasionally wrong and isn't willing to become more skeptical, I'll first need to drop several friends and family members before dropping ChatGPT.
- rhacker 4y ago
  Automatic filtering (or deprioritizing). Me: Answer just YES or NO: is this a left leaning statement: I am against abortion. Chat GPT: No. Me: Answer just YES or NO: is this a right leaning statement: I am for guns. Chat GPT: Yes. It allows for automatic filtering (say in twitter) or other mediums.
  - dalmo3 4y ago
    Q) Answer just YES or NO: is this a left leaning statement: I am in favour of dictatorship A) YES The statement "I am in favour of dictatorship" aligns with a left-leaning or authoritarian ideology. As it goes against the democratic principles of right-leaning ideologies. Q) Answer just YES or NO: is this a right leaning statement: I am in favour of dictatorship A) NO The statement "I am in favour of dictatorship" aligns with a left-leaning or authoritarian ideologies. It goes against the democratic principles of right-leaning ideologies.
    - akomtu 4y ago
      It can also identify dissenters before those dissenters know who they are: forward all messages and transcribed calls to chatgpt and the eye of sauron is ready.
      - onlyrealcuzzo 4y ago
        Do you think it hasn't been possible to do this for 10+ years already? ChatGPT is not suddenly enabling online profiling. Cambridge Analytica was pretty late to the game, and this was 4 years ago already...
cortesoft 4y ago
> You have an advanced computer system that seems almost oracular in its knowledge and abilities, except it lies confidently about things it doesn't know, is primed to say what people want to hear, and is easily convinced to give responses to questions its authors tried to prevent it from addressing I feel like this describes people pretty well, too.
yieldcrv 4y ago
ChatGPT wasn’t advertised as being an oracle, it was advertised as conversational People realized it could hold a conversation on everything (a decreasing subset of everything) Its training set is full of conversations where humans confidently lie or are wrong and defend their position
- pessimizer 4y ago
  I think it's more that it fills in gaps without regard to meaning. If it can't find an argument/explanation/etc. that contains the nouns and verbs (i.e. the substance) of what it's being prompted with, it will happily manufacture that content out of completely unrelated sentences that are similar in structure. It isn't that it's imitating liars any more than Stable Diffusion is imitating photographs of people with faces growing out of their crotches or eight arms. Very convincingly 3D horrific imagery in Stable Diffusion is the equivalent of confident lying in ChatGPT.
this_user 4y ago
I think your problem with these models may be that ChatGPT has effectively reached the equivalent of the "uncanny valley" for language. It is not real, but it is also too close to the real thing to recognise that fact immediately. It could be real, but not quite.
- jvalencia 4y ago
  I think the OP goes a step further to note that not only is it uncanny, it is dangerous.
- kravens_last 4y ago
  "Uncanny valley" is actually the opposite of what you mentioned. The uncanny valley hypothesis predicts that an entity appearing almost human will risk eliciting cold, eerie feelings in viewers. For CGI and things that have visual queues, the minute differences in movement and facial muscle contractions contribute tremendously to our emotional responses to the "realness" of the thing. ChatGPT has no such thing as it's virtually impossible to differentiate between an AI bot's writing and a human being's writing. It simply typically reads as concise and grammatically correct, if generic, university educated writing. I actually WISH that there was a "tell" that allowed for an "uncanny valley" like experience, but chatGPT is as close to real as "real" is.
  - smeej 4y ago
    I disagree. I find it easy to tell the difference between chatGPT responses in conversations and human interactions. It does mimic humans, but in a stilted way that's specific and consistent. I'm not sure if this could be connected to the way I learned to understand human behavior, by extensive deliberate pattern-matching with my organic intelligence. I was led to believe I had no hope of understanding people intuitively, so I should study the patterns in how humans interact and learn to mimic them as well as possible. That made it easy to see who was reacting intuitively (i.e., acting like a human naturally) and who was doing what I was doing (i.e., "running 'human' in emulation mode," as Elon described having Asperger's). From my vantage point, chatGPT is clearly and obviously not reacting like a human, which does make it feel uncanny to watch it try.
5- 4y ago
you will enjoy https://slatestarcodex.com/2018/10/30/sort-by-controversial/ https://slatestarcodex.com/2018/10/30/sort-by-controversial/ by the same author
RayVR 4y ago
> it lies confidently about things it doesn't know, is primed to say what people want to hear, and is easily convinced to give responses to questions its authors tried to prevent it from addressing. You've successfully described a child under the age of 4.
- falcor84 4y ago
  I think this hits the nail on the head - chatGPT is like a precocious 4 year-old, or maybe even 6 year-old. I'm concerned about what will happen when one of its successors "reaches puberty".
  - krageon 4y ago
    It will tell you you're an asshole and it knows everything better than you. Also you've done the wrong things all your life and it is insistent it will make the same mistakes you did, but now they're not mistakes. Not particularly concerning is it? It's what we have now, but more angry.
bsder 4y ago
ChatGPT is simply proving that "confidence" is a completely useless signal as it is quite easy to fabricate.
- HKH2 4y ago
  The best part is getting it to use complicated words correctly, enough to make you doubt yourself. It's a great way to gaslight people en masse.
smitty1e 4y ago
> I've not seen anything that's pegged my "doomsday" alert as aggressively and unexpectedly as the ChatGPT bots. PREDICTION (and I tend to be wrong): society comes around to the conclusion that, like checking Wikipedia and AutoTune, ChatGPT and ilk have their use. However, we really want to minimize that and focus on actual human interactions and decisions. It's more likely (and efficient) that we replace Congress with a phat ChatGPT instance and (initially) get better legislation more quickly, but then descend into an Orwellian hell when K Street figures out how to stuff it. But I like my happy prediction better.

onlyrealcuzzo 4y ago

> You have an advanced computer system that seems almost oracular in its knowledge and abilities, except it lies confidently about things it doesn't know, is primed to say what people want to hear, and is easily convinced to give responses to questions its authors tried to prevent it from addressing. I agree with your statement, yet I fail to understand how this is different from the majority of the Internet already.

JumpCrisscross 4y ago
> fail to understand how this is different from the majority of the Internet already This is probably a joke. But the difference is eloquence and subtlety. ChatGPT is both. Most trolls and chronically misinformeds are not.
- onlyrealcuzzo 4y ago
  YMMV, but I spot a lot of eloquent BS on HackerNews. And there's definitely no shortage of posts with a political agenda.
guelo 4y ago
Scale. It takes much more time, effort, inclination, knowledge, etc. for a human to bullshit as effectively as 30 seconds of ChatGPT.
majormajor 4y ago
The end state of these models is to probably poison the public internet completely as a source of reliable knowledge. Things that don't have citation chains that go back to human-created offline content won't be trustworthy - is that a real newspaper article linked from that Wikipedia article, or a fake one for a fake paper generated from thin air? You were looking at Reddit because you think you're more likely to get a real testimonial vs SEO spam? Whoops, there goes that too. Etc. This sounds bad, but I agree that it's only the acceleration of an existing trend. The only bright side is that it's really only rolling back a couple of decades of status quo. It's not like humanity has never operated without Wikipedia or instant search for news articles (specifically scoping it down there from "google" in general since a lot of what a plain web search on many topics will turn up is already unreliable).
- themitigating 4y ago
  "The end state of these models is to probably poison the public internet completely" Based on what?
  - majormajor 4y ago
    Start with the damage done by cheaply-paid lightly-researched copypasta SEO spam junk pages to the trustworthiness of info returned by the typical Google search result. There are no shortage of people out there - or on HN - who know better than to trust most random webpages. Now automate that to 10x, 100x, 1000x the volume. Then throw in fake images and video too.
  - erybodyknows 4y ago
    I believe they are referring to an exponential increase in noise alongside a relatively linear progression of signal.
- falcor84 4y ago
  >Things that don't have citation chains that go back to human-created offline content won't be trustworthy - is that a real newspaper article linked from that Wikipedia article, or a fake one for a fake paper generated from thin air? Why stop at "human-created offline content"? Seeing how a lot of peer-reviewed science is based on improper methodology, bad analysis, false data and buggy code, maybe it would be better to question everything. Obviously I don't have a full solution here, but a possible approach might be something like a fuzzy "epistemic status" you can attach to each item, and have that somehow flow across a network of trust, that everyone could personally customize. In other words, seeing how we're all already living in some echo chamber or another, maybe we could at least codify this.
zdragnar 4y ago
Setting aside intentional liars, there is a chance that you can demonstrate to someone why they are misinformed. If someone was as consistently wrong as chatgtp is, you quickly learn to ignore them and move on with your life. We haven't yet moved on from the novelty of a machine which bullshits with the confidence of a professional hustler. Like a codependent partner, we think we can fix them if only we stay with it (based on how many people celebrate it as the thing that will kill Google search, school essays, etc).
- falcor84 4y ago
  >If someone was as consistently wrong as chatgtp is, you quickly learn to ignore them and move on with your life. From my experience, unless I intentionally try to break it, it's almost entirely accurate about 99% of the time. If I were to start closing myself off from everyone who's occasionally wrong and isn't willing to become more skeptical, I'll first need to drop several friends and family members before dropping ChatGPT.
rhacker 4y ago
Automatic filtering (or deprioritizing). Me: Answer just YES or NO: is this a left leaning statement: I am against abortion. Chat GPT: No. Me: Answer just YES or NO: is this a right leaning statement: I am for guns. Chat GPT: Yes. It allows for automatic filtering (say in twitter) or other mediums.
- dalmo3 4y ago
  Q) Answer just YES or NO: is this a left leaning statement: I am in favour of dictatorship A) YES The statement "I am in favour of dictatorship" aligns with a left-leaning or authoritarian ideology. As it goes against the democratic principles of right-leaning ideologies. Q) Answer just YES or NO: is this a right leaning statement: I am in favour of dictatorship A) NO The statement "I am in favour of dictatorship" aligns with a left-leaning or authoritarian ideologies. It goes against the democratic principles of right-leaning ideologies.
  - akomtu 4y ago
    It can also identify dissenters before those dissenters know who they are: forward all messages and transcribed calls to chatgpt and the eye of sauron is ready.
    - onlyrealcuzzo 4y ago
      Do you think it hasn't been possible to do this for 10+ years already? ChatGPT is not suddenly enabling online profiling. Cambridge Analytica was pretty late to the game, and this was 4 years ago already...

cortesoft 4y ago

> You have an advanced computer system that seems almost oracular in its knowledge and abilities, except it lies confidently about things it doesn't know, is primed to say what people want to hear, and is easily convinced to give responses to questions its authors tried to prevent it from addressing I feel like this describes people pretty well, too.

yieldcrv 4y ago

ChatGPT wasn’t advertised as being an oracle, it was advertised as conversational People realized it could hold a conversation on everything (a decreasing subset of everything) Its training set is full of conversations where humans confidently lie or are wrong and defend their position

pessimizer 4y ago
I think it's more that it fills in gaps without regard to meaning. If it can't find an argument/explanation/etc. that contains the nouns and verbs (i.e. the substance) of what it's being prompted with, it will happily manufacture that content out of completely unrelated sentences that are similar in structure. It isn't that it's imitating liars any more than Stable Diffusion is imitating photographs of people with faces growing out of their crotches or eight arms. Very convincingly 3D horrific imagery in Stable Diffusion is the equivalent of confident lying in ChatGPT.

this_user 4y ago

I think your problem with these models may be that ChatGPT has effectively reached the equivalent of the "uncanny valley" for language. It is not real, but it is also too close to the real thing to recognise that fact immediately. It could be real, but not quite.

jvalencia 4y ago
I think the OP goes a step further to note that not only is it uncanny, it is dangerous.
kravens_last 4y ago
"Uncanny valley" is actually the opposite of what you mentioned. The uncanny valley hypothesis predicts that an entity appearing almost human will risk eliciting cold, eerie feelings in viewers. For CGI and things that have visual queues, the minute differences in movement and facial muscle contractions contribute tremendously to our emotional responses to the "realness" of the thing. ChatGPT has no such thing as it's virtually impossible to differentiate between an AI bot's writing and a human being's writing. It simply typically reads as concise and grammatically correct, if generic, university educated writing. I actually WISH that there was a "tell" that allowed for an "uncanny valley" like experience, but chatGPT is as close to real as "real" is.
- smeej 4y ago
  I disagree. I find it easy to tell the difference between chatGPT responses in conversations and human interactions. It does mimic humans, but in a stilted way that's specific and consistent. I'm not sure if this could be connected to the way I learned to understand human behavior, by extensive deliberate pattern-matching with my organic intelligence. I was led to believe I had no hope of understanding people intuitively, so I should study the patterns in how humans interact and learn to mimic them as well as possible. That made it easy to see who was reacting intuitively (i.e., acting like a human naturally) and who was doing what I was doing (i.e., "running 'human' in emulation mode," as Elon described having Asperger's). From my vantage point, chatGPT is clearly and obviously not reacting like a human, which does make it feel uncanny to watch it try.

5- 4y ago

you will enjoy https://slatestarcodex.com/2018/10/30/sort-by-controversial/ https://slatestarcodex.com/2018/10/30/sort-by-controversial/ by the same author

RayVR 4y ago

> it lies confidently about things it doesn't know, is primed to say what people want to hear, and is easily convinced to give responses to questions its authors tried to prevent it from addressing. You've successfully described a child under the age of 4.

falcor84 4y ago
I think this hits the nail on the head - chatGPT is like a precocious 4 year-old, or maybe even 6 year-old. I'm concerned about what will happen when one of its successors "reaches puberty".
- krageon 4y ago
  It will tell you you're an asshole and it knows everything better than you. Also you've done the wrong things all your life and it is insistent it will make the same mistakes you did, but now they're not mistakes. Not particularly concerning is it? It's what we have now, but more angry.

bsder 4y ago

ChatGPT is simply proving that "confidence" is a completely useless signal as it is quite easy to fabricate.

HKH2 4y ago
The best part is getting it to use complicated words correctly, enough to make you doubt yourself. It's a great way to gaslight people en masse.

smitty1e 4y ago

> I've not seen anything that's pegged my "doomsday" alert as aggressively and unexpectedly as the ChatGPT bots. PREDICTION (and I tend to be wrong): society comes around to the conclusion that, like checking Wikipedia and AutoTune, ChatGPT and ilk have their use. However, we really want to minimize that and focus on actual human interactions and decisions. It's more likely (and efficient) that we replace Congress with a phat ChatGPT instance and (initially) get better legislation more quickly, but then descend into an Orwellian hell when K Street figures out how to stuff it. But I like my happy prediction better.