11 ms·

I've said it before, but it needs repeating. chatGPT is a very confident FICTION generator. Any facts it produces are purely coincidental. Please stop assumin

I've said it before, but it needs repeating. chatGPT is a very confident FICTION generator. Any facts it produces are purely coincidental. Please stop assuming anything it says it true. It was never designed to do that and it provably doesn't do that.

KineticLensman 4y ago
A description I saw here a couple of days ago was ‘blarney generator’. I think that captures it perfectly
haolez 4y ago
I'm using it for summarization and classification of big texts and it's showing a hit-ratio bigger than human analysts. I'm satisfied with it as is :)
- vidarh 4y ago
  I'm ambivalent. It summarises text on papers that are simple enough, but interestingly it fails in ways that suggests it sometimes fails because it isn't "smart enough." E.g. I asked it about "Code Generation on the Fly: A Key to Portable Software", the doctoral dissertation of Professor Michel Franz, and a paper I know very well, and it spit back a description of the process that was perhaps superficially correct but the kind of correct you'd expect of a lazy student who'd not bothered to actually understand the paper. The summary was largely meaningless twaddle that didn't actually tell you anything useful. I then started probing it for more detail, and pointing out mistaken assumptions, and it got better. But it was first when I pointed out that the technique for encoding and decoding has a resemblance to Lempel-Ziv-Welch compression that it succeeded in identifying the key part of the paper and give a description that while still not quite accurate at least captured the essence of the method. It felt like I was giving an exam on the topic rather than "getting help" summarising it, and if I didn't know the paper, it would've been hard to tell if I was getting close without reading it. For conceptually simpler things, though, I've gotten great results, and it seems to work great for things where I know what a good result looks like ahead of time, and "just" want to save looking up details where I'll recognise the right thing when I see it but don't remember it by heart. That said, to be able to point out the resemblance to LZW and have it connect the dots is at the same time fairly impressive - I've had to point that out to experienced developers (the paper mentions Welch, "This method bears some resemblance to commonly used data compression schemes [Wel84]", with Wel84 reading: "T. A. Welch; A Technique for High_Performance Data Compression; IEEE Computer, 17:6, 8-19; 1984. {2}" but it does not go into any further detail of how they relate). I think, overall, that this is an indication that it will get better at this as the model size increases further. It was able to at least recognise the semantic similarity or identify that this similiarity has been described elsewhere and use it to guide it's response in a way that strikes me as far from trivial. [It did go on to hallucinate several relevant papers on the subject; interestingly it confirmed it had made them up when I asked if it had]
historymystery 4y ago
I had some fun asking chatGPT some historical questions where it quickly veered into saying things I knew were false. I then asked for citations: it gave me what sounded like nice academic journal articles and books, but these were made entirely made up. Pointing this out just made it give different references, also totally fake.
baq 4y ago
Yeah it isn’t an encyclopedia so don’t treat it is one. The thing knows language not knowledge. But if you’ve got language to give it to work with, results are spectacular.
- epicureanideal 4y ago
  Maybe it could be used as a tool for people to better understand and better express their own arguments in debates? “What I’m trying to say is…”
abracadaniel 4y ago
I was able to convince it that it was the year 2050, and then got it to describe historical events from future, including the destruction of the moon, and the subsequent rebuilding of the moon. Leading questions are fun, because I asked why the new moon was built using cheese, and it happily gave justification regarding the unique material properties.
- williamcotton 4y ago
  So you went out of your way to ensure that it was creating fiction.
hooande 4y ago
> Please stop assuming anything it says it true. This is overstated and easily disproved. ChatGPT produces accurate facts as a matter of course. test it right now: "How tall is the Statue of Liberty?", "In what state was Abraham Lincoln born?", etc. There are an infinite list of factual questions for which it produces accurate and correct answers. It loves to hallucinate api methods that don't exist. It can struggle with indivudual logic problems or questions. But these limitations have clear explanations. It's doing language completion so it will infer probable things that don't actually exist. It wasn't designed to do logic problems so it will struggle with classes of them. Dismissing it as completely unreliable is unnecessary hyperbole. It's a software tool that has strengths, weaknesses and limitations. Like with any other software, it's up to us to learn what those are and how it can be used to make useful products.
- sarchertech 4y ago
  It’s completely unreliable because it sounds confident and plausible even when it’s wrong. Essentially it fails in ways that are very hard to detect. If you ask it question you have no way to know whether it’s right or not unless you already know the answer. I’ve asked it many questions about my area of expertise, distributed systems, and it was often flat out wrong, but to a non expert it sounded perfectly plausible. I asked my my wife, a physician, to try it out and she reported the exact same problem. Some of the answers it gave had even veered off from just annoying to actually dangerous. That doesn’t mean it can’t be useful, but using it to answer important questions or using it to teach yourself something you don’t already know is dangerous.
  - beefield 4y ago
    > It’s completely unreliable because it sounds confident and plausible even when it’s wrong This describes pretty accurately a non-trivial amount of people I have worked with.
    - sarchertech 4y ago
      And hopefully after someone shows you that they can be described that way, you stop trusting what they say (at least about the thing they are known to be unreliable about).
paganel 4y ago
> chatGPT is a very confident FICTION generator. Any facts it produces are purely coincidental. Wish Stanislaw Lem were still alive and still writing great books, this whole mania around chatGPT and especially articles like this one remind me of his excellent Cyberiad, where at some point two robots from those stories almost get drowned by useless info after useless info (they were printing that info out on pieces of paper). I'm really curious how Trurl and Klapaucius would have handled all this chatGPT non-sense.
reaperducer 4y ago
Please stop assuming anything it says it true. It would be nice if ChatGPT would police itself and not say anything that isn't true. That's what people expect of computers, and why we're in the mess we are now. But then the problem is people disagreeing about whether ChatGPT is a fair arbiter of truth.
williamcotton 4y ago
It is both a reliable synthesizer of fiction and a translator of fact, and empirically so: https://github.com/williamcotton/empirical-philosophy/blob/main/articles/from-prompt-alchemy-to-prompt-engineering-an-introduction-to-analytic-agumentation.md https://github.com/williamcotton/empirical-philosophy/blob/m...
Oxidation 4y ago
I assume it's only a matter of time before someone plugs something similar into Wikidata and rather than using AI to generate facts statistically, look the specifics up where known.
jeffbee 4y ago
Yesterday I asked it to describe the improvements between the 4th and 5th generation Xeon Scalable Processor and it just made some shit up.
dogcomplex 4y ago
Agreed. And fact-checking layers will be coming soon and ya'll don't even want to know what kind of disruption those combined will bring
xxr 4y ago
Should include the classic disclaimer: >The stories and information posted here are artistic^wmechanically generated works of fiction and falsehood. Only a fool would take anything posted here as fact.
deleted 4y ago
[deleted]
hinkley 4y ago
Kind of wonder if there’s going to be a counterculture movement that looks a bit like old school PGP webs of trust. Humans inviting humans and trying to keep out the synths.
PicassoCTs 4y ago
It does funny things though, if you feed it a webcam as png encoded and give it a text based api for a simple robotic gripper on wheels. Give it a marker and ask it to label the world in handwriting.

KineticLensman 4y ago

A description I saw here a couple of days ago was ‘blarney generator’. I think that captures it perfectly

haolez 4y ago

I'm using it for summarization and classification of big texts and it's showing a hit-ratio bigger than human analysts. I'm satisfied with it as is :)

vidarh 4y ago
I'm ambivalent. It summarises text on papers that are simple enough, but interestingly it fails in ways that suggests it sometimes fails because it isn't "smart enough." E.g. I asked it about "Code Generation on the Fly: A Key to Portable Software", the doctoral dissertation of Professor Michel Franz, and a paper I know very well, and it spit back a description of the process that was perhaps superficially correct but the kind of correct you'd expect of a lazy student who'd not bothered to actually understand the paper. The summary was largely meaningless twaddle that didn't actually tell you anything useful. I then started probing it for more detail, and pointing out mistaken assumptions, and it got better. But it was first when I pointed out that the technique for encoding and decoding has a resemblance to Lempel-Ziv-Welch compression that it succeeded in identifying the key part of the paper and give a description that while still not quite accurate at least captured the essence of the method. It felt like I was giving an exam on the topic rather than "getting help" summarising it, and if I didn't know the paper, it would've been hard to tell if I was getting close without reading it. For conceptually simpler things, though, I've gotten great results, and it seems to work great for things where I know what a good result looks like ahead of time, and "just" want to save looking up details where I'll recognise the right thing when I see it but don't remember it by heart. That said, to be able to point out the resemblance to LZW and have it connect the dots is at the same time fairly impressive - I've had to point that out to experienced developers (the paper mentions Welch, "This method bears some resemblance to commonly used data compression schemes [Wel84]", with Wel84 reading: "T. A. Welch; A Technique for High_Performance Data Compression; IEEE Computer, 17:6, 8-19; 1984. {2}" but it does not go into any further detail of how they relate). I think, overall, that this is an indication that it will get better at this as the model size increases further. It was able to at least recognise the semantic similarity or identify that this similiarity has been described elsewhere and use it to guide it's response in a way that strikes me as far from trivial. [It did go on to hallucinate several relevant papers on the subject; interestingly it confirmed it had made them up when I asked if it had]

historymystery 4y ago

I had some fun asking chatGPT some historical questions where it quickly veered into saying things I knew were false. I then asked for citations: it gave me what sounded like nice academic journal articles and books, but these were made entirely made up. Pointing this out just made it give different references, also totally fake.

baq 4y ago

Yeah it isn’t an encyclopedia so don’t treat it is one. The thing knows language not knowledge. But if you’ve got language to give it to work with, results are spectacular.

epicureanideal 4y ago
Maybe it could be used as a tool for people to better understand and better express their own arguments in debates? “What I’m trying to say is…”

abracadaniel 4y ago

I was able to convince it that it was the year 2050, and then got it to describe historical events from future, including the destruction of the moon, and the subsequent rebuilding of the moon. Leading questions are fun, because I asked why the new moon was built using cheese, and it happily gave justification regarding the unique material properties.

williamcotton 4y ago
So you went out of your way to ensure that it was creating fiction.

hooande 4y ago

> Please stop assuming anything it says it true. This is overstated and easily disproved. ChatGPT produces accurate facts as a matter of course. test it right now: "How tall is the Statue of Liberty?", "In what state was Abraham Lincoln born?", etc. There are an infinite list of factual questions for which it produces accurate and correct answers. It loves to hallucinate api methods that don't exist. It can struggle with indivudual logic problems or questions. But these limitations have clear explanations. It's doing language completion so it will infer probable things that don't actually exist. It wasn't designed to do logic problems so it will struggle with classes of them. Dismissing it as completely unreliable is unnecessary hyperbole. It's a software tool that has strengths, weaknesses and limitations. Like with any other software, it's up to us to learn what those are and how it can be used to make useful products.

sarchertech 4y ago
It’s completely unreliable because it sounds confident and plausible even when it’s wrong. Essentially it fails in ways that are very hard to detect. If you ask it question you have no way to know whether it’s right or not unless you already know the answer. I’ve asked it many questions about my area of expertise, distributed systems, and it was often flat out wrong, but to a non expert it sounded perfectly plausible. I asked my my wife, a physician, to try it out and she reported the exact same problem. Some of the answers it gave had even veered off from just annoying to actually dangerous. That doesn’t mean it can’t be useful, but using it to answer important questions or using it to teach yourself something you don’t already know is dangerous.
- beefield 4y ago
  > It’s completely unreliable because it sounds confident and plausible even when it’s wrong This describes pretty accurately a non-trivial amount of people I have worked with.
  - sarchertech 4y ago
    And hopefully after someone shows you that they can be described that way, you stop trusting what they say (at least about the thing they are known to be unreliable about).

paganel 4y ago

> chatGPT is a very confident FICTION generator. Any facts it produces are purely coincidental. Wish Stanislaw Lem were still alive and still writing great books, this whole mania around chatGPT and especially articles like this one remind me of his excellent Cyberiad, where at some point two robots from those stories almost get drowned by useless info after useless info (they were printing that info out on pieces of paper). I'm really curious how Trurl and Klapaucius would have handled all this chatGPT non-sense.

reaperducer 4y ago

Please stop assuming anything it says it true. It would be nice if ChatGPT would police itself and not say anything that isn't true. That's what people expect of computers, and why we're in the mess we are now. But then the problem is people disagreeing about whether ChatGPT is a fair arbiter of truth.

williamcotton 4y ago

It is both a reliable synthesizer of fiction and a translator of fact, and empirically so: https://github.com/williamcotton/empirical-philosophy/blob/main/articles/from-prompt-alchemy-to-prompt-engineering-an-introduction-to-analytic-agumentation.md https://github.com/williamcotton/empirical-philosophy/blob/m...

Oxidation 4y ago

I assume it's only a matter of time before someone plugs something similar into Wikidata and rather than using AI to generate facts statistically, look the specifics up where known.

jeffbee 4y ago

Yesterday I asked it to describe the improvements between the 4th and 5th generation Xeon Scalable Processor and it just made some shit up.

dogcomplex 4y ago

Agreed. And fact-checking layers will be coming soon and ya'll don't even want to know what kind of disruption those combined will bring

xxr 4y ago

Should include the classic disclaimer: >The stories and information posted here are artistic^wmechanically generated works of fiction and falsehood. Only a fool would take anything posted here as fact.

deleted 4y ago

[deleted]

hinkley 4y ago

Kind of wonder if there’s going to be a counterculture movement that looks a bit like old school PGP webs of trust. Humans inviting humans and trying to keep out the synths.

PicassoCTs 4y ago

It does funny things though, if you feed it a webcam as png encoded and give it a text based api for a simple robotic gripper on wheels. Give it a marker and ask it to label the world in handwriting.