3 ms·
>> but the mistakes are not "parroted bullshit", but minor issues here and there. Nonsense. I'm not saying that it is not useful, but I've easily found ChatGP
by toss1 4y ago
>> but the mistakes are not "parroted bullshit", but minor issues here and there.
Nonsense.
I'm not saying that it is not useful, but I've easily found ChatGPT produces far worse than "parroted bullshit", and does so reliably, and multiple times (repeated patterns of mistakes).
Again, I have found ChatGPT4 (but not 3.5) useful and even educational in some cases, but it's usability requires far more skeptical and error-checked approach.
As one example, since I have a top specialist attorney in the household, I started with some broad legal questions about their field, and then asked their opinion. ChatGPT4's answers were impressive to both of us, and especially the solid grammar.
When the questions got more particular about their specialist area of the law, ChatGPT4's answers got shaky very fast. Asked about a particular somewhat obscure technique, it parroted some very nice-sounding bullshit, which got the concepts very backwards (but it was still very confident and grammatical).
When asked to summarize a published paper about a new innovation in this context, ChatGPT4 very grammatically and confidently completely missed every significant point, again in a very backwards way.
Worse yet, when asked to tell us the paper's publication date, author, and author's employer, all written clearly at the top, it confidently got everything massively wrong. Even after being told to look on the line after "Author: ". Even after being told to look after "Author: FName". It eventually settled on the correct date, but even after being told FName, it continuously kept spouting false authors (sometimes including correct FName) and hallucinated middle and surnames and employers. Interestingly enough, most of the hallucinated authors and employers were well known in the region. So, it is literally establishing the regional context and just spitting out plausibly relevant stuff for that region, but not even figuring out how to summarize something that a 6yo kid could easily sort.
It was hilariously bad. And only hilarious because it was a test (if someone actually relied upon it, it'd be very costly).
Similar results for my field. I got ChatGPT4 to produce some genuinely useful information. But it would also happily hallucinate papers, authors, companies, chemicals, and pretty much anything else.
Middle-of-the-road, where ChatGPT4 has ingested insane amounts of data, it probably knows more than we do, and may be useful for some things.
As soon as you go even slightly off the centerline, it becomes VERY unreliable.
If you take it at face value, you are a fool, being fooled.
EDIT: typos, readability
- IanCal 4y agoThe chat context isn't as big as is available in the API - was the author name in its context window? Papers can be fairly long
- dfox 4y agoApparently the model is intentionally trained such that it does not “know” any kind of metadata about its training inputs. So it can deduce the name of the author if it was referenced elsewhere, but not from the paper itself.
- toss1 4y agoThat's interesting, but why would the data we provide for context or explicitly for summarization be treated like training data? And if it is excluded, why wouldn't it just default to a response resembling "I am not setup to identify and extract that data."?
- toss1 4y agoGood thought, but the paper was a couple of pages long, max. Less than 2500 words, less than 15K characters (sorry, I should have mentioned that). Also, the fact that it could pick up the provided FName and still hallucinate MidName and Surname and Employer kind of gives away that it was in context. It is really funny (as long as it isn't a serious situation) how it cannot identify anything, but the context will give it cues to hallucinate