4 ms·

>they correctly identified only 68% of the generated abstracts and 86% of the genuine abstracts. I think you and I are basically in alignment... what this tell

>they correctly identified only 68% of the generated abstracts and 86% of the genuine abstracts. I think you and I are basically in alignment... what this tells me is that 14% of real abstracts are so bad that other human beings call their BS. Meanwhile, this AI stuff is kinda working 32% of the time in generating legitimately interesting ideas. So at that point - yeah, that sounds about right. The 32% is still so low that it shows AI is not anywhere near maturity, whereas 14% of human-generated is crap. And, yeah - a short blurb like an abstract seems to be exactly the kind of text that ChatGPT is conditioned to do well generating. As others below note - once a human starts reading the rest, the alarm bells trigger.

sdenton4 4y ago
(Buuuut let us also enjoy the fact that ML can generate a decent-looking abstract to a scientific paper 32% of the time. This represents massive progress; I would venture that most humans cannot write such an abstract convincingly.)
- tduberne 4y ago
  I would argue the opposite. Give anyone who can write reasonnably the abstract of a few dozen papers in a given field, and I am confident they would be able to produce a convincing bullshit abstract. Most abstracts in a given field sound extremely similar, and are mostly keyword dropping with an ounce of self-promotion. I actually think that the only reason ChatGPT did not produce a higher rate of convincing abstracts is that the reaserchers assessing them knew they had a high probability to look at an AI result and were thus extra careful. Most of the abstracts correctly labelled as AI would probably be considered legit (which does not mean good) if sent to a conference without warning.
seanmcdirmid 4y ago
It really would depend on the quality of the work/paper. I bet mis-identification by humans of hand written or generated abstracts goes down in higher tier conferences vs. lower tier ones, since the writing standards in lower tier conferences can be pretty bad (well, in CS at least).
- poulsbohemian 4y ago
  I suspect you are right Sean - and I know you are a guy who has read a lot of papers and been to plenty of conferences.
sarchertech 4y ago
>Meanwhile, this AI stuff is kinda working 32% of the time in generating legitimately interesting ideas. I don't know that interesting has anything to do with whether people found them plausible.
klyrs 4y ago
> ...this AI stuff is kinda working 32% of the time in generating legitimately interesting ideas. The bar is not "this is legitimately interesting," it's "I've seen real abstracts worse than this."

sdenton4 4y ago

(Buuuut let us also enjoy the fact that ML can generate a decent-looking abstract to a scientific paper 32% of the time. This represents massive progress; I would venture that most humans cannot write such an abstract convincingly.)

tduberne 4y ago
I would argue the opposite. Give anyone who can write reasonnably the abstract of a few dozen papers in a given field, and I am confident they would be able to produce a convincing bullshit abstract. Most abstracts in a given field sound extremely similar, and are mostly keyword dropping with an ounce of self-promotion. I actually think that the only reason ChatGPT did not produce a higher rate of convincing abstracts is that the reaserchers assessing them knew they had a high probability to look at an AI result and were thus extra careful. Most of the abstracts correctly labelled as AI would probably be considered legit (which does not mean good) if sent to a conference without warning.

seanmcdirmid 4y ago

It really would depend on the quality of the work/paper. I bet mis-identification by humans of hand written or generated abstracts goes down in higher tier conferences vs. lower tier ones, since the writing standards in lower tier conferences can be pretty bad (well, in CS at least).

poulsbohemian 4y ago
I suspect you are right Sean - and I know you are a guy who has read a lot of papers and been to plenty of conferences.

sarchertech 4y ago

>Meanwhile, this AI stuff is kinda working 32% of the time in generating legitimately interesting ideas. I don't know that interesting has anything to do with whether people found them plausible.

klyrs 4y ago

> ...this AI stuff is kinda working 32% of the time in generating legitimately interesting ideas. The bar is not "this is legitimately interesting," it's "I've seen real abstracts worse than this."