5 ms·

Just so we're all on the same page, this is a random GPT-3 sample (I clicked "Random" twice, the first one was a short Wikipedia-like article, this was the seco

Just so we're all on the same page, this is a random GPT-3 sample (I clicked "Random" twice, the first one was a short Wikipedia-like article, this was the second): https://read-the-samples.netlify.app/sample_1749 https://read-the-samples.netlify.app/sample_1749 As far as I'm concerned, I would probably pick this over a random fanfic in a Turing Test.

mkl 6y ago
That's amazing. The only problems I caught are that there's a spurious double quote right at the start, and Murphy seems to have two children, a son and a daughter, but then later refers to "the boys". To really evaluate these samples, we need a way to search for phrases in the training data, to see how much is just learned and copied. I've tried Google and DDG, but not found anything.
- hyperman1 6y ago
  I also wonder how much is invented, and how much is simply copied. I saw this one by GPT3 https://read-the-samples.netlify.app/sample_2007 https://read-the-samples.netlify.app/sample_2007 and compare it to e.g. https://www.bbc.com/news/world-europe-35869985 https://www.bbc.com/news/world-europe-35869985 GPT-3 has quite a lot of correct facts, manages to mix fouad belkacem and Sharia4Belgium into it, etc... I suspect this is what happens when there is not enough training data: You get a lookalike, not something new. To be clear: It's impressive what GPT-3 is capable of, and it's very probable something better is comming in the near future. (It's also quite amusing how a lot of these terrorists get first killed by the police and then thrown into jail for 20 years. That'll teach their corpses! )
  - gomox 6y ago
    > I also wonder how much is invented, and how much is simply copied Well, the average human probably does not excel at this metric either
- taneq 6y ago
  I don't know much about these kinds of neural nets but this is the thing I've wondered about - with billions of parameters and enormous training sets, it seems more than likely that it's squirreled away fair chunks of text inside itself that it can regurgitate. Edit: I pasted the random sample above into Grammarly's plagiarism detector and it says it "detected plagiarism" but I'm not paying to sign up just to find out what it said (also they make you sign up with your email address before mentioning that it costs money... rude!) Maybe someone with a Grammarly subscription can try it? Edit 2: Getting off-topic but wow, Grammarly is a nest of dark patterns. Doesn't tell you you need to create an account until you've pasted your text in. Doesn't tell you you need to pay a subscription until you've created an account. Sends you a follow-up email a few minutes later if you don't subscribe. Puts the account deletion link in a different colour and location to the rest of the account settings so it looks like a footer. Swaps the styling for the 'delete account' and 'keep account' links when you do find the delete button to try and steer you away from deleting your account. After that lot I'm never giving them a cent.
deleted 6y ago
[deleted]
Veedrac 6y ago
See also Gwern's Twitter posts: https://twitter.com/gwern/status/1273643406766505987 https://twitter.com/gwern/status/1273643406766505987 https://twitter.com/gwern/status/1274380298910826501 https://twitter.com/gwern/status/1274380298910826501 https://twitter.com/gwern/status/1273658044677853184 https://twitter.com/gwern/status/1273658044677853184
sooheon 6y ago
I was very impressed, but two clicks later landed me here: https://read-the-samples.netlify.app/sample_1792 https://read-the-samples.netlify.app/sample_1792 This reads exactly like bad plagiarism, with the word count padding repetitive phrases and too-on-the-nose facts. It's not generating, it's memorizing.
- jeremysalwen 6y ago
  Well to be fair it's learning from the internet, where there are lots of low-information content and repetitive web pages. I think also the lack of formatting makes things seem more artificial than they are.
  - imtringued 6y ago
    I don't think a film analysis written by a human would consist of five separate introductions.
- afiori 6y ago
  > This reads exactly like bad plagiarism Are you also saying that it is plagiarism or that it just look like it? To be honest considering the huge training sets and knowing nothing of the testing methodology I have some lurking suspicion that it could be just copy-pasting chunks of texts... On the other hand if it is just writing "articles that feels a lot like they are plagiarism" then I suppose it doing its job properly considering what you find on the internet.
  - sooheon 6y ago
    I guess it's both. It reads like plagiarism of bad plagiarism.
    - deleted 6y ago
      [deleted]
aidenn0 6y ago
Most paragraphs read fine, but the story itself is fairly incoherent.

mkl 6y ago

That's amazing. The only problems I caught are that there's a spurious double quote right at the start, and Murphy seems to have two children, a son and a daughter, but then later refers to "the boys". To really evaluate these samples, we need a way to search for phrases in the training data, to see how much is just learned and copied. I've tried Google and DDG, but not found anything.

hyperman1 6y ago
I also wonder how much is invented, and how much is simply copied. I saw this one by GPT3 https://read-the-samples.netlify.app/sample_2007 https://read-the-samples.netlify.app/sample_2007 and compare it to e.g. https://www.bbc.com/news/world-europe-35869985 https://www.bbc.com/news/world-europe-35869985 GPT-3 has quite a lot of correct facts, manages to mix fouad belkacem and Sharia4Belgium into it, etc... I suspect this is what happens when there is not enough training data: You get a lookalike, not something new. To be clear: It's impressive what GPT-3 is capable of, and it's very probable something better is comming in the near future. (It's also quite amusing how a lot of these terrorists get first killed by the police and then thrown into jail for 20 years. That'll teach their corpses! )
- gomox 6y ago
  > I also wonder how much is invented, and how much is simply copied Well, the average human probably does not excel at this metric either
taneq 6y ago
I don't know much about these kinds of neural nets but this is the thing I've wondered about - with billions of parameters and enormous training sets, it seems more than likely that it's squirreled away fair chunks of text inside itself that it can regurgitate. Edit: I pasted the random sample above into Grammarly's plagiarism detector and it says it "detected plagiarism" but I'm not paying to sign up just to find out what it said (also they make you sign up with your email address before mentioning that it costs money... rude!) Maybe someone with a Grammarly subscription can try it? Edit 2: Getting off-topic but wow, Grammarly is a nest of dark patterns. Doesn't tell you you need to create an account until you've pasted your text in. Doesn't tell you you need to pay a subscription until you've created an account. Sends you a follow-up email a few minutes later if you don't subscribe. Puts the account deletion link in a different colour and location to the rest of the account settings so it looks like a footer. Swaps the styling for the 'delete account' and 'keep account' links when you do find the delete button to try and steer you away from deleting your account. After that lot I'm never giving them a cent.

deleted 6y ago

[deleted]

Veedrac 6y ago

See also Gwern's Twitter posts: https://twitter.com/gwern/status/1273643406766505987 https://twitter.com/gwern/status/1273643406766505987 https://twitter.com/gwern/status/1274380298910826501 https://twitter.com/gwern/status/1274380298910826501 https://twitter.com/gwern/status/1273658044677853184 https://twitter.com/gwern/status/1273658044677853184

sooheon 6y ago

I was very impressed, but two clicks later landed me here: https://read-the-samples.netlify.app/sample_1792 https://read-the-samples.netlify.app/sample_1792 This reads exactly like bad plagiarism, with the word count padding repetitive phrases and too-on-the-nose facts. It's not generating, it's memorizing.

jeremysalwen 6y ago
Well to be fair it's learning from the internet, where there are lots of low-information content and repetitive web pages. I think also the lack of formatting makes things seem more artificial than they are.
- imtringued 6y ago
  I don't think a film analysis written by a human would consist of five separate introductions.
afiori 6y ago
> This reads exactly like bad plagiarism Are you also saying that it is plagiarism or that it just look like it? To be honest considering the huge training sets and knowing nothing of the testing methodology I have some lurking suspicion that it could be just copy-pasting chunks of texts... On the other hand if it is just writing "articles that feels a lot like they are plagiarism" then I suppose it doing its job properly considering what you find on the internet.
- sooheon 6y ago
  I guess it's both. It reads like plagiarism of bad plagiarism.
  - deleted 6y ago
    [deleted]

aidenn0 6y ago

Most paragraphs read fine, but the story itself is fairly incoherent.