4 ms·

> GPT-3 continuation: A confused voice came from inside. When I opened the door, the person that looked back at me was Hayama Hayato. Why was Hayama, who I only

by bozzcl 6y ago

> GPT-3 continuation: A confused voice came from inside. When I opened the door, the person that looked back at me was Hayama Hayato. Why was Hayama, who I only shared memories of me playing soccer with, in my room at this hour of the night? That question immediately flew out from my mouth. [I knew that name felt familiar](https://oregairu.fandom.com/wiki/Hayato_Hayama https://oregairu.fandom.com/wiki/Hayato_Hayama). Does that mean GPT-3 was trained on an arbitrary, huge database of text? I wonder how copyright applies here.

lumost 6y ago
There's an open question if GPT-3 simply provides a text completion API for the internet. The original paper discusses the risk that datasets for specific evaluation tasks (or equivalents) were simply memorized by the model. A completion API for the internet could still be an incredibly valuable component.
- fock 6y ago
  yes! if a thing of similar performance as GPT-3 could just spew out links to where its "inspiration" comes from, this would be (super expensive?) real context-aware search. This could be really great.
CharlesW 6y ago
> Does that mean GPT-3 was trained on an arbitrary, huge database of text? Yes, 500 billion words, for a model with model 175 billion parameters.
- bozzcl 6y ago
  So that continues my question: how much of that text would be on the private domain? If there is, is it all legally purchased?
  - jcims 6y ago
    I’m sure some lawyers figured it out (fair use maybe?) but they basically used scraped content from Common Crawl. Trying to figure out how to license that would be harder than trying to get all the land rights for a railroad across the US. So... probably not purchased.
  - pontifier 6y ago
    I wonder if that's one reason they can't open it up. Perhaps they are ensuring that responses that come from the AI are sufficiently different from anything in the training corpus, so it can't be queried for sensitive data or large chunks of copyrighted material.
    - Trasmatta 6y ago
      It's basically already open through AI Dungeon. I've had all sorts of conversations so far with copyrighted characters.
      - dddbbb 6y ago
        That's still coming via the API. They may be blocking any large chunks of text from being reproduced, rather than individual characters (which would be harder to police).
  - rcxdude 6y ago
    This is an excellent question which applies to a lot of machine learning datasets. AFAIK there is no specific licensing of much of it (licenses you see are generally attached to the labels: the images/text/etc are often not even part of the download and you need to go scrape them yourself) and it's often claimed that the results of the network are free from the copyright of the training data used to create it, but this is contentious and has definitely not been tested in court.
YeGoblynQueenne 6y ago
>> [I knew that name felt familiar](https://oregairu.fandom.com/wiki/Hayato_Hayama https://oregairu.fandom.com/wiki/Hayato_Hayama). Does that mean GPT-3 was trained on an arbitrary, huge database of text? I wonder how copyright applies here. It could have lifted the name (and some additional context in the generated sentence) from the fandom wiki you link to, or something similar. It probably wasn't trained with the text of light novels; even though you can er find some of those online, they are generally scans and GPT-3 is trained on text, as far as I can tell. In any case if it was lifted from sources about Oregairu and not the light novel itself, then it'd most likely be considered fair use. I mean, there's a wikipedia article that describes the characters (including Hayato Hayama) and all. P.S. I haven't read that one. Is it any good?

by bozzcl 6y ago

lumost 6y ago

There's an open question if GPT-3 simply provides a text completion API for the internet. The original paper discusses the risk that datasets for specific evaluation tasks (or equivalents) were simply memorized by the model. A completion API for the internet could still be an incredibly valuable component.

fock 6y ago
yes! if a thing of similar performance as GPT-3 could just spew out links to where its "inspiration" comes from, this would be (super expensive?) real context-aware search. This could be really great.

CharlesW 6y ago

> Does that mean GPT-3 was trained on an arbitrary, huge database of text? Yes, 500 billion words, for a model with model 175 billion parameters.

bozzcl 6y ago
So that continues my question: how much of that text would be on the private domain? If there is, is it all legally purchased?
- jcims 6y ago
  I’m sure some lawyers figured it out (fair use maybe?) but they basically used scraped content from Common Crawl. Trying to figure out how to license that would be harder than trying to get all the land rights for a railroad across the US. So... probably not purchased.
- pontifier 6y ago
  I wonder if that's one reason they can't open it up. Perhaps they are ensuring that responses that come from the AI are sufficiently different from anything in the training corpus, so it can't be queried for sensitive data or large chunks of copyrighted material.
  - Trasmatta 6y ago
    It's basically already open through AI Dungeon. I've had all sorts of conversations so far with copyrighted characters.
    - dddbbb 6y ago
      That's still coming via the API. They may be blocking any large chunks of text from being reproduced, rather than individual characters (which would be harder to police).
- rcxdude 6y ago
  This is an excellent question which applies to a lot of machine learning datasets. AFAIK there is no specific licensing of much of it (licenses you see are generally attached to the labels: the images/text/etc are often not even part of the download and you need to go scrape them yourself) and it's often claimed that the results of the network are free from the copyright of the training data used to create it, but this is contentious and has definitely not been tested in court.

YeGoblynQueenne 6y ago

>> [I knew that name felt familiar](https://oregairu.fandom.com/wiki/Hayato_Hayama https://oregairu.fandom.com/wiki/Hayato_Hayama). Does that mean GPT-3 was trained on an arbitrary, huge database of text? I wonder how copyright applies here. It could have lifted the name (and some additional context in the generated sentence) from the fandom wiki you link to, or something similar. It probably wasn't trained with the text of light novels; even though you can er find some of those online, they are generally scans and GPT-3 is trained on text, as far as I can tell. In any case if it was lifted from sources about Oregairu and not the light novel itself, then it'd most likely be considered fair use. I mean, there's a wikipedia article that describes the characters (including Hayato Hayama) and all. P.S. I haven't read that one. Is it any good?