3 ms·

Curious as to what people think wrt the final question " Do we trust [OpenAI] to take on this role [of deciding who can have access to newer SOTA models and who

Curious as to what people think wrt the final question " Do we trust [OpenAI] to take on this role [of deciding who can have access to newer SOTA models and who can't]? And if not, how can academics and practitioners fight for the continued democratization of AI, as some of its most important techniques become as hard to replicate as GPT-3?" It seems like for super gigantic models, good infra to train on cloud deployment (for which many labs can have credits / grants) would be the first priority, followed perhaps by good compression/pruning tech so after training inference can be done on one/fewer GPUs. Any other things?

visarga 6y ago
Maybe the authors of the text should also have a stake in GPT-3. After all, OpenAI didn't write the corpus. Google benefits from the web, and Facebook from the real-life social networks and their activities that it replicates online (messages, meetings, news, etc). We are all the source of the training data. Why should we be at the whims of these derivative product companies?
- andreyk 6y ago
  Fun fact, OpenAI did not collect the data itself - they mainly used data from Common Crawl (in addition to a couple other datasets), which is compiled by a non profit that shares the dataset for free. So perhaps the license of such datasets can encourage free sharing of research outcomes. https://commoncrawl.org/ https://commoncrawl.org/
  - searchableguy 6y ago
    Shouldn't I own part of gpt3 because it's trained on my data and likely spitting out what I have commented somewhere on the internet? I think ml models should be public unless the data itself isn't.
    - searchableguy 6y ago
      Curious about the downvotes. If you remove compute and code, what remains is data that isn't owned by a single entity in the case of gpt3. You can sell both compute and code which is what you own but should you be able to sell data? You can't translate a commerical book without paying the copyright holder. You own the copyright for the translation but everything else still remains that of the original author. Why wouldn't this apply in case of a machine learning model trained on my images, messages, intents, likeness, etc?
      - isx726552 6y ago
        But you don’t understand, OpenAI has hype and “rockstars” and big names attached, so those rules don’t apply. Your data? It’s their data now. (And Microsoft’s.)
      - PeterisP 6y ago
        The current legal understanding is that according to the current copyright laws the ML models trained on some data are not considered derivative works of that data, so if the data was obtained legally and without other restrictions (e.g. if the researchers sign a contract with the data owner that gives them access to data but agree to certain conditions) then the authors of the data do not have any claim on some rights to the resulting model. One aspect of this is that historically statistical models calculated from large volumes of text (which is a notion that predates computers, e.g. frequency dictionaries and the whole [sub]field of quantitative corpus linguistics) have been considered facts about that corpus of text and thus not copyrightable at all or (depending on jurisdiction) entitled to different set of protections/limitations assigned to compilations of facts, which give some rights to the people who compiled the facts but no rights to the source of these facts (since facts as such can't be copyrighted). This also applies to many forms of analysis of audiovisual data, where the copyrights of the source works do not transfer to the results of the statistical or qualitative analysis and can't limit their creation, distribution or sale. The appropriate analogy to a commercial book or movie is not a translation, but some analysis of it - e.g. a thorough literary review and critique of some book or movie is a separate work with its own copyright, and the original author has no claim on it despite the fact that is (obviously) based on the contents of the work and describes it in great detail. Including verbatim fragments of the work is limited (fair use allows some inclusions but not all), but all the other details are not. Also, the whole notion of copyrightability of ML model weight files is interesting and IMHO not settled. You could argue that there is some creative expression in forming the model (which would support it being copyrightable) or you could argue that it's a mechanistic result of the application of some algorithm and settings (which have the creative part, and are copyrightable on their own), and so the output can't be copyrightable, no matter how much work (human or machine), time and cost it took - at least in USA copyright law doctrine (e.g. Feist Publications v. Rural Telephone Service) is that mere "sweat of the brow" (no matter how much) does not entitle a work to copyright protection; it requires application of human creativity to create an original work, and automated processes can't satisfy that requirement. So from the perspective of GPT-3 authors it might be prudent to distribute the model only to people who agree to specific, explicit contractual/licensing restrictions on redistribution and usage of the model instead of simply relying on the default application of copyright laws. And crucially, if some output is not copyrightable, it can't be considered a derived work to which the authors of a source work could have some claim.
whichquestion 6y ago
Would we be able to democratize these large ML models using something like SETI@Home or similar? Would more research in distributing models across large networks be useful? Has research on this topic been done?

visarga 6y ago

Maybe the authors of the text should also have a stake in GPT-3. After all, OpenAI didn't write the corpus. Google benefits from the web, and Facebook from the real-life social networks and their activities that it replicates online (messages, meetings, news, etc). We are all the source of the training data. Why should we be at the whims of these derivative product companies?

andreyk 6y ago
Fun fact, OpenAI did not collect the data itself - they mainly used data from Common Crawl (in addition to a couple other datasets), which is compiled by a non profit that shares the dataset for free. So perhaps the license of such datasets can encourage free sharing of research outcomes. https://commoncrawl.org/ https://commoncrawl.org/
- searchableguy 6y ago
  Shouldn't I own part of gpt3 because it's trained on my data and likely spitting out what I have commented somewhere on the internet? I think ml models should be public unless the data itself isn't.
  - searchableguy 6y ago
    Curious about the downvotes. If you remove compute and code, what remains is data that isn't owned by a single entity in the case of gpt3. You can sell both compute and code which is what you own but should you be able to sell data? You can't translate a commerical book without paying the copyright holder. You own the copyright for the translation but everything else still remains that of the original author. Why wouldn't this apply in case of a machine learning model trained on my images, messages, intents, likeness, etc?
    - isx726552 6y ago
      But you don’t understand, OpenAI has hype and “rockstars” and big names attached, so those rules don’t apply. Your data? It’s their data now. (And Microsoft’s.)
    - PeterisP 6y ago
      The current legal understanding is that according to the current copyright laws the ML models trained on some data are not considered derivative works of that data, so if the data was obtained legally and without other restrictions (e.g. if the researchers sign a contract with the data owner that gives them access to data but agree to certain conditions) then the authors of the data do not have any claim on some rights to the resulting model. One aspect of this is that historically statistical models calculated from large volumes of text (which is a notion that predates computers, e.g. frequency dictionaries and the whole [sub]field of quantitative corpus linguistics) have been considered facts about that corpus of text and thus not copyrightable at all or (depending on jurisdiction) entitled to different set of protections/limitations assigned to compilations of facts, which give some rights to the people who compiled the facts but no rights to the source of these facts (since facts as such can't be copyrighted). This also applies to many forms of analysis of audiovisual data, where the copyrights of the source works do not transfer to the results of the statistical or qualitative analysis and can't limit their creation, distribution or sale. The appropriate analogy to a commercial book or movie is not a translation, but some analysis of it - e.g. a thorough literary review and critique of some book or movie is a separate work with its own copyright, and the original author has no claim on it despite the fact that is (obviously) based on the contents of the work and describes it in great detail. Including verbatim fragments of the work is limited (fair use allows some inclusions but not all), but all the other details are not. Also, the whole notion of copyrightability of ML model weight files is interesting and IMHO not settled. You could argue that there is some creative expression in forming the model (which would support it being copyrightable) or you could argue that it's a mechanistic result of the application of some algorithm and settings (which have the creative part, and are copyrightable on their own), and so the output can't be copyrightable, no matter how much work (human or machine), time and cost it took - at least in USA copyright law doctrine (e.g. Feist Publications v. Rural Telephone Service) is that mere "sweat of the brow" (no matter how much) does not entitle a work to copyright protection; it requires application of human creativity to create an original work, and automated processes can't satisfy that requirement. So from the perspective of GPT-3 authors it might be prudent to distribute the model only to people who agree to specific, explicit contractual/licensing restrictions on redistribution and usage of the model instead of simply relying on the default application of copyright laws. And crucially, if some output is not copyrightable, it can't be considered a derived work to which the authors of a source work could have some claim.

whichquestion 6y ago

Would we be able to democratize these large ML models using something like SETI@Home or similar? Would more research in distributing models across large networks be useful? Has research on this topic been done?