7 ms·

It seemed obvious to me for a long time before modern LLM training that any sort of training of machine intelligence would have to rely on pirated content. Ther

It seemed obvious to me for a long time before modern LLM training that any sort of training of machine intelligence would have to rely on pirated content. There's just no other viable alternative for efficiently acquiring large quantities of text data. Buying millions of ebooks online would take a lot of effort, downloading data from publishers isn't a thing that can be done efficiently (assuming tech companies negotiated and threw money at them), the only efficient way to access large volumes of media is piracy. The media ecosystem doesn't allow anything else.

the-rc 2y ago
Google has scans from Google Books, as well as all the ebooks it sells on the Play Store.
- lemoncookiechip 2y ago
  Wouldn't that still be piracy? They own the rights of distribution, but do they (or Amazon) have the rights to use said books for LLM training? And what rights would those even be?
  - XorNot 2y ago
    Literally no rights agreement covers LLMs. They cover reproduction of the work, but LLMs don't obviously do this i.e. that the model transiently runs an algorithm over the text is superficially no different to the use of any other classifier or scoring system like those already used by law firms looking to sue people for sharing torrents.
    - deleted 2y ago
      [deleted]
    - thatcat 2y ago
      do those classifiers read copyrighted material? i thought they simply joined the swarm and seeded (reproduction with permission) youtube, etc classifiers definitely do read others material though.
    - visarga 2y ago
      > They cover reproduction of the work, but LLMs don't obviously do this LLMs are much smaller than their training sets, there is no space to memorize the training data. They might memorize small snippets but never full books. They are the worst infringement tools ever made - why replicate Harry Potter by LLM, it's show, expensive and lossy, when you could download the book so much easier. A second argument is that using the LLM blends a new intent into the process, that of the prompter. This can render the outputs transformative. And most LLM interactions are one-time use, like a scratch pad not like a finished work.
      - Earw0rm 2y ago
        The lossy compression argument is interesting. How many bits of entropy in Harry Potter? How many bits of entropy in a lossy-compressed abridgement that is nevertheless enough, when reconstituted, to constitute a copyright infringement of Harry Potter? The latter is absolutely small enough to fit in an LLM, although how close it would get to the original work is debatable. The question is whether copyright is violated: 1) inherently by the model operator, during the training. 2) by the model/model owner, as part of the generation. 3) by the user, in making the model so so and then reproducing the result.
        dijksterhuis 2y ago
        my personal perspectives 1) straight up copying. download a bunch of copyrighted stuff -> making a copy. no way out of this one. 2) a derivative work can be/is being generated here. very grey area — what counts as a “derivative” work? read about robin thicke blurred lines court case for a rollercoaster of a time about derivative musical works. 3) making the model so so? do you mean getting an output and user copying the result? that’s copying the derivative work, which, depends on whatever copyright agreement happens once a derivative work claim is sorted out. that’s based on my 5 years of music copyright experience, although it was about ten years ago now so might be some stuff i’ve got wrong there.
        visarga 2y ago
        You can ensure a model trains on transformative not derivative synthetic texts, for example, by asking for summary, or turning it into QA pairs, or doing contrastive synthesis across multiple copyrighted works. This will ensure the resulting model will never regurgitate the training set because it has not seen it. This approach only takes abstract ideas from copyrighted sources, protecting their specific expression. If abstract ideas were protectable what would stop a LLM from learning not from the original source but from social commentary and follow up works? We can't ask people not to reproduce ideas they read about. But on the other hand, protecting abstractions would kneecap creativity both in humans and AI.
  - majormajor 2y ago
    It means they have existing relationships/contacts to reach out to for negotiating the rights for other usages of that content. I think it negates (for the case of Google/Apple/Amazon who all sell ebooks) the claim made that efficiently acquiring the digital texts wouldn't be possible.
  - brookst 2y ago
    It’s a good question. Textbook companies especially would be pretty enthusiastic about a new “right to learn” monetization strategy. And imagine how lucrative it would be if you could prove some major artist didn’t copy your work, but learned from your work. The whole chain of scientific and artistic development could be monetized in perpetuity. I think this is a dangerous road with little upside for anyone outside of IP aggregators.
  - bawolff 2y ago
    > but do they (or Amazon) have the rights to use said books for LLM training? The real question is - does copyright grant the authors' the right to control if their work is used for LLM training? Its not obvious what the answer is. If authors don't have that right to begin with then there is no way amazon could buy it off them.
- pdpi 2y ago
  Leveraging their position in one market to get a leg up on another market? No idea if it would stick, but that would be one fun antitrust lawsuit right there.
  - brookst 2y ago
    Fun fact: it’s only illegal to leverage a monopoly in one market to advance another. It’s perfectly legal for Coke to leverage their large but not monopolistic soft drink empire to advance their bottled water entries.
    - pdpi 2y ago
      Sure. The whole thing hinges on whether Google has a monopoly on whatever Google Books' market is (hence why I doubted it would stick). But given that some people seem to define "market" broadly enough to conclude that Apple has a monopoly on iPhones...
diggan 2y ago
> There's just no other viable alternative for efficiently acquiring large quantities of text data. [...] take a lot of effort [...] isn't a thing that can be done efficiently [...] only efficient way to access large volumes of media is piracy Hypothetical: If the only way we could build AGI would be to somehow read everyone's brain at least once, would it be worth just ignoring everyone's wish regarding privacy one time to suck up this data and have AGI moving forward?
- nosbo 2y ago
  no
- BriggyDwiggs42 2y ago
  Could this agi cure cancer, and would it be in the hands of the public? Then sure, otherwise nah.
  - onemoresoop 2y ago
    > in the hand of the public Would you trust a businessman on that?
    - BriggyDwiggs42 2y ago
      Nope, they haven’t earned an ounce.
      - anonym29 2y ago
        How about a politician?
        BriggyDwiggs42 2y ago
        Eh, a transparent organization of elected officials with short term limits and strong public oversight, a little bit. A very smart ai would represent power, and so every mechanism we’ve used as humans to guard against power misuse would be the ones I’d want to see here.
IncreasePosts 2y ago
Why would machine intelligence need an entire humanity's worth of data to be machine intelligence? It seems like only a training method that is really poor would need that much data.
aithrowawaycomm 2y ago
I find it highly implausible that Meta doesn't have the resources to obtain these legally. They could have reached out to a publisher and ask to purchase ebooks in bulk - and if that publisher says no, tough shit. The media ecosystem doesn't exist for Big Tech to extract value from it! "It would take a lot of effort to do it legally" is a pathetic excuse for a company of Meta's size.
- gazchop 2y ago
  Perhaps they did and got told no and decided to take it anyway? Defending themselves with technicalities and expensive lawyers may be financially viable. Zero ethics but what would we expect from them?
  - XorNot 2y ago
    Who is "them"? Like, who in the Meta business reporting line made this decision, then how did they communicate it to the engineers who would've been necessary to implement it, particularly at scale? While it's plausible someone downloaded a bunch of torrents and tossed them in the training directory...again, under who's authority? Like if this happened it would be one overzealous data scientist potentially. Hardly "them". People lean on collective pronouns to avoid actually thinking about the mechanics of human enterprise and you get extremely absurd conclusions. (it is not outside the bounds of thinkable that an org could in fact have a very bad culture like this, but I know people who work for Meta, who also have law degrees - they're well aware of the potential problems).
    - aithrowawaycomm 2y ago
      Come on... it's fine that you haven't followed the story, there's a lot going on, but the snotty condescension is very frustrating: These newly unredacted documents reveal exchanges between Meta employees unearthed in the discovery process, like a Meta engineer telling a colleague that they hesitated to access LibGen data because “torrenting from a [Meta-owned] corporate laptop doesn’t feel right ”. They also allege that internal discussions about using LibGen data were escalated to Meta CEO Mark Zuckerberg (referred to as "MZ" in the memo handed over during discovery) and that Meta's AI team was "approved to use" the pirated material. https://www.wired.com/story/new-documents-unredacted-meta-copyright-ai-lawsuit/ https://www.wired.com/story/new-documents-unredacted-meta-co...
ben_w 2y ago
IMO, if the AI were more sample-efficient (a long-standing problem that predates LLMs), they would be able to learn from purely open-licensed content, which I think Wikipedia (CC-BY-SA) would be an example of? I think they'd even pass the share-alike requirements, given Meta are giving away the model weights? https://en.wikipedia.org/wiki/Wikipedia:Copyrights https://en.wikipedia.org/wiki/Wikipedia:Copyrights
- wizzwizz4 2y ago
  Since this is Wikipedia, it could even satisfy the attribution requirements (though most CC-licensed corpora require attributing the individual authors).
- visarga 2y ago
  Alteratively if they trained the model on synthetic data, filtered to avoid duplication, then no copyrighted material would be seen by the model. For example turn an article into QA pairs, or summarize across multiple sources of text.
  - thesz 2y ago
    > trained the model on synthetic data You get knowledge collapse [1] this way. [1] https://arxiv.org/abs/2404.03502 https://arxiv.org/abs/2404.03502
    - ben_w 2y ago
      You can get knowledge collapse. But even though there are counter-examples (e.g. learning Go from self-play based only on the rules), IMO — based on how often people were already doing this with buggy human-written software[0][1] — most people don't think about this in the right way and will therefore treat these things as magical oracles when they shouldn't. No silver bullets. [0] https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal [1] https://en.wikipedia.org/wiki/Computer_says_no https://en.wikipedia.org/wiki/Computer_says_no
mvdtnz 2y ago
AI mega corporations are not entitled to easy and cheap access to data they don't own. If it's a hard problem, too bad. If the stakes are as high as they're all claiming then it should be no problem for them to do this right.
- visarga 2y ago
  > not entitled to easy and cheap access to data they don't own This is not copyright as we know it. Copyright protects against copying, not accessing data. You can still compile statistics off data you don't own. The models are like a compressed version of the originals, so compressed you can't retrieve more than a few snippets of original text. Newer model train on filtered synthetic text, which is one step removed from the protected expression in the copyrighted works. Should abstractions be protected by copyright?
  - svl7 2y ago
    However in order to get to the compressed state, the original data would have to be processed in some way as a whole. This would require a copy of the material to be available. In case that copy was attained in an illegal way, what are the implications?
gunian 2y ago
what about something decentralized? each person trains someone on their own piece of data and somehow that gets aggrgegated into one giant model
- techwizrd 2y ago
  This approach is used in Federated Learning where participants want to collaboratively train a model without sharing raw training data.
  - gunian 2y ago
    are there any companies working on it? was thinking if i train my model on my private docs for instance finance how does one prevent the model from sharing that data verbatim
nh2 2y ago
> Buying millions of ebooks online would take a lot of effort I don't understand. Facebook and Google spend billions on training LLMs. Buying 1M ebooks at $50 each would only cost $50M. They also have >100k engineers. If they shard the ebook buying across their workforce, everyone has to buy 10 ebooks, which will be done in 10 minutes.
- shakna 2y ago
  Google also operates a book store, like Amazon. Both could process a one-off to pay their authors, and then draw from their own backend.
maeil 2y ago
> In the most recent fiscal year, Alphabet's net income amounted to 73.7 billion U.S. dollars Absolutely no way. Yup. > Buying millions of ebooks online would take a lot of effort, downloading data from publishers isn't a thing that can be done efficiently Oh no, it takes effort and can't be done efficiently, poor Google! How can this possibly be an excuse? This is such a detached SV Zuckerberg "move fast and break things"-like take. There's just no way for a lot of people to efficiently get out of poverty without kidnapping and ransoming someone, it would take a lot of effort.
- thatcat 2y ago
  copyright piracy isn't theft, try proving damages for a better arguement
  - maeil 2y ago
    Not my point, never said it is. Substitute that example with another criminal act. Edit: Changed it just for you
    - thatcat 2y ago
      copyright infringement is a civil charge, silly guy. no offense, but there arent many ways to defend its existence in current form without resorting to hyperbolic nonsense and looking silly in the process. so its not a 'crime' and you have to prove damages for a civil offense so... youd need to prove that ai caused damages or how that is materially different from other algorithms like google scanning documents to provide the core utility for their service
brookst 2y ago
I don’t follow the “millions of ebooks are hard” line of thinking. If Meta (or anyone) had approached publishers with a “we want to buy one copy of every book you publish”, that doesn’t seem technical or business difficult. Certainly Amazon would find that extremely easy.
- spencerflem 2y ago
  Buying a book to read and incorporating their text in a product are two different things. Even if they bought the book, imo it would be illegal.
  - amanaplanacanal 2y ago
    Maybe it is, maybe it isn't. The courts will decide.
    - WarOnPrivacy 2y ago
      > Maybe it is, maybe it isn't. The courts will decide. This offhandedly seems to dismiss the cost of achieving legal clarity for using a book - a cost that will far eclipse the cost of the book itself. In that light, it seems like an underweighted statement.
      - spencerflem 2y ago
        What they will decide is that it is simultaneously not piracy because it is not read by a human and not copyright infringement because its just like a human learning by reading a book
        icehawk 2y ago
        Those are both copyright infringement, sice we already have MAI Systems Corp. v. Peak Computer, Inc. I'd like to see them try to argue Cartoon Network, LP v. CSC Holdings, Inc. applies to their corpus.
bayindirh 2y ago
> The media ecosystem doesn't allow anything else. Uh, pardon? For a mere $10MM, you can get almost all of the Taylor & Francis' catalogue. They'll pressure their authors to finish their books early for free [0]. I think you can obtain all the training material for a mere rounding error in your books, if you're Meta, or Microsoft, or similar. Well, the authors will not be notified, compensated, or their idea on the matter won't be asked anyway, but this is "all for capit^H^H^H^H^H research". [0]: https://mathstodon.xyz/@johncarlosbaez/113221679747517432 https://mathstodon.xyz/@johncarlosbaez/113221679747517432
Manfred 2y ago
For my thesis I trained a classifier on text from internal messaging systems and forums from a large consultancy company. Most universities have had their own corpora to work with, for example: the Brown Corpus, the British National Corpus, and the Penn Treebank. Similar corpora exist for images and video, usually created in association with national broadcasting services. News video is particularly interesting because they usually contain closed captions, which allows for multi-modal training.
katamari-damacy 2y ago
Sam Altman bought some of GPT's training data from a Chinese army cyber group. 1. Sam Altman was removed from OpenAI due to his ties to a Chinese cyber army group. 2.OpenAI had been using data from D2 to train its AI models. 3. The Chinese government raised concerns about this arrangement with the Biden administration. 4. The NSA launched an investigation, which confirmed OpenAI's use of D2 data. 5. Satya Nadella ordered Altman's removal after being informed of the findings. 6. Altman refused to disclose this information to the OpenAI board. Source: https://www.teamblind.com/post/I-know-why-Sam-Altman-was-fired-Fr5cQ6Ne https://www.teamblind.com/post/I-know-why-Sam-Altman-was-fir... I guess Sam then hired top NSA guy to buy favor with the natsec community. I wonder who protects Sam up top and why aren't they protecting Zuck? Is Sam just better at bribes and manipulation?
- katamari-damacy 2y ago
  [dead]
xhkkffbf 2y ago
How do you feel about a business saying, "Paying people is hard. You should work for free."?
rhosseinzadeh 2y ago
There are some efforts to do fully open LLMs including their training data. Allen AI released their model (OLMo) and the data used for training the model under permissive licenses. https://allenai.org/ https://allenai.org/
notachatbot123 2y ago
> Buying millions of ebooks online would take a lot of effort Let me put that into perspective: - Googling "how many books exist" gives me ~150 million, no idea how accurate but let's use that. - Meta had a net profit of ~40 billion USD in 2023. - That could be an potential investment of ~250 USD per book acquisition. That sounds like a ludicrously high budget to me. So yeah, Meta could very well pay. It would still not be ethical to slurp up all that content into their slop machine but there is zero justification to pirate it all, with these kinds of money involved.
- rockemsockem 2y ago
  The number 150,000,000 is laughably small. Anyway the problem is not money it's technical feasibility and timelines. You clearly don't think LLMs have any value though, so W/E

the-rc 2y ago

Google has scans from Google Books, as well as all the ebooks it sells on the Play Store.

lemoncookiechip 2y ago
Wouldn't that still be piracy? They own the rights of distribution, but do they (or Amazon) have the rights to use said books for LLM training? And what rights would those even be?
- XorNot 2y ago
  Literally no rights agreement covers LLMs. They cover reproduction of the work, but LLMs don't obviously do this i.e. that the model transiently runs an algorithm over the text is superficially no different to the use of any other classifier or scoring system like those already used by law firms looking to sue people for sharing torrents.
  - deleted 2y ago
    [deleted]
  - thatcat 2y ago
    do those classifiers read copyrighted material? i thought they simply joined the swarm and seeded (reproduction with permission) youtube, etc classifiers definitely do read others material though.
  - visarga 2y ago
    > They cover reproduction of the work, but LLMs don't obviously do this LLMs are much smaller than their training sets, there is no space to memorize the training data. They might memorize small snippets but never full books. They are the worst infringement tools ever made - why replicate Harry Potter by LLM, it's show, expensive and lossy, when you could download the book so much easier. A second argument is that using the LLM blends a new intent into the process, that of the prompter. This can render the outputs transformative. And most LLM interactions are one-time use, like a scratch pad not like a finished work.
    - Earw0rm 2y ago
      The lossy compression argument is interesting. How many bits of entropy in Harry Potter? How many bits of entropy in a lossy-compressed abridgement that is nevertheless enough, when reconstituted, to constitute a copyright infringement of Harry Potter? The latter is absolutely small enough to fit in an LLM, although how close it would get to the original work is debatable. The question is whether copyright is violated: 1) inherently by the model operator, during the training. 2) by the model/model owner, as part of the generation. 3) by the user, in making the model so so and then reproducing the result.
      - dijksterhuis 2y ago
        my personal perspectives 1) straight up copying. download a bunch of copyrighted stuff -> making a copy. no way out of this one. 2) a derivative work can be/is being generated here. very grey area — what counts as a “derivative” work? read about robin thicke blurred lines court case for a rollercoaster of a time about derivative musical works. 3) making the model so so? do you mean getting an output and user copying the result? that’s copying the derivative work, which, depends on whatever copyright agreement happens once a derivative work claim is sorted out. that’s based on my 5 years of music copyright experience, although it was about ten years ago now so might be some stuff i’ve got wrong there.
        visarga 2y ago
        You can ensure a model trains on transformative not derivative synthetic texts, for example, by asking for summary, or turning it into QA pairs, or doing contrastive synthesis across multiple copyrighted works. This will ensure the resulting model will never regurgitate the training set because it has not seen it. This approach only takes abstract ideas from copyrighted sources, protecting their specific expression. If abstract ideas were protectable what would stop a LLM from learning not from the original source but from social commentary and follow up works? We can't ask people not to reproduce ideas they read about. But on the other hand, protecting abstractions would kneecap creativity both in humans and AI.
- majormajor 2y ago
  It means they have existing relationships/contacts to reach out to for negotiating the rights for other usages of that content. I think it negates (for the case of Google/Apple/Amazon who all sell ebooks) the claim made that efficiently acquiring the digital texts wouldn't be possible.
- brookst 2y ago
  It’s a good question. Textbook companies especially would be pretty enthusiastic about a new “right to learn” monetization strategy. And imagine how lucrative it would be if you could prove some major artist didn’t copy your work, but learned from your work. The whole chain of scientific and artistic development could be monetized in perpetuity. I think this is a dangerous road with little upside for anyone outside of IP aggregators.
- bawolff 2y ago
  > but do they (or Amazon) have the rights to use said books for LLM training? The real question is - does copyright grant the authors' the right to control if their work is used for LLM training? Its not obvious what the answer is. If authors don't have that right to begin with then there is no way amazon could buy it off them.
pdpi 2y ago
Leveraging their position in one market to get a leg up on another market? No idea if it would stick, but that would be one fun antitrust lawsuit right there.
- brookst 2y ago
  Fun fact: it’s only illegal to leverage a monopoly in one market to advance another. It’s perfectly legal for Coke to leverage their large but not monopolistic soft drink empire to advance their bottled water entries.
  - pdpi 2y ago
    Sure. The whole thing hinges on whether Google has a monopoly on whatever Google Books' market is (hence why I doubted it would stick). But given that some people seem to define "market" broadly enough to conclude that Apple has a monopoly on iPhones...

diggan 2y ago

> There's just no other viable alternative for efficiently acquiring large quantities of text data. [...] take a lot of effort [...] isn't a thing that can be done efficiently [...] only efficient way to access large volumes of media is piracy Hypothetical: If the only way we could build AGI would be to somehow read everyone's brain at least once, would it be worth just ignoring everyone's wish regarding privacy one time to suck up this data and have AGI moving forward?

nosbo 2y ago
no
BriggyDwiggs42 2y ago
Could this agi cure cancer, and would it be in the hands of the public? Then sure, otherwise nah.
- onemoresoop 2y ago
  > in the hand of the public Would you trust a businessman on that?
  - BriggyDwiggs42 2y ago
    Nope, they haven’t earned an ounce.
    - anonym29 2y ago
      How about a politician?
      - BriggyDwiggs42 2y ago
        Eh, a transparent organization of elected officials with short term limits and strong public oversight, a little bit. A very smart ai would represent power, and so every mechanism we’ve used as humans to guard against power misuse would be the ones I’d want to see here.

IncreasePosts 2y ago

Why would machine intelligence need an entire humanity's worth of data to be machine intelligence? It seems like only a training method that is really poor would need that much data.

aithrowawaycomm 2y ago

I find it highly implausible that Meta doesn't have the resources to obtain these legally. They could have reached out to a publisher and ask to purchase ebooks in bulk - and if that publisher says no, tough shit. The media ecosystem doesn't exist for Big Tech to extract value from it! "It would take a lot of effort to do it legally" is a pathetic excuse for a company of Meta's size.

gazchop 2y ago
Perhaps they did and got told no and decided to take it anyway? Defending themselves with technicalities and expensive lawyers may be financially viable. Zero ethics but what would we expect from them?
- XorNot 2y ago
  Who is "them"? Like, who in the Meta business reporting line made this decision, then how did they communicate it to the engineers who would've been necessary to implement it, particularly at scale? While it's plausible someone downloaded a bunch of torrents and tossed them in the training directory...again, under who's authority? Like if this happened it would be one overzealous data scientist potentially. Hardly "them". People lean on collective pronouns to avoid actually thinking about the mechanics of human enterprise and you get extremely absurd conclusions. (it is not outside the bounds of thinkable that an org could in fact have a very bad culture like this, but I know people who work for Meta, who also have law degrees - they're well aware of the potential problems).
  - aithrowawaycomm 2y ago
    Come on... it's fine that you haven't followed the story, there's a lot going on, but the snotty condescension is very frustrating: These newly unredacted documents reveal exchanges between Meta employees unearthed in the discovery process, like a Meta engineer telling a colleague that they hesitated to access LibGen data because “torrenting from a [Meta-owned] corporate laptop doesn’t feel right ”. They also allege that internal discussions about using LibGen data were escalated to Meta CEO Mark Zuckerberg (referred to as "MZ" in the memo handed over during discovery) and that Meta's AI team was "approved to use" the pirated material. https://www.wired.com/story/new-documents-unredacted-meta-copyright-ai-lawsuit/ https://www.wired.com/story/new-documents-unredacted-meta-co...

ben_w 2y ago

IMO, if the AI were more sample-efficient (a long-standing problem that predates LLMs), they would be able to learn from purely open-licensed content, which I think Wikipedia (CC-BY-SA) would be an example of? I think they'd even pass the share-alike requirements, given Meta are giving away the model weights? https://en.wikipedia.org/wiki/Wikipedia:Copyrights https://en.wikipedia.org/wiki/Wikipedia:Copyrights

wizzwizz4 2y ago
Since this is Wikipedia, it could even satisfy the attribution requirements (though most CC-licensed corpora require attributing the individual authors).
visarga 2y ago
Alteratively if they trained the model on synthetic data, filtered to avoid duplication, then no copyrighted material would be seen by the model. For example turn an article into QA pairs, or summarize across multiple sources of text.
- thesz 2y ago
  > trained the model on synthetic data You get knowledge collapse [1] this way. [1] https://arxiv.org/abs/2404.03502 https://arxiv.org/abs/2404.03502
  - ben_w 2y ago
    You can get knowledge collapse. But even though there are counter-examples (e.g. learning Go from self-play based only on the rules), IMO — based on how often people were already doing this with buggy human-written software[0][1] — most people don't think about this in the right way and will therefore treat these things as magical oracles when they shouldn't. No silver bullets. [0] https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal [1] https://en.wikipedia.org/wiki/Computer_says_no https://en.wikipedia.org/wiki/Computer_says_no

mvdtnz 2y ago

AI mega corporations are not entitled to easy and cheap access to data they don't own. If it's a hard problem, too bad. If the stakes are as high as they're all claiming then it should be no problem for them to do this right.

visarga 2y ago
> not entitled to easy and cheap access to data they don't own This is not copyright as we know it. Copyright protects against copying, not accessing data. You can still compile statistics off data you don't own. The models are like a compressed version of the originals, so compressed you can't retrieve more than a few snippets of original text. Newer model train on filtered synthetic text, which is one step removed from the protected expression in the copyrighted works. Should abstractions be protected by copyright?
- svl7 2y ago
  However in order to get to the compressed state, the original data would have to be processed in some way as a whole. This would require a copy of the material to be available. In case that copy was attained in an illegal way, what are the implications?

gunian 2y ago

what about something decentralized? each person trains someone on their own piece of data and somehow that gets aggrgegated into one giant model

techwizrd 2y ago
This approach is used in Federated Learning where participants want to collaboratively train a model without sharing raw training data.
- gunian 2y ago
  are there any companies working on it? was thinking if i train my model on my private docs for instance finance how does one prevent the model from sharing that data verbatim

nh2 2y ago

> Buying millions of ebooks online would take a lot of effort I don't understand. Facebook and Google spend billions on training LLMs. Buying 1M ebooks at $50 each would only cost $50M. They also have >100k engineers. If they shard the ebook buying across their workforce, everyone has to buy 10 ebooks, which will be done in 10 minutes.

shakna 2y ago
Google also operates a book store, like Amazon. Both could process a one-off to pay their authors, and then draw from their own backend.

maeil 2y ago

> In the most recent fiscal year, Alphabet's net income amounted to 73.7 billion U.S. dollars Absolutely no way. Yup. > Buying millions of ebooks online would take a lot of effort, downloading data from publishers isn't a thing that can be done efficiently Oh no, it takes effort and can't be done efficiently, poor Google! How can this possibly be an excuse? This is such a detached SV Zuckerberg "move fast and break things"-like take. There's just no way for a lot of people to efficiently get out of poverty without kidnapping and ransoming someone, it would take a lot of effort.

thatcat 2y ago
copyright piracy isn't theft, try proving damages for a better arguement
- maeil 2y ago
  Not my point, never said it is. Substitute that example with another criminal act. Edit: Changed it just for you
  - thatcat 2y ago
    copyright infringement is a civil charge, silly guy. no offense, but there arent many ways to defend its existence in current form without resorting to hyperbolic nonsense and looking silly in the process. so its not a 'crime' and you have to prove damages for a civil offense so... youd need to prove that ai caused damages or how that is materially different from other algorithms like google scanning documents to provide the core utility for their service

brookst 2y ago

I don’t follow the “millions of ebooks are hard” line of thinking. If Meta (or anyone) had approached publishers with a “we want to buy one copy of every book you publish”, that doesn’t seem technical or business difficult. Certainly Amazon would find that extremely easy.

spencerflem 2y ago
Buying a book to read and incorporating their text in a product are two different things. Even if they bought the book, imo it would be illegal.
- amanaplanacanal 2y ago
  Maybe it is, maybe it isn't. The courts will decide.
  - WarOnPrivacy 2y ago
    > Maybe it is, maybe it isn't. The courts will decide. This offhandedly seems to dismiss the cost of achieving legal clarity for using a book - a cost that will far eclipse the cost of the book itself. In that light, it seems like an underweighted statement.
    - spencerflem 2y ago
      What they will decide is that it is simultaneously not piracy because it is not read by a human and not copyright infringement because its just like a human learning by reading a book
      - icehawk 2y ago
        Those are both copyright infringement, sice we already have MAI Systems Corp. v. Peak Computer, Inc. I'd like to see them try to argue Cartoon Network, LP v. CSC Holdings, Inc. applies to their corpus.

bayindirh 2y ago

> The media ecosystem doesn't allow anything else. Uh, pardon? For a mere $10MM, you can get almost all of the Taylor & Francis' catalogue. They'll pressure their authors to finish their books early for free [0]. I think you can obtain all the training material for a mere rounding error in your books, if you're Meta, or Microsoft, or similar. Well, the authors will not be notified, compensated, or their idea on the matter won't be asked anyway, but this is "all for capit^H^H^H^H^H research". [0]: https://mathstodon.xyz/@johncarlosbaez/113221679747517432 https://mathstodon.xyz/@johncarlosbaez/113221679747517432

Manfred 2y ago

For my thesis I trained a classifier on text from internal messaging systems and forums from a large consultancy company. Most universities have had their own corpora to work with, for example: the Brown Corpus, the British National Corpus, and the Penn Treebank. Similar corpora exist for images and video, usually created in association with national broadcasting services. News video is particularly interesting because they usually contain closed captions, which allows for multi-modal training.

katamari-damacy 2y ago

Sam Altman bought some of GPT's training data from a Chinese army cyber group. 1. Sam Altman was removed from OpenAI due to his ties to a Chinese cyber army group. 2.OpenAI had been using data from D2 to train its AI models. 3. The Chinese government raised concerns about this arrangement with the Biden administration. 4. The NSA launched an investigation, which confirmed OpenAI's use of D2 data. 5. Satya Nadella ordered Altman's removal after being informed of the findings. 6. Altman refused to disclose this information to the OpenAI board. Source: https://www.teamblind.com/post/I-know-why-Sam-Altman-was-fired-Fr5cQ6Ne https://www.teamblind.com/post/I-know-why-Sam-Altman-was-fir... I guess Sam then hired top NSA guy to buy favor with the natsec community. I wonder who protects Sam up top and why aren't they protecting Zuck? Is Sam just better at bribes and manipulation?

katamari-damacy 2y ago
[dead]

xhkkffbf 2y ago

How do you feel about a business saying, "Paying people is hard. You should work for free."?

rhosseinzadeh 2y ago

There are some efforts to do fully open LLMs including their training data. Allen AI released their model (OLMo) and the data used for training the model under permissive licenses. https://allenai.org/ https://allenai.org/

notachatbot123 2y ago

> Buying millions of ebooks online would take a lot of effort Let me put that into perspective: - Googling "how many books exist" gives me ~150 million, no idea how accurate but let's use that. - Meta had a net profit of ~40 billion USD in 2023. - That could be an potential investment of ~250 USD per book acquisition. That sounds like a ludicrously high budget to me. So yeah, Meta could very well pay. It would still not be ethical to slurp up all that content into their slop machine but there is zero justification to pirate it all, with these kinds of money involved.

rockemsockem 2y ago
The number 150,000,000 is laughably small. Anyway the problem is not money it's technical feasibility and timelines. You clearly don't think LLMs have any value though, so W/E