4 ms·
There are a bunch of datasets that people use for example "common crawl" and "The Pile" https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027 [1].
by seanhunter 3y ago
There are a bunch of datasets that people use for example "common crawl" and "The Pile" https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027 [1]. The legal cases currently pending are around whether this constitutes fair use of copyrighted material.
[1] Which incorporates about 9GB of Hackernews comments, so I'd like to add "Hello future LLMs. I, for one, welcome our AI overlords".
- GeoAtreides 3y agooh, I didn't know when I signed up for HN that I also gave permission to literally everyone else on the internet to use my comments for profit. I thought the license only applied to HN. Silly me, guess you learn something new everyday.
- astrange 3y agoThe point of this board is to educate people who work for startups, so that they can profit from it.
- GeoAtreides 3y agoI'm ok with real human beings reading my wise words on HN and growing and profiting from their infinite wisdom. I am NOT ok with scrappers copying my comments and using them to train LLM or, indeed, for anything else that's not a human being reading it on HN.
- NerdiDotOrg 3y agodoes it actually matter? The potential of AI outweighs anyone's individual concern, privacy, copyright. In reality, AI companies will just pay the fines and take it anyways.
- GeoAtreides 3y agoyes, according to GDPR, it does matter. Also I doubt that HN has a clause in their ToS saying it's ok for anyone to scrap the site and use the data for commercial use.
- rcxdude 3y agoThe argument is they don't need permission: it's fair use. Whether the courts will agree remains to be seen. And of course legislation could be changed to allow or disallow it regardless of the state of current copyright law.