4 ms·
They didn't care for any "usage policy" when scraping whole internet and whatever else available for the purpose of building their product
by pxtail 4y ago
They didn't care for any "usage policy" when scraping whole internet and whatever else available for the purpose of building their product
- malborodog 4y agoVery similar to google's behavior with google scholar (and pretty much everything else). They built up according to one rule, but then we can't do the same thing to them. Funny that.
- PUSH_AX 4y agoWell they used common crawl which respects individual site policies such as robots.txt. https://commoncrawl.org/big-picture/frequently-asked-questions/ https://commoncrawl.org/big-picture/frequently-asked-questio... So I think that's nonsense
- amelius 4y agoI don't think that's true. For example, someone copying the web (but still respecting robots.txt) and claiming it's their own would be problematic surely? This is only one small step away from blatantly copying.
- PUSH_AX 4y agoWait, are we still talking about usage policies? Seems like a different discussion
- amelius 4y agoWell, I think the point is that robots.txt != usage_policy We probably need a license.txt convention to resolve this.
- PUSH_AX 4y agoWould it also apply to humans that read and remember the content?
- amelius 4y agoThat depends on the wording in license.txt. In general, automated systems and humans are considered different by the law. For example: looking out the window and noticing that your neighbor is going to the shop is ok; building an automated system that tracks people everywhere: not ok.
- PUSH_AX 4y agoAgreed, but it sounded like your initial concern was: > someone copying the web and claiming it's their own would be problematic surely? Is it ok for humans to repurpose, repackage and regurgitate knowledge they've "scraped" and not software?
- amelius 4y agoI'm not a judge/lawyer but that might depend on the scale on which it is happening.
- pxtail 4y ago> they used common crawl which respects individual site policies such as robots.txt Nice, so what? I seriously doubt that authors of the content crawled by CommonCrawl agreed to terms that their content will be used by openai. Moreover CC seems to be opt-in by default according to their faq: > You configure your robots.txt file which uses the Robots Exclusion Protocol to block the crawler. Our bot’s Exclusion User-Agent string is: CCBot. Add these lines to your robots.txt file and our crawler will stop crawling your website Again, I doubt that plenty of people are putting CC specific rules into their robots.txt, moreover I'm not naive and I doubt that in our reality where "move fast and break things" is THE motto for any big corpo any rules like that are respected at all.
- PUSH_AX 4y agoI mean even if you really want to believe they’ve broken every rule in the book, the concept of two wrongs making a right is still a poor take.
- danielbln 4y agoBe that as it may, they still might lock down your account.