2 ms·
News industry is struggling but scraping public web content is legal as far as it is not commercially used....at least that's my layman understanding (but Perpl
by mrkramer 2y ago
News industry is struggling but scraping public web content is legal as far as it is not commercially used....at least that's my layman understanding (but Perplexity is using it commercially). The solution for news industry with regards to LLMs is licensing your news and content to LLMs products because they are going to keep scraping it anyways.
- jaredwiener 2y ago"You might as well make a deal with the people stealing your content because they are going to keep stealing it anyway" is not a great option. Part of licensing is the implicit understanding that if the parties do not come to a deal, they cannot use the licensed content. It seems from this article that Perplexity is not operating in good faith, ignoring robots.txt, using unpublished IP addresses, and then lying about the amount of referral traffic generated. Not to mention, whether or not its legal (and im not a lawyer) doesn't seem to be the issue. It's being so shortsighted that you build a business that steals a product from an unwitting third party, and when that third party is drained of resources, where will you pull from?
- mrkramer 2y ago>ignoring robots.txt Robots.txt is not legally binding, it is a good will among web stakeholders to respect it. But if you are repeatedly breaking it, you will have an army of web masters against you. And as far as I remember, big news outlets are already suing LLMs companies or are already licensing their content to them.
- jaredwiener 2y agoYes to all of your points, but I think you are missing my distinction between "potentially legal" and "not being terrible." And yes, of course they are suing the LLM companies. Because they shouldn't be stealing their stuff, regardless of robots.txt.