3 ms·

The internet/web was founded on permission-less, innovation, permissiveness, and openness. People were invited to share content, to aggregate, to link, and reli

The internet/web was founded on permission-less, innovation, permissiveness, and openness. People were invited to share content, to aggregate, to link, and relink. 99% of the people reading this now, who learned to code on the Web, learned by copying other people's code snippets, and not paying them a dime. Scraping content isn't a bug, it's a feature. View Source is a the primary feature of published Web content. Content scraping with transformation, is fair use, and a feature. IMHO, if you publish something on the web, and you don't have a paywall, and you don't block in robots.txt, and you don't have a LICENSE that covers use, people are free to link to your content. Otherwise, why even participate in the web when you don't want to be part of it (the linking). If people start charging for links, you may as well pack up the open web. If there was an federated free search engine, like bitcoin-like for search, these news publishers would still be in trouble, and they'd have no one go after, which suggests the problem is fundamentally, something with their business model.

ocdtrekkie 6y ago
Ironically, Google seems to feel scraper sites that aren't Google are bad. For instance, https://developers.google.com/search/docs/advanced/guidelines/scraped-content https://developers.google.com/search/docs/advanced/guideline... Google has regularly defended deranking or banning competitors for allegedly scraping content without providing value as well.
- cromwellian 6y ago
  They're talking about purely scraped content without anything added, like someone mirroring another site, wholesale copying without any change. "Purely scraped content, even from high-quality sources, may not provide any added value to your users without additional useful services or content provided by your site;" This is pretty easy to understand and I don't think controversial. Scrape your site and transform some of the information to make a useful service? Yes. Scrape your whole site and just host its pages? No. Maybe your site is a medical blog and over the years has written about pretty much every condition. If someone else were to scrape this, and make an interactive map of the human body, where if you click on a part, it lists links to the original blog, plus a pop-up summative snippet, very few people are going to scream "You did nothing! This guy spent years writing medical articles, and you come along and make a fancy interactive body as a way to linking to his articles, and now you're getting ad revenue and traffic!" Obviously, if lots of people are using this new interactive service, it's a value add. But what if instead, I setup a site like WebMD, supposedly with expertly written medical content from the Web's best medical bloggers, but I just scrape all their articles, and host those pages on my own site, as if they had written the articles as contributors to my site. And you don't think the latter should be downranked? Do you think GPT-3 should somehow pay someone everytime it generates an answer? What if future search engines crawl the web and build giant transformer models which just answer questions by whole-sale authoring a new article from scratch, how do we compare that to a human being doing the same thing? Scraping is how human beings learn and there's a big difference between wholesale piracy, and transformation.

ocdtrekkie 6y ago

Ironically, Google seems to feel scraper sites that aren't Google are bad. For instance, https://developers.google.com/search/docs/advanced/guidelines/scraped-content https://developers.google.com/search/docs/advanced/guideline... Google has regularly defended deranking or banning competitors for allegedly scraping content without providing value as well.

cromwellian 6y ago
They're talking about purely scraped content without anything added, like someone mirroring another site, wholesale copying without any change. "Purely scraped content, even from high-quality sources, may not provide any added value to your users without additional useful services or content provided by your site;" This is pretty easy to understand and I don't think controversial. Scrape your site and transform some of the information to make a useful service? Yes. Scrape your whole site and just host its pages? No. Maybe your site is a medical blog and over the years has written about pretty much every condition. If someone else were to scrape this, and make an interactive map of the human body, where if you click on a part, it lists links to the original blog, plus a pop-up summative snippet, very few people are going to scream "You did nothing! This guy spent years writing medical articles, and you come along and make a fancy interactive body as a way to linking to his articles, and now you're getting ad revenue and traffic!" Obviously, if lots of people are using this new interactive service, it's a value add. But what if instead, I setup a site like WebMD, supposedly with expertly written medical content from the Web's best medical bloggers, but I just scrape all their articles, and host those pages on my own site, as if they had written the articles as contributors to my site. And you don't think the latter should be downranked? Do you think GPT-3 should somehow pay someone everytime it generates an answer? What if future search engines crawl the web and build giant transformer models which just answer questions by whole-sale authoring a new article from scratch, how do we compare that to a human being doing the same thing? Scraping is how human beings learn and there's a big difference between wholesale piracy, and transformation.