3 ms·
Search engines use scraping. I would guess that most websites want to be featured on the search engines?
by BlueTemplar 6y ago
Search engines use scraping. I would guess that most websites want to be featured on the search engines?
- jchw 6y agoNot all websites want to be scraped even by search engines, and some only want to be scraped by search engines. Unwanted scrapers may refuse to follow robots.txt or do things with the scraped data that is either undesired to their business model or possibly even illegal, though that is complex. The point is, for better or worse, as far as I know, websites have the right to block scrapers while allowing search engines, and as long as it remains viable to prevent scraping, it remains likely that sites intending to do this will end up blocking or obstructing Tor traffic. Is it a good thing? I would argue it’s too abstract to judge in such a generalized fashion. Whether or not it is a good thing or a bad thing, though, is not important. It is the world we live in right now.
- colinmhayes 6y agoI wrote a search engine in school. Since I was naive/lazy/wanted a large corpus it didn't follow robots.txt. It ended up DOSing multiple sites, including Duke's law school during class sign up. I don't think those sites wanted to be featured on my search engine.