4 ms·
Not sure it's a good idea to block all search engines but Google.
by muyuu 12y ago
Not sure it's a good idea to block all search engines but Google.
- yhager 12y agoProbably not, but if they don't want to be scraped, the first place they should notify about this is robots.txt. I was just stating an example that you can allow some bots and not others if you like. Of course, forging your user agent, disobeying robots.txt or scraping after you were told not to, is wrong ethically.