4 ms·
> I would suggest adding a stricter robots.txt or perhaps blocking some of these bots that are scraping your site. The only people who care about robots.txt ar
by aaron695 11y ago
> I would suggest adding a stricter robots.txt or perhaps blocking some of these bots that are scraping your site.
The only people who care about robots.txt are some of the big companies. Even Baidu ignores it (As they can, it's purely there as etiquette)
Blocking bots is hard.