4 ms·
I wish every company which crawls had such a page. Maybe a centralized directory of crawlers would be a good for the internet.
by qwertox 3y ago
I wish every company which crawls had such a page. Maybe a centralized directory of crawlers would be a good for the internet.
- Gasp0de 3y agoI mean, technically you can just disallow all robots in your robots.txt and only allow those of which you are aware and you like?