3 ms·
They do! If you create a "robots.txt" file [1] and use a disallow flag, they will not scan the site. Ex: http://web.archive.org/web/20070103112847/http://www.i
by Qub3d 10y ago
They do! If you create a "robots.txt" file [1] and use a disallow flag, they will not scan the site.
Ex: http://web.archive.org/web/20070103112847/http://www.infoceptor.com/ http://web.archive.org/web/20070103112847/http://www.infocep...
[1]: https://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard
- nimchimpsky 10y agolol, its always extra amusing when someones sarcastic, negative, unfunny, and smug comment is immediately disproved.
- drivingmenuts 10y agoYep, my bad on that one. I was thinking of archiveteam.org.