3 ms·
But the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and
by soco 2y ago
But the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and google, from malicious crawlers?
- delichon 2y agoYes, please explain. How does an entry in robots.txt distinguish humans from bots that ignore it?
- voidUpdate 2y agoWhen was the last time you looked at robots.txt to find a page that wasn't linked anywhere else?
- sharlos201068 2y agoCrawlers aren't interested in fake pages that aren't linked to anywhere, they're crawling the same pages your users are viewing.
- danielheath 2y agoAdding a disallowed url to your robots.txt is a quick way to get a ton of crawlers to hit it, without linking to it from anywhere. Try it sometime.
- brookst 2y agorobots.txt is not a sitemap. If it worked that way you could just make a 5TB file linking to a billion pages that look like static links but are dynamically generated.
- ccgreg 2y agorobots.txt has a maximum relevant size of 500 kib.
- gadflyinyoureye 2y agoTuesday. But I have odd hobbies.
- zzo38computer 2y agoIt was a while ago, and it was not deliberate (wget downloaded robots.txt as well as the files I requested, and I was able to find many other files due to that, some of which could not be accessed due to requiring a password, but some were interesting (although I did not use wget to copy those other files; I only wanted to copy the files I originally requested)).
- majewsky 2y agorobots.txt is a somewhat useful tool for keeping search engines in line, because it's rather easy to prove that a search engine ignores robots.txt: when a noindex page shows up in SERPs. This evidence trail does not exist for AI crawlers.
- ccgreg 2y agoI'd say a bigger problem is that people disagree about the meaning of nofollow and noindex.
- sigmoid10 2y agoThe detection and bypass is trivial: Access the site from two IPs, one disrespecting robots.txt. If the content changes, you know it's garbage.