4 ms·
This relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's
by sigmoid10 2y ago
This relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's creators. Distinguishing genuine traffic has always been hard and it will not get easier in the age of AI.
- hec126 2y agoYou can sprinkle your site with almost-invisible hyperlinks. Bots will follow, humans will not.
- rustc 2y agoThis would be terrible for accessibility for users using a screen reader.
- mostlysimilar 2y agoSo would the site shutting down because AI bots are too much traffic.
- MrResearcher 2y ago<a aria-hidden="true"> ... </a> will result in a link ignored by screen readers.
- rustc 2y agoRemoving elements that match `[hidden], [aria-hidden]` is the most trivial cleanup transform a crawler can do and I'm sure most crawlers already do that.
- soco 2y agoBut the very comment you answered explains how to do it: a page forbidden in robots.txt. Does this method need explanation why it's ideal for sorting humans and google, from malicious crawlers?
- delichon 2y agoYes, please explain. How does an entry in robots.txt distinguish humans from bots that ignore it?
- voidUpdate 2y agoWhen was the last time you looked at robots.txt to find a page that wasn't linked anywhere else?
- sharlos201068 2y agoCrawlers aren't interested in fake pages that aren't linked to anywhere, they're crawling the same pages your users are viewing.
- danielheath 2y agoAdding a disallowed url to your robots.txt is a quick way to get a ton of crawlers to hit it, without linking to it from anywhere. Try it sometime.
- brookst 2y agorobots.txt is not a sitemap. If it worked that way you could just make a 5TB file linking to a billion pages that look like static links but are dynamically generated.
- ccgreg 2y agorobots.txt has a maximum relevant size of 500 kib.
- gadflyinyoureye 2y agoTuesday. But I have odd hobbies.
- zzo38computer 2y agoIt was a while ago, and it was not deliberate (wget downloaded robots.txt as well as the files I requested, and I was able to find many other files due to that, some of which could not be accessed due to requiring a password, but some were interesting (although I did not use wget to copy those other files; I only wanted to copy the files I originally requested)).