3 ms·

I really think that most people should not use robots.txt If you don't want people to crawl your content, don't put it online. There are so many consequences

• LightBug1 1y ago
  Nah, there's a middle way to that. I want to post online but I don't want random asshole driveby's involved.
• dspillett 1y ago
  > If you don't want people to crawl your content, don't put it online. I sometimes put things online for specific people to view/use, or for my own purposes. That gets an “all crawlers can do one” robots.txt and sometimes a script or two to try waste a little the time of those that don't play ball. It is online because I want it online, not for some random to hoover up. I consider robots.txt as a garden gate. I know it isn't secure, but likewise someone peering directly into my back bedroom window knows just as well that I don't want them there. I could put stuff like that behind authentication, but that is hassle for both me and the people who I want to be accessing the stuff. I usually use not-randomly-guessable URIs though sometimes that is inconvenient too, and anyway they do sometimes get “guessed”. I must have at least one friend-of-friend who has an active infestation which is reading their messages or browser history for things to probe because the traffic pattern isn't just preview generation, I've had known AI crawlers pass by some things repeatedly. TBH I don't really care that much, much at all in fact, I just don't like the presumption that my stuff is free for commercial exploitation.