3 ms·
Right?! robots.txt is basically a map to the inner secret parts of your website, the "real" content. Robots.txt is basically treated as "these rules are for the
by taftster 2y ago
Right?! robots.txt is basically a map to the inner secret parts of your website, the "real" content. Robots.txt is basically treated as "these rules are for thee, not for me" access control.
If you don't want your content crawled, you really need to put it behind a login of some sort (and watch it still get crawled regardless) or just not publish it at all. Sad state we're in.
- scoofy 2y agoI'm getting close to a building a honey pot, because I want responsible robots to be able to crawl my site! It's about sharing information after all. If it gets large enough to have a positive monetary benefit to me, I'd happily let the ai bots crawl it responsibly too. It's a wiki, so I don't want it to me my information, ever. I'm just the one footing the bill right now.
- taftster 2y agoYeah, I used to do that. I'd put bad actors into a randomly generated never ending link farm. I'd put a special link in my robots.txt and any bot that visited it, I would assume was a bad actor. This link was not accessible from any other location. The results were striking, how many bots just stupidly tumbled down the random link rabbit hole. I'm not sure what they were searching for, but all they got from me was gobbledygook text and more links.