3 ms·
That is a brilliant idea! I should use that in all my robots.txt
by Swizec 15y ago
That is a brilliant idea!
I should use that in all my robots.txt
- ck2 15y agoIf you want to take it to the next level, embed it as a microdot (single pixel) link somewhere on a side page, and use css to make it display:none;visibility:hidden;margin:-9999px; so that no human could possibly click on it. Because of robots.txt, no bot should touch that link. And because it's off the page and hidden, no human either. But you'll be amazed how much it's hit. But a warning: the problem with today's browsers is prefetching. You'd have to make sure you disallow it for prefetching too or you'll trap innocent humans using hyperactive browsers. This technique also will catch mass downloading plugins where they are saving your entire website from a browser plugin - but they can die too in my book. If you want to be nice, write the banlist to a file you clear out ever hour via cron or with a time check.