fnulp

4 ms·

1. 1.▲
   by fnulp 14y ago

   What? I simply meant that if some brainiacs think robots.txt can just be disregarded, it's time to make it a minimum requirement of every self-respecting webmaster to make a tarpit (disallowed in robots.txt) and ban any and all bots going t
2. 2.▲
   by fnulp 14y ago

   "before swearing at me." Followed by shitty" and "web crawlers" in all caps ^^ Someone ate a clown for breakfast I see. "No, we are not going to ban all the links from GitHub" What? Slow down there -- why would you care about invalid links
3. 3.▲
   by fnulp 14y ago

   "I wouldn't recommend having the "pretty" part not validated. It can cause some serious issues with google & duplicate content" Not if you properly generate and apply canonical links :)
4. 4.▲
   by fnulp 14y ago

   "just ignore robots.txt?" how about "fuck you"? I guess it's high time to make honeypots, tarpits and bans common practice.