5 ms·
Doesn't it mostly penalise the honest bots, though? Or do some bots use robots.txt files as a proxy sitemap? I'm not that familiar with this area.
by samhw 5y ago
Doesn't it mostly penalise the honest bots, though? Or do some bots use robots.txt files as a proxy sitemap? I'm not that familiar with this area.
- cookiengineer 5y agoYou can also insert a Disallow: /wp-admin/ in there and just serve the gzip bomb then :D
- berkes 5y agoI have anyone requesting this automatic going to fail2ban¹, but serving a bomb at that url is actually much smarter. ¹anything .php,.asp and some other tech we never use, also phpmyadmin, cgi-bin, and some more, actually.
- hughrr 5y agoActually now you mention that I’m going to see if I can set a trap for our pen testers :)
- spookthesunset 5y agoDamn, that is a good idea. I wish I thought of that back when I ran a web forum.
- berkes 5y agoThere are fail2ban configs for most popular software ready to use. E.g. WP fail2ban¹ is a config that bans an IP after failed login attempts. The defaults it ships with are well-balanced between security and not-locking-yourself-out. Drupal, PHPBB, etc all have such generic config defaults online. For frameworks such as Rails, Django etc its a bit harder because the logging and url-schemes are not standard, but defined by the person developing in that framework. ¹https://wordpress.org/plugins/wp-fail2ban/ https://wordpress.org/plugins/wp-fail2ban/
- Shadonototra 5y agoi hope you don't ban the IPs permanently nobody own their IP, and at some point a legitimate person will be assigned that IP and most people who do bad things on the internet use some proxies, usually residential ones it always makes me sad when people put such rules
- nixgeek 5y agoMost crawlers aren’t going to come from ASN which are populated with eyeballs.
- berkes 5y agoFail2ban, by default, bans an IP for a limited time. For me that's 20mins. This means that commonly only about 30 IPs are on the banlist. The chance that anyone using the same IP as a bot who is visiting any of the apps and sites on that server is not 0%, but tiny. The only goal is to stop bots and worms from hammering my server. It hardly improves security; it shouldn't anyway.
- alfiedotwtf 5y agoLOL that's actually really neat
- comonoid 5y agoSane implementations do not read more than 500KiB of robots.txt contents.