7 ms·
It's not that bad. First, of course, you have cloudflare and recaptcha, which are free and very efficient, as the author say. But even if you don't want to us
by BiteCode_dev 4y ago
It's not that bad.
First, of course, you have cloudflare and recaptcha, which are free and very efficient, as the author say.
But even if you don't want to use them (some of my services don't), most bots are very dumb:
- require JS, and you lose half of the web ones
- silly tricks like hidden input fields in forms that worked in 2000 still work in 2022. Use a bunch of them, and you can yet again halve the bot traffic.
- many URL should have impossible to guess paths. E.G: just changing the /admin/ url to a uuid in django or the /wp-admin/ in wordpress, you save so many requests.
- bots are usually not tailored to your site, meaning if you require JS, you can actually embed anti-bot measure in the client code and they will work. E.G: exponential backoff + some heavy calculations if too many fast consecutive ajax requests.
- fail2ban + a few iptables rules (mitigate syn flood, etc) will help
- varnish + redis gets you very far to shave excess dummy traffic
It's not great, but it's not an apocalypse.
Unless you are under targeted attack.
Then it sucks and you die.
- efitz 4y agoAlso, attackers are rarely going to try to guess your URLs - they’re going to find them via Google or Shodan, or, if you’re a good rest citizen, via “/<yourapp>/“
- bryanrasmussen 4y ago>Also, attackers are rarely going to try to guess your URLs - because then the attack becomes DOS as they cycle through dictionaries of words?
- raverbashing 4y agoAny website gets probed for wp-admin.php etc, even if you don't use WP
- BiteCode_dev 4y agoIn fact, if someone is probing for wp-admin, you should insta ban them, no matter the site.
- marginalia_nu 4y agoI get quite a lot of guesses in my logs, probing in /solr/ and so on (fairly pointlessly, I might add, as I run bespoke software).
- efitz 4y agoNo, then you hide behind CloudFlare, because only the CSPs and network operators have the infrastructure to deal with volumetric attacks.
- sparkling 4y agoVery nice list of countermeasures. I agree that doing these small things like hidden input fields really go a long way. I would add to that: - block signups/comments from known throwaway email domains - block known datacenter IP ranges, at least for POST requests. Honestly on our sites 50% of spam was coming from AWS EC2 IPs - use a proxy/vpn/bot detection service like https://focsec.com https://focsec.com
- emptyparadise 4y agoBut then you end up forcing people to use Gmail.
- EdwardDiego 4y agoYup, in adtech, "IP is an AWS block" was a bot 99.999% of the time. The 0.001% was that person using EC2 as a proxy or VPN server.
- mobilio 4y agoIt's not only AWS. Also happens on Azure and GCP.
- EdwardDiego 4y agoTrue, but at the time, 3 - 4 years ago, Azure and GCP IPs were minimal. Guess the fraudsters were vendor locked lol.
- Zak 4y agoPlease don't make blocking VPNs plan A. Between snooping ISPs and public wifi networks that have indiscriminate content filters, I'm on a VPN about half the time. Many other legitimate users are as well.
- account42 4y ago> block signups/comments from known throwaway email domains Throwaway email services have caught up to the point where even bigger players don't manage to block them, i.e. they rotate the domains used for the emails.
- BasiliusCarver 4y agoOne of the things I’ve done before among the other suggestions is to put a hidden link like /followmeifyouscraping.html in the landing page to get a bit of info about scraping volume and then you can use fail2ban filters to block if it’s visited if you want
- jiggawatts 4y agoI would add that link to robots.txt as an exclusion. That way well-behaved search engines won’t be affected, but naive scrapers get auto-banned.
- SyneRyder 4y ago> First, of course, you have cloudflare and recaptcha, which are free and very efficient, as the author say. Recaptcha has been almost useless, in my experience. If you read the spam logs, you'll quickly learn about the spam software they (claim to) use to bypass Recaptcha, because that's what they end up promoting. I started tagging in logs if Recaptcha had validated on messges, and sure enough these spam posts had all successfully passed it. Great opportunity to rip out more Google dependencies from my website. I've found my own custom written filters to be vastly more effective than Recaptcha. Lots of the bots are running full Chrome with JS, lots of HeadlessChrome being used lately. The fact that they're using HeadlessChrome is something that makes them easy to detect, ahem.
- BiteCode_dev 4y agoThose are very specific bots, recaptch will stop a lot of casual ones. Most of them in fact.
- SyneRyder 4y agoThat really hasn't been my experience, perhaps I'm just getting hit more by the sophisticated bots than the naive ones. I'm glad that it works for some people. Recaptcha was also filtering out some legit humans (I logged all posts regardless of captcha status to be reviewed later), so it just wasn't worth reducing the user experience when the captcha bot detection rate was so low.