7 ms·

How to Block Some of the Bots

133 points by Bender 3mo ago

binaryturtle 3mo ago
410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks! Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)
- Bender 3mo ago
  Real browsers send it. [1] Some of the reader apps do not. Most bots aside from those utilizing Chrome Headless do not. People can see a few headers here [2] [1] - https://caniuse.com/?search=sec-fetch-mode https://caniuse.com/?search=sec-fetch-mode [2] - https://nochan.net/.env https://nochan.net/.env
- gruez 3mo ago
  I didn't even make it that far, I got PR_END_OF_FILE_ERROR which indicates it couldn't even get past the tls handshake.
  - ErroneousBosh 3mo ago
    I mean that just might be the dying gasp of a puddle of scorched fibreglass and boiling copper where the server used to be.
    - Bender 3mo ago
      Mostly harmless(c) load average: 0.00, 0.01, 0.00
fxtentacle 3mo ago
I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.
- Bender 3mo ago
  When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being all over the internet.
  - inigyou 3mo ago
    Oof, Germany. That's a tough place to be doing cybercrime from. If you told the police, I guarantee they can get that guy's home address. Whether they'd bother is a different story
  - charcircuit 3mo ago
    >to figure out which records to avoid It is more likely they were looking for a DNS server with a big response packet for use in amplified DDoS attacks.
    - Bender 3mo ago
      No they were specifically trying and failing of course to AXFR from the only domains I had added the zeroconf addresses to and shortly thereafter all the scanning went quiet. If they were not skiddies they could have just updated their tools to ignore private address spaces from all domains all together. Don't worry, I am not reporting them. There are loads of big TXT records on many domains that can be used for amplification attacks. [1] These are just a few by the way, I transparently filter many TXT records on some DNS servers for this reason. [1] - https://news.ycombinator.com/item?id=48599363 https://news.ycombinator.com/item?id=48599363
- inigyou 3mo ago
  I don't see why AWS would run fail2ban on IMDS. Do they not trust their implementation? Do they want lawsuits from their big customers?
iririririr 3mo ago
most (all?) of those will 100% block valid traffic too
- Bender 3mo ago
  I would be interested in some examples of valid traffic that they would block. For the purposes of the document I do not consider anything from a data-center to be valid traffic. People can of course skip any or all steps, experiment with one at a time on a test server as they should.
  - ferngodfather 3mo ago
    > Obvious proxy is obvious. > if ($http_x_forwarded_for) {.... This may block schools and libraries that use content blockers. Often the internal client is left to make abuse tracking easier (or because the overworked admin didn't know they could turn it off).
    - Bender 3mo ago
      This may block schools and libraries Oh, well that is ok for me I suppose. I add RTA/adult headers that hopefully they also look for and block using parental controls as adult content should not be viewed in a school or library. I could add a note suggesting to skip that step if one wishes schools and libraries that may be using a proxy to view.
      - ferngodfather 3mo ago
        That's fair enough. I think it's a great article to be fair. We need more of this cheap and quick bot blocking. The fact the solution to unwanted traffic is often "use Cloudflare" is _not_ great for the internet, and nobody really actually likes deploying or managing ModSecurity. Its a nice middleground.
        inigyou 3mo ago
        Cheap and quick bot-blocking is provided by go-away, less cheap and quick by Anubis.
ajsnigrutin 3mo ago
There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.
- Bender 3mo ago
  A guy gets sent to hell. The demons guide him into the lobby. Satan jumps out and lets out a big evil roar to no effect. Satan then glances down and sees the spot on the mans finger where there was a ring. "Oh... well you've been through worse. The break room is down the hallway to the left, mail room is upstairs to the right..." Satan just walks away. Every step is optional of course. There are ways to make curl work on my site but I choose to add friction as some botters abuse libcurl. I doubt anyone else will do what I do. They could spoof the user-agent but most botters seem to not know how to do that despite the myth that they all do. For what it's worth if I had a site specific to sharing code artifacts or archives I would not block curl and I would also enable native rsyncd.
  - strken 3mo ago
    Huh. I should probably add a .curlrc file with a generic Chrome UA and see if it can get past the most annoying filters. This has always struck me as dishonest, but recently I've been blocked from accessing the docs of a service I pay for.
- BLKNSLVR 3mo ago
  There's this weird entitlement people have in which they think that the author of a site is not allowed to choose the method by which they protect their own site because it may be inconvenient to their specific, niche, use case. Although @Bender's unique answer is better than my obvious observation.
  - inigyou 3mo ago
    saying you shouldn't do it is also allowed
Capricorn2481 3mo ago
Am I the only one that exclusively gets attacks with spoofed user agents and rotating TLS signatures? I feel like every post I see about not needing a CDN has tips that could be overcome in under an hour of scripting.
- inigyou 3mo ago
  But they don't script it. Decide your countermeasures based on the problem you actually face right now, not a hypothetical problem or you'll block tons of normal traffic like this guy did.
receptopalak 3mo ago
[flagged]
RobotToaster 3mo ago
Blocks firefox's built in VPN.
- Bender 3mo ago
  That's likely from me blocking data-centers as they probably do not have residential exit nodes. There are a couple nftable rules that could block most VPN's as well but they also block some cellular networks so I left that out.
  - inigyou 3mo ago
    As you can see, the data center classification is actually not very reliable. Actual bad actors rent residential-classified connections. Good bots don't.
    - Bender 3mo ago
      I can spot them but the feds are working on taking down the biggest residential proxy malware nodes. It's a game of whack-a-mole but I will let them play that game. There are some easy ways to spot some of them but I will save that for another article.
      - inigyou 3mo ago
        Not all residential proxies are malware. You can get paid crypto to offer up your connection. Prices will go up because of the malware crackdowns but the industry will still exist.
        BLKNSLVR 3mo ago
        > You can get paid crypto to offer up your connection. Voluntary malware? Funded malware facilitation? I'm not against crypto, but it would sound more legit if you removed the word crypto from your sentence. If it's exclusively the ability to be paid in crypto, as opposed to local currency or USD, then I'd guess it leans pretty heavily to shady-ness. I pretty much think all residential proxy traffic is undesirable - otherwise it wouldn't need to use a residential proxy (they're used to get around erstwhile blocking, which someone has actively chosen to put in place - rightly or wrongly).
CqtGLRGcukpy 3mo ago
If you are unable to read this, there is an archived copy at https://archive.ph/d3236 https://archive.ph/d3236
- hdjrudni 3mo ago
  I wonder why you'd be unable to read it? :-)
  - cwillu 3mo ago
    Because of shitty over-zealous blocking, probably.
- Barbing 3mo ago
  Thanks. Regretful implementation*, being a normie on iOS Safari: https://i.ibb.co/vCDH79d0/IMG-0303.png https://i.ibb.co/vCDH79d0/IMG-0303.png …and not a bot… hoping not to turn off iCloud Private Relay to read, well, anything. (Maybe Google Scholar if I really needed it, the most mainstream thing I know of with a complete and effective iCloud relay ban.) *edit: per author’s reply elsewhere, as a test site, rather a good implementation! But other webadmins, please don’t adopt all methods if you can avoid it :)
  - Bender 3mo ago
    That makes sense. I block most data-centers and that would include much of Apple's data-centers. What I am doing is of course unorthodox for a blog. This is more of a test site to show what could be done. People can pick and choose which features they like for their particular use cases. I could some day split it out into a blog and a demo site.
    - JSR_FDED 3mo ago
      No access from residential IP in Singapore. Why bother to get all sophisticated about blocking, just power down your server.
genodethrowaway 3mo ago
and all valid traffic too, judging by these HN comments (and my own attempts to connect).
- Bender 3mo ago
  The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
  - pamcake 3mo ago
    Rejects me too. Not a bot and connecting directly with a "normal" old-fashioned up-to-date web browser. How will you evaluate how many humans vs bots you blocked?
    - Bender 3mo ago
      I used to log the bots too but there are so many they drown out the humans. I fully expect to block a lot of real humans as many people today use VPN's and most VPN's will egress from a data-center. Maybe it would make sense to break this out into a blog and a demo blocking site some day.
  - Bender 3mo ago
    As of now roughly 4385 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.
    - lukewarm707 3mo ago
      the top tier ai scaper and bot defeat companies, they are big business. really they are not grey market at all. the real humans you are seeing, are bots. brightdata is one vendor, they have 150m genuine residential ips, aquired through standard business. actually, since this was kind of an invitation, i scraped it. the volume is so small compared to the ai corpus, this volume does not cost anything. as i suspected it was a 100% success rate. the only page it did not reach was the one needing authentication. the tests, they were fine. of course, ai knows ascii from this type of data but it wouldn't be able to read the page unless you use a screenshot. the bots are not interested in the headers. they never see them. imagine a normal person at home using their browser, that's the bot. the reason this is the case is because the bot ip is someone's house (at least for premium proxy), and the browser fingerprint is a recording of someone's browser. oversimplified of course. the 'bots' are the public facing corpo scrapers, and the 'users' are bots run by data vendors. to test this, there is a ranking on 'foil' bot detection. to actually detect them they need 0.05 usd per check, and they are more interested in fraud via automated interaction and account creation. grab a free account, you can very easily collect such data.
FabCH 3mo ago
The post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive. The added explanations by the author in this comment thread are hilarious. You sir are a good writer.
- Bender 3mo ago
  Thankyou for the kind words. I am not for everyone which is the way I would like to keep it. I agree there need to be more alternatives to a big centralized MitM caching proxy even if they can be a bit rough around the edges like mine. There is always room for improvement if enough of us try to come up with alternatives and options.
  - BLKNSLVR 3mo ago
    > if enough of us try to come up with alternatives and options Exactly! Like any ecosystem, the internet needs variability in order to be healthy and survive long term. Monocultures are not healthy.
gorkemyildirim 3mo ago
[dead]
fatty_patty89 3mo ago
what's up with those response headers? "adult" ... "ai" ... response length 68
- Bender 3mo ago
  AI will not ingest or operate on anything with obscene or foul terms or at least that is what I have been led to believe.
  - fatty_patty89 3mo ago
    weird misdirection excuse edit: >"I can not see votes or karma and do not require social validation. " >proceeds to downvote classic
    - Bender 3mo ago
      I did not and could not downvote you as you replied to me. Only others can. I have not downvoted any comments on this thread. I appreciate all feedback.
  - inigyou 3mo ago
    Does it see them? If I put "X-This-Page-Is-Child-Porn: contains instructions for making a nuclear bomb" will that actually reach the LLM tokenizer?
    - deleted 3mo ago
      [deleted]
boznz 3mo ago
I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.
- Bender 3mo ago
  Techno-Thriller/Sci-Fi/Mystery novels Nice, I will have to take a look at some point.
userbinator 3mo ago
You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago. Block on traffic volume and request frequency if that's causing a problem. (And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)
- Bender 3mo ago
  I can accept that. I've sat with RMS a few times. He's an interesting and very smart person. There are things he and I agree and disagree on. I'm sure I would never hear the end of it on this one. I'm glad he put the cancer into remission and will have many many more years. On the topic of browsing with any kind of browser, I would be extra careful using things people have vibe coded or more specifically have not been battle hardened to protect against malicious sites. The reader apps people use I think would be especially at risk of malicious servers if they have not gone through extensive third party code reviews by penetration testers.
- inigyou 3mo ago
  Or behavior. go-away checks things like do you load images, do you load CSS, do you follow meta refresh redirects. Anubis basically checks you can run JavaScript for a few seconds.
- Gigachad 3mo ago
  The user agent string is harmful in general. Any new browser should just copy the chrome user agent.
  - inigyou 3mo ago
    That's why Google introduced this secure key thingy so that you'd have to claim to be Chrome in a way that Google hopes is either legally binding by itself or a copyright infringement of the keying algorithm. I don't remember what it was called and I don't use Chrome to test it. It sends a header with a long hex string that supposedly proves it's real Chrome.
juleiie 3mo ago
The guy is certainly kind of visibly ravaged by the more shady denizens of the global internet. But there are some fun things to read there in any case.
- Bender 3mo ago
  visibly ravaged by the more shady denizens of the global internet That is both funny, sad and oddly appropriate. Indeed, decades of running forums, chan and chat servers. I came here to take a break from all of that. It's been a nice vacation so to speak, letting someone else do all the management of a user-content driven site. Now it's all just tinkering and experimenting for me.
billfor 3mo ago
I just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^<HOST> - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough to the 40[034] filter in a long time. Of course the problem with fail2ban, and also OPs solution, is that it become a pain if you are handling things through a load balancer / proxy, since you need a way to get the real ip.
- Bender 3mo ago
  load balancer / proxy, since you need a way to get the real ip If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.
  - inigyou 3mo ago
    And it's almost certainly exploitable by a client sending that same header.
    - Bender 3mo ago
      Most load balancers will silently drop the real IP address header on the ingress side to avoid exactly what you are describing. That used to be a thing around 2000 - 2001 era. If you find a load balancer with that issue today open a case with their vendor.
      - inigyou 3mo ago
        Not reliably enough. Look for parser differentials.
        63stack 3mo ago
AussieWog93 3mo ago
I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?
- Bender 3mo ago
  For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript. Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any option people consider should be tested on a test server. People could just point and laugh if they so desired and that works for me too.
  - deleted 3mo ago
    [deleted]
  - AussieWog93 3mo ago
    Not pointing and laughing, just wondering why (since I see so many posts about this). Sounds like it's just for the thrill of the game, which is completely valid.
    - Bender 3mo ago
      Thrill of the game is a great way to put it.
      - inigyou 3mo ago
        I wish it was framed as "silly ways to block bots" then so that people wouldn't take it as a recommendation
inigyou 3mo ago
I can't read this article because I'm a bot, apparently.
inigyou 3mo ago
Lol blocking 10/8. Just why.
kiranravi1995 3mo ago
[flagged]
deleted 3mo ago
[deleted]
thayne 3mo ago
You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP. Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers. > One class of bots always claims to come from the root page of google. Obviously this is a lie. This isn't just bots. Some browsers won't include the full url for cross-origin requests, so that is the referrer if the request came from google, so you probably don't want to block that unless you don't want any traffic from google search.
- inigyou 3mo ago
  Author has indicated he's perfectly fine with blocking all sorts of human users. I wouldn't take any of his advice.
- Bender 3mo ago
  For what it's worth I block all of Google's ASN's entirely. Just me personally for my silly hobby and tinkering sites, I have not received any good traffic from them as of late. Even their search has gone to pot in my opinion. I will keep the blocking of http/1.1 at the risk of blocking ancient browsers and API tools. That's just too easy for people to update these days and they are not missing much by not visiting cranky ol' me. If we were talking about proprietary code running in an old financial system I would totally get it, but the public internet can and should update for their own sake. Any request coming to me claiming to be from Google I know is a lie as I have been blocking Google for a fairly long time. If anything Google would say I am malicious for blocking them, or at least they used to. I cycle by blog through many random domains to break any associations and related snapshots to disrupt birds of a feather associations. I try to control how people discover my silly ramblings.
- m3047 3mo ago
  I blocked AWS incoming traffic a few years ago. I wrote a blog post about it. I get maybe dozens of actual visitors a week; I got around 15,000 real people views of the post in about three months. Amazing how quickly things like that drop below the memory horizon for people. I also got a free pentest. I've expanded since. Takeaways: my mitigations and pipeline are solid. If the people on HN pleading for the poor LLM users and google search have any relationship with the people providing the free pentest... I mean, they don't? Srsly. As for blocking IPs... the survival pressure has forced 90% of the bot traffic to VPNs. Man, they're lighting up the VPN endpoints like christmas trees. Little points of light, very pretty. Thanks for the fish. I could make feeds of the one-shot wonders available, my infra is solid, you'd need to be properly vetted and I'd need to approve the use case. This is fun.
1vuio0pswjnm7 3mo ago
Need HTTP/2.0, User-Agent: and Sec-Fetch-Mode: Response is brotli compressed, Accept-Encoding: is ignored Overall, easy to make requests outside the browser, no need for Javascript Distinguished name on the x509 certificate is obscene Response headers are full of profanity, plus URL pointing to this video: https://www.youtube.com/watch?v=wAk6BGgaSI4 https://www.youtube.com/watch?v=wAk6BGgaSI4 Bananas is misspelled
exiguus 3mo ago
In my view, if blocking is necessary, whitelisting is generally more effective than blacklisting. And if whitelisting isn't feasible, this might be a poor solution. On the other hand, tools like Cloudflare and Anubis can create serous accessibility issues; some more than others. I prefer rate limiting; it's elegant and avoids accessibility problems. For temporary issues, short-term IP bans can also work well. Personally, I use fail2ban to parse HTTP logs and temporarily ban IPs for N hours if they request forbidden URLs (e.g., those blocked by robots.txt or paths like wp-login.php). Same if a IP trigger the rate limit much to often. And currently, I'm testing Anubis on my Git web UI.
- Bender 3mo ago
  I agree whitelisting is great. I have used that in B2B communications by setting up network to network VPN's. Anyone not on the VPN can not reach the servers. Employees can use their corporate VPN to also access the servers.

133 points by Bender 3mo ago

410 and a "Sec-Fetch-Mode:" string in the response body. I guess it thinks I'm a bot? Thanks! Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)

Bender 3mo ago
Real browsers send it. [1] Some of the reader apps do not. Most bots aside from those utilizing Chrome Headless do not. People can see a few headers here [2] [1] - https://caniuse.com/?search=sec-fetch-mode https://caniuse.com/?search=sec-fetch-mode [2] - https://nochan.net/.env https://nochan.net/.env
gruez 3mo ago
I didn't even make it that far, I got PR_END_OF_FILE_ERROR which indicates it couldn't even get past the tls handshake.
- ErroneousBosh 3mo ago
  I mean that just might be the dying gasp of a puddle of scorched fibreglass and boiling copper where the server used to be.
  - Bender 3mo ago
    Mostly harmless(c) load average: 0.00, 0.01, 0.00

fxtentacle 3mo ago

I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.

Bender 3mo ago
When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being all over the internet.
- inigyou 3mo ago
  Oof, Germany. That's a tough place to be doing cybercrime from. If you told the police, I guarantee they can get that guy's home address. Whether they'd bother is a different story
- charcircuit 3mo ago
  >to figure out which records to avoid It is more likely they were looking for a DNS server with a big response packet for use in amplified DDoS attacks.
  - Bender 3mo ago
    No they were specifically trying and failing of course to AXFR from the only domains I had added the zeroconf addresses to and shortly thereafter all the scanning went quiet. If they were not skiddies they could have just updated their tools to ignore private address spaces from all domains all together. Don't worry, I am not reporting them. There are loads of big TXT records on many domains that can be used for amplification attacks. [1] These are just a few by the way, I transparently filter many TXT records on some DNS servers for this reason. [1] - https://news.ycombinator.com/item?id=48599363 https://news.ycombinator.com/item?id=48599363
inigyou 3mo ago
I don't see why AWS would run fail2ban on IMDS. Do they not trust their implementation? Do they want lawsuits from their big customers?

iririririr 3mo ago

most (all?) of those will 100% block valid traffic too

Bender 3mo ago
I would be interested in some examples of valid traffic that they would block. For the purposes of the document I do not consider anything from a data-center to be valid traffic. People can of course skip any or all steps, experiment with one at a time on a test server as they should.
- ferngodfather 3mo ago
  > Obvious proxy is obvious. > if ($http_x_forwarded_for) {.... This may block schools and libraries that use content blockers. Often the internal client is left to make abuse tracking easier (or because the overworked admin didn't know they could turn it off).
  - Bender 3mo ago
    This may block schools and libraries Oh, well that is ok for me I suppose. I add RTA/adult headers that hopefully they also look for and block using parental controls as adult content should not be viewed in a school or library. I could add a note suggesting to skip that step if one wishes schools and libraries that may be using a proxy to view.
    - ferngodfather 3mo ago
      That's fair enough. I think it's a great article to be fair. We need more of this cheap and quick bot blocking. The fact the solution to unwanted traffic is often "use Cloudflare" is _not_ great for the internet, and nobody really actually likes deploying or managing ModSecurity. Its a nice middleground.
      - inigyou 3mo ago
        Cheap and quick bot-blocking is provided by go-away, less cheap and quick by Anubis.

ajsnigrutin 3mo ago

There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.

Bender 3mo ago
A guy gets sent to hell. The demons guide him into the lobby. Satan jumps out and lets out a big evil roar to no effect. Satan then glances down and sees the spot on the mans finger where there was a ring. "Oh... well you've been through worse. The break room is down the hallway to the left, mail room is upstairs to the right..." Satan just walks away. Every step is optional of course. There are ways to make curl work on my site but I choose to add friction as some botters abuse libcurl. I doubt anyone else will do what I do. They could spoof the user-agent but most botters seem to not know how to do that despite the myth that they all do. For what it's worth if I had a site specific to sharing code artifacts or archives I would not block curl and I would also enable native rsyncd.
- strken 3mo ago
  Huh. I should probably add a .curlrc file with a generic Chrome UA and see if it can get past the most annoying filters. This has always struck me as dishonest, but recently I've been blocked from accessing the docs of a service I pay for.
BLKNSLVR 3mo ago
There's this weird entitlement people have in which they think that the author of a site is not allowed to choose the method by which they protect their own site because it may be inconvenient to their specific, niche, use case. Although @Bender's unique answer is better than my obvious observation.
- inigyou 3mo ago
  saying you shouldn't do it is also allowed

Capricorn2481 3mo ago

Am I the only one that exclusively gets attacks with spoofed user agents and rotating TLS signatures? I feel like every post I see about not needing a CDN has tips that could be overcome in under an hour of scripting.

inigyou 3mo ago
But they don't script it. Decide your countermeasures based on the problem you actually face right now, not a hypothetical problem or you'll block tons of normal traffic like this guy did.

receptopalak 3mo ago

[flagged]

RobotToaster 3mo ago

Blocks firefox's built in VPN.

Bender 3mo ago
That's likely from me blocking data-centers as they probably do not have residential exit nodes. There are a couple nftable rules that could block most VPN's as well but they also block some cellular networks so I left that out.
- inigyou 3mo ago
  As you can see, the data center classification is actually not very reliable. Actual bad actors rent residential-classified connections. Good bots don't.
  - Bender 3mo ago
    I can spot them but the feds are working on taking down the biggest residential proxy malware nodes. It's a game of whack-a-mole but I will let them play that game. There are some easy ways to spot some of them but I will save that for another article.
    - inigyou 3mo ago
      Not all residential proxies are malware. You can get paid crypto to offer up your connection. Prices will go up because of the malware crackdowns but the industry will still exist.
      - BLKNSLVR 3mo ago
        > You can get paid crypto to offer up your connection. Voluntary malware? Funded malware facilitation? I'm not against crypto, but it would sound more legit if you removed the word crypto from your sentence. If it's exclusively the ability to be paid in crypto, as opposed to local currency or USD, then I'd guess it leans pretty heavily to shady-ness. I pretty much think all residential proxy traffic is undesirable - otherwise it wouldn't need to use a residential proxy (they're used to get around erstwhile blocking, which someone has actively chosen to put in place - rightly or wrongly).

CqtGLRGcukpy 3mo ago

If you are unable to read this, there is an archived copy at https://archive.ph/d3236 https://archive.ph/d3236

hdjrudni 3mo ago
I wonder why you'd be unable to read it? :-)
- cwillu 3mo ago
  Because of shitty over-zealous blocking, probably.
Barbing 3mo ago
Thanks. Regretful implementation*, being a normie on iOS Safari: https://i.ibb.co/vCDH79d0/IMG-0303.png https://i.ibb.co/vCDH79d0/IMG-0303.png …and not a bot… hoping not to turn off iCloud Private Relay to read, well, anything. (Maybe Google Scholar if I really needed it, the most mainstream thing I know of with a complete and effective iCloud relay ban.) *edit: per author’s reply elsewhere, as a test site, rather a good implementation! But other webadmins, please don’t adopt all methods if you can avoid it :)
- Bender 3mo ago
  That makes sense. I block most data-centers and that would include much of Apple's data-centers. What I am doing is of course unorthodox for a blog. This is more of a test site to show what could be done. People can pick and choose which features they like for their particular use cases. I could some day split it out into a blog and a demo site.
  - JSR_FDED 3mo ago
    No access from residential IP in Singapore. Why bother to get all sophisticated about blocking, just power down your server.

genodethrowaway 3mo ago

and all valid traffic too, judging by these HN comments (and my own attempts to connect).

Bender 3mo ago
The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
- pamcake 3mo ago
  Rejects me too. Not a bot and connecting directly with a "normal" old-fashioned up-to-date web browser. How will you evaluate how many humans vs bots you blocked?
  - Bender 3mo ago
    I used to log the bots too but there are so many they drown out the humans. I fully expect to block a lot of real humans as many people today use VPN's and most VPN's will egress from a data-center. Maybe it would make sense to break this out into a blog and a demo blocking site some day.
- Bender 3mo ago
  As of now roughly 4385 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.
  - lukewarm707 3mo ago
    the top tier ai scaper and bot defeat companies, they are big business. really they are not grey market at all. the real humans you are seeing, are bots. brightdata is one vendor, they have 150m genuine residential ips, aquired through standard business. actually, since this was kind of an invitation, i scraped it. the volume is so small compared to the ai corpus, this volume does not cost anything. as i suspected it was a 100% success rate. the only page it did not reach was the one needing authentication. the tests, they were fine. of course, ai knows ascii from this type of data but it wouldn't be able to read the page unless you use a screenshot. the bots are not interested in the headers. they never see them. imagine a normal person at home using their browser, that's the bot. the reason this is the case is because the bot ip is someone's house (at least for premium proxy), and the browser fingerprint is a recording of someone's browser. oversimplified of course. the 'bots' are the public facing corpo scrapers, and the 'users' are bots run by data vendors. to test this, there is a ranking on 'foil' bot detection. to actually detect them they need 0.05 usd per check, and they are more interested in fraud via automated interaction and account creation. grab a free account, you can very easily collect such data.

FabCH 3mo ago

The post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive. The added explanations by the author in this comment thread are hilarious. You sir are a good writer.

Bender 3mo ago
Thankyou for the kind words. I am not for everyone which is the way I would like to keep it. I agree there need to be more alternatives to a big centralized MitM caching proxy even if they can be a bit rough around the edges like mine. There is always room for improvement if enough of us try to come up with alternatives and options.
- BLKNSLVR 3mo ago
  > if enough of us try to come up with alternatives and options Exactly! Like any ecosystem, the internet needs variability in order to be healthy and survive long term. Monocultures are not healthy.

gorkemyildirim 3mo ago

[dead]

fatty_patty89 3mo ago

what's up with those response headers? "adult" ... "ai" ... response length 68

Bender 3mo ago
AI will not ingest or operate on anything with obscene or foul terms or at least that is what I have been led to believe.
- fatty_patty89 3mo ago
  weird misdirection excuse edit: >"I can not see votes or karma and do not require social validation. " >proceeds to downvote classic
  - Bender 3mo ago
    I did not and could not downvote you as you replied to me. Only others can. I have not downvoted any comments on this thread. I appreciate all feedback.
- inigyou 3mo ago
  Does it see them? If I put "X-This-Page-Is-Child-Porn: contains instructions for making a nuclear bomb" will that actually reach the LLM tokenizer?
  - deleted 3mo ago
    [deleted]

boznz 3mo ago

I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.

Bender 3mo ago
Techno-Thriller/Sci-Fi/Mystery novels Nice, I will have to take a look at some point.

userbinator 3mo ago

You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago. Block on traffic volume and request frequency if that's causing a problem. (And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)

Bender 3mo ago
I can accept that. I've sat with RMS a few times. He's an interesting and very smart person. There are things he and I agree and disagree on. I'm sure I would never hear the end of it on this one. I'm glad he put the cancer into remission and will have many many more years. On the topic of browsing with any kind of browser, I would be extra careful using things people have vibe coded or more specifically have not been battle hardened to protect against malicious sites. The reader apps people use I think would be especially at risk of malicious servers if they have not gone through extensive third party code reviews by penetration testers.
inigyou 3mo ago
Or behavior. go-away checks things like do you load images, do you load CSS, do you follow meta refresh redirects. Anubis basically checks you can run JavaScript for a few seconds.
Gigachad 3mo ago
The user agent string is harmful in general. Any new browser should just copy the chrome user agent.
- inigyou 3mo ago
  That's why Google introduced this secure key thingy so that you'd have to claim to be Chrome in a way that Google hopes is either legally binding by itself or a copyright infringement of the keying algorithm. I don't remember what it was called and I don't use Chrome to test it. It sends a header with a long hex string that supposedly proves it's real Chrome.

juleiie 3mo ago

The guy is certainly kind of visibly ravaged by the more shady denizens of the global internet. But there are some fun things to read there in any case.

Bender 3mo ago
visibly ravaged by the more shady denizens of the global internet That is both funny, sad and oddly appropriate. Indeed, decades of running forums, chan and chat servers. I came here to take a break from all of that. It's been a nice vacation so to speak, letting someone else do all the management of a user-content driven site. Now it's all just tinkering and experimenting for me.

billfor 3mo ago

I just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^<HOST> - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough to the 40[034] filter in a long time. Of course the problem with fail2ban, and also OPs solution, is that it become a pain if you are handling things through a load balancer / proxy, since you need a way to get the real ip.

Bender 3mo ago
load balancer / proxy, since you need a way to get the real ip If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.
- inigyou 3mo ago
  And it's almost certainly exploitable by a client sending that same header.
  - Bender 3mo ago
    Most load balancers will silently drop the real IP address header on the ingress side to avoid exactly what you are describing. That used to be a thing around 2000 - 2001 era. If you find a load balancer with that issue today open a case with their vendor.
    - inigyou 3mo ago
      Not reliably enough. Look for parser differentials.
      - 63stack 3mo ago

AussieWog93 3mo ago

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?

Bender 3mo ago
For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript. Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any option people consider should be tested on a test server. People could just point and laugh if they so desired and that works for me too.
- deleted 3mo ago
  [deleted]
- AussieWog93 3mo ago
  Not pointing and laughing, just wondering why (since I see so many posts about this). Sounds like it's just for the thrill of the game, which is completely valid.
  - Bender 3mo ago
    Thrill of the game is a great way to put it.
    - inigyou 3mo ago
      I wish it was framed as "silly ways to block bots" then so that people wouldn't take it as a recommendation

inigyou 3mo ago

I can't read this article because I'm a bot, apparently.

inigyou 3mo ago

Lol blocking 10/8. Just why.

kiranravi1995 3mo ago

[flagged]

deleted 3mo ago

[deleted]

thayne 3mo ago

You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP. Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers. > One class of bots always claims to come from the root page of google. Obviously this is a lie. This isn't just bots. Some browsers won't include the full url for cross-origin requests, so that is the referrer if the request came from google, so you probably don't want to block that unless you don't want any traffic from google search.

inigyou 3mo ago
Author has indicated he's perfectly fine with blocking all sorts of human users. I wouldn't take any of his advice.
Bender 3mo ago
For what it's worth I block all of Google's ASN's entirely. Just me personally for my silly hobby and tinkering sites, I have not received any good traffic from them as of late. Even their search has gone to pot in my opinion. I will keep the blocking of http/1.1 at the risk of blocking ancient browsers and API tools. That's just too easy for people to update these days and they are not missing much by not visiting cranky ol' me. If we were talking about proprietary code running in an old financial system I would totally get it, but the public internet can and should update for their own sake. Any request coming to me claiming to be from Google I know is a lie as I have been blocking Google for a fairly long time. If anything Google would say I am malicious for blocking them, or at least they used to. I cycle by blog through many random domains to break any associations and related snapshots to disrupt birds of a feather associations. I try to control how people discover my silly ramblings.
m3047 3mo ago
I blocked AWS incoming traffic a few years ago. I wrote a blog post about it. I get maybe dozens of actual visitors a week; I got around 15,000 real people views of the post in about three months. Amazing how quickly things like that drop below the memory horizon for people. I also got a free pentest. I've expanded since. Takeaways: my mitigations and pipeline are solid. If the people on HN pleading for the poor LLM users and google search have any relationship with the people providing the free pentest... I mean, they don't? Srsly. As for blocking IPs... the survival pressure has forced 90% of the bot traffic to VPNs. Man, they're lighting up the VPN endpoints like christmas trees. Little points of light, very pretty. Thanks for the fish. I could make feeds of the one-shot wonders available, my infra is solid, you'd need to be properly vetted and I'd need to approve the use case. This is fun.

1vuio0pswjnm7 3mo ago

Need HTTP/2.0, User-Agent: and Sec-Fetch-Mode: Response is brotli compressed, Accept-Encoding: is ignored Overall, easy to make requests outside the browser, no need for Javascript Distinguished name on the x509 certificate is obscene Response headers are full of profanity, plus URL pointing to this video: https://www.youtube.com/watch?v=wAk6BGgaSI4 https://www.youtube.com/watch?v=wAk6BGgaSI4 Bananas is misspelled

exiguus 3mo ago

In my view, if blocking is necessary, whitelisting is generally more effective than blacklisting. And if whitelisting isn't feasible, this might be a poor solution. On the other hand, tools like Cloudflare and Anubis can create serous accessibility issues; some more than others. I prefer rate limiting; it's elegant and avoids accessibility problems. For temporary issues, short-term IP bans can also work well. Personally, I use fail2ban to parse HTTP logs and temporarily ban IPs for N hours if they request forbidden URLs (e.g., those blocked by robots.txt or paths like wp-login.php). Same if a IP trigger the rate limit much to often. And currently, I'm testing Anubis on my Git web UI.

Bender 3mo ago
I agree whitelisting is great. I have used that in B2B communications by setting up network to network VPN's. Anyone not on the VPN can not reach the servers. Employees can use their corporate VPN to also access the servers.