3 ms·

Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few th

Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.

basilikum 2mo ago
> /commercial IP from legitimate users No, because legitimate users do not just use residential and "commercial" IPs. Like me, right now
- VladVladikoff 2mo ago
  You are the 0.001%
  - basilikum 2mo ago
    Much more than 0.001% of people care about their privacy or (the larger portion) do not have unfiltered access to the internet.
    - Bender 2mo ago
      I second this. When I have tested blocking VPS/data-centers to my silly blog there were about a dozen people on HN [1] that could not view my site out of the roughly ~17,000 (not counting bots) that could. It's not a big number but those are real people and they count. I am going to move full blocking to a test node that people can play with but I have to finish working with Claude to revise someones repo is is no longer maintained because one does not simply put an anonymous chan board on the great wide open internets without some critical thinking. [1] - https://news.ycombinator.com/item?id=49060945 https://news.ycombinator.com/item?id=49060945
      - VladVladikoff 2mo ago
        What did you use for detection?
        Bender 2mo ago
        The complaints in the thread. I am aware the lurkers would not have said anything.
      - inigyou 2mo ago
        There were a dozen people who reported not being able to access your site. The complete hysteria people go to over the near-non-issue of bots is, well, completely hysterical. Don't be that guy.
VladVladikoff 2mo ago
Yes but it’s not cheap. Maxmind and ipinfo etc sell a tier that tells you this information, then you can 403 based on it. But the price is nuts like $40,000 a year.
- KomoD 2mo ago
  You don't need to spend anywhere near $40k a year to get that info... You don't even need to spend $1
  - gavinhking 2mo ago
    What's your strategy?
- sparkling 2mo ago
  Focsec.com IP database for offline use (datacenters, VPNs, proxies, bots) runs around $1k/month for internal-only use.
- reincoder 2mo ago
  I work for IPinfo. We offer IPinfo Lite for free. With a little bit of time in identifying the ASNs, you can implement a decent way to block a good number of bots fairly easily using the free data alone.
  - gavinhking 2mo ago
    Good to know, thank you. Would you do this by fully blocking particular ASNs? Or something more granular?
    - reincoder 2mo ago
      You can block entire ASNs. If you are frustrated with bots, blocking Tencent's entire IP address space would have very few downsides. If you have fail2ban or NGINX logs, you can use our CLI to summarize those IPs and identify the ASNs you want to block. But before you block entire ASNs, make sure they are not classified as "ISP" type. For that, visit our website's ASN page first. I have quite a few community posts around this approach. https://community.ipinfo.io/ https://community.ipinfo.io/ If you have raw logs, you can send them to me as well, and I can review them and provide some guidance.
djkurlander 2mo ago
I run an open source honeypot that collects these botnet scans and produces blocklists. Blocklist download and configuration: https://knock-knock.net/blocklist https://knock-knock.net/blocklist Honeypot dashboard, where you can see attempted attacks in realtime: http://knock-knock.net http://knock-knock.net API: http://knock-knock.net/api http://knock-knock.net/api
- skinfaxi 2mo ago
  This looks cool, where can I find the source?
  - djkurlander 2mo ago
    It's on github with an MIT license: https://github.com/djkurlander/knock-knock https://github.com/djkurlander/knock-knock. Have fun!
    - skinfaxi 2mo ago
      Thank you!!
- deleted 2mo ago
  [deleted]
- codegeek 2mo ago
  Thank you for sharing. I will take a look.
- Bender 2mo ago
  Cool site. I was curious and dropped your 100k list into a reverse DNS lookup site [1]. They may still have some of the records cached. I recognized quite a few of the scanner nodes and some other usual suspects. [1] - https://adver.tools/reverse-dns-lookup/ https://adver.tools/reverse-dns-lookup/
  - djkurlander 2mo ago
Bender 2mo ago
There are several methods. [1] The most aggressive method-02 and method-03 on my document will block VPS and some data-centers but that also means it will block some legit users that are on a VPN. Most VPNs transit a data-center. If experimenting with these methods use a test server that you do not care about and set up a dummy site and ask people in your circle of friends to test it. I have to step away for a bit but if you have questions I will try to answer. [1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Some-Of-The-Bots/ https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
- VogonPoetry 2mo ago
  Your site does not currently seem to be reachable / responsive when I try to reach it from a US Comcast IPv4 address - you are not advertising IPv6. Edit / Update: It was Apple's Private browsing mode that causes it not to respond. I can now see it when this is disabled.
  - Bender 2mo ago
    I've noticed they strip away a header [1] in private browsing mode but I don't know why they do it since it does not disclose anything about the person. I think that may be the same thing that causes some people grief on Cloudflare as well. [1] - https://caniuse.com/?search=sec-fetch https://caniuse.com/?search=sec-fetch
- nubinetwork 2mo ago
  Per your link... > block http 1.1, real users only use 2.0 Chrome on android and Firefox on linux both appear to use 1.1 still...
  - Bender 2mo ago
    By default they use 2.0 [1] unless someone or an addon disables it or unless the person is on a really old version. OperaMini however will use 1.1. No idea if anyone here uses OperaMini. There are some reader apps that act as a proxy that only support http/1.1. Be careful, some of those are not just readers and do not trust what they claim to be the source code. Some of them are created by cute and fuzzy bunnies. There are a number of botters on HN, some that control residential and phone browser-hijacked systems. One was sending me playful messages the other day. I enjoyed the bot block-jousting with them. [1] - https://caniuse.com/http2 https://caniuse.com/http2
inigyou 2mo ago
No, because the bad guys use residential proxies if you block DCs. They just prefer not to because it's more expensive. Meanwhile that silly HN project that scrapes and reformats your site is now dead before it began. Who are you actually defending and from what?

basilikum 2mo ago

> /commercial IP from legitimate users No, because legitimate users do not just use residential and "commercial" IPs. Like me, right now

VladVladikoff 2mo ago
You are the 0.001%
- basilikum 2mo ago
  Much more than 0.001% of people care about their privacy or (the larger portion) do not have unfiltered access to the internet.
  - Bender 2mo ago
    I second this. When I have tested blocking VPS/data-centers to my silly blog there were about a dozen people on HN [1] that could not view my site out of the roughly ~17,000 (not counting bots) that could. It's not a big number but those are real people and they count. I am going to move full blocking to a test node that people can play with but I have to finish working with Claude to revise someones repo is is no longer maintained because one does not simply put an anonymous chan board on the great wide open internets without some critical thinking. [1] - https://news.ycombinator.com/item?id=49060945 https://news.ycombinator.com/item?id=49060945
    - VladVladikoff 2mo ago
      What did you use for detection?
      - Bender 2mo ago
        The complaints in the thread. I am aware the lurkers would not have said anything.
    - inigyou 2mo ago
      There were a dozen people who reported not being able to access your site. The complete hysteria people go to over the near-non-issue of bots is, well, completely hysterical. Don't be that guy.

VladVladikoff 2mo ago

Yes but it’s not cheap. Maxmind and ipinfo etc sell a tier that tells you this information, then you can 403 based on it. But the price is nuts like $40,000 a year.

KomoD 2mo ago
You don't need to spend anywhere near $40k a year to get that info... You don't even need to spend $1
- gavinhking 2mo ago
  What's your strategy?
sparkling 2mo ago
Focsec.com IP database for offline use (datacenters, VPNs, proxies, bots) runs around $1k/month for internal-only use.
reincoder 2mo ago
I work for IPinfo. We offer IPinfo Lite for free. With a little bit of time in identifying the ASNs, you can implement a decent way to block a good number of bots fairly easily using the free data alone.
- gavinhking 2mo ago
  Good to know, thank you. Would you do this by fully blocking particular ASNs? Or something more granular?
  - reincoder 2mo ago
    You can block entire ASNs. If you are frustrated with bots, blocking Tencent's entire IP address space would have very few downsides. If you have fail2ban or NGINX logs, you can use our CLI to summarize those IPs and identify the ASNs you want to block. But before you block entire ASNs, make sure they are not classified as "ISP" type. For that, visit our website's ASN page first. I have quite a few community posts around this approach. https://community.ipinfo.io/ https://community.ipinfo.io/ If you have raw logs, you can send them to me as well, and I can review them and provide some guidance.

djkurlander 2mo ago

I run an open source honeypot that collects these botnet scans and produces blocklists. Blocklist download and configuration: https://knock-knock.net/blocklist https://knock-knock.net/blocklist Honeypot dashboard, where you can see attempted attacks in realtime: http://knock-knock.net http://knock-knock.net API: http://knock-knock.net/api http://knock-knock.net/api

skinfaxi 2mo ago
This looks cool, where can I find the source?
- djkurlander 2mo ago
  It's on github with an MIT license: https://github.com/djkurlander/knock-knock https://github.com/djkurlander/knock-knock. Have fun!
  - skinfaxi 2mo ago
    Thank you!!
deleted 2mo ago
[deleted]
codegeek 2mo ago
Thank you for sharing. I will take a look.
Bender 2mo ago
Cool site. I was curious and dropped your 100k list into a reverse DNS lookup site [1]. They may still have some of the records cached. I recognized quite a few of the scanner nodes and some other usual suspects. [1] - https://adver.tools/reverse-dns-lookup/ https://adver.tools/reverse-dns-lookup/
- djkurlander 2mo ago

Bender 2mo ago

There are several methods. [1] The most aggressive method-02 and method-03 on my document will block VPS and some data-centers but that also means it will block some legit users that are on a VPN. Most VPNs transit a data-center. If experimenting with these methods use a test server that you do not care about and set up a dummy site and ask people in your circle of friends to test it. I have to step away for a bit but if you have questions I will try to answer. [1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Some-Of-The-Bots/ https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...

VogonPoetry 2mo ago
Your site does not currently seem to be reachable / responsive when I try to reach it from a US Comcast IPv4 address - you are not advertising IPv6. Edit / Update: It was Apple's Private browsing mode that causes it not to respond. I can now see it when this is disabled.
- Bender 2mo ago
  I've noticed they strip away a header [1] in private browsing mode but I don't know why they do it since it does not disclose anything about the person. I think that may be the same thing that causes some people grief on Cloudflare as well. [1] - https://caniuse.com/?search=sec-fetch https://caniuse.com/?search=sec-fetch
nubinetwork 2mo ago
Per your link... > block http 1.1, real users only use 2.0 Chrome on android and Firefox on linux both appear to use 1.1 still...
- Bender 2mo ago
  By default they use 2.0 [1] unless someone or an addon disables it or unless the person is on a really old version. OperaMini however will use 1.1. No idea if anyone here uses OperaMini. There are some reader apps that act as a proxy that only support http/1.1. Be careful, some of those are not just readers and do not trust what they claim to be the source code. Some of them are created by cute and fuzzy bunnies. There are a number of botters on HN, some that control residential and phone browser-hijacked systems. One was sending me playful messages the other day. I enjoyed the bot block-jousting with them. [1] - https://caniuse.com/http2 https://caniuse.com/http2

inigyou 2mo ago

No, because the bad guys use residential proxies if you block DCs. They just prefer not to because it's more expensive. Meanwhile that silly HN project that scrapes and reformats your site is now dead before it began. Who are you actually defending and from what?