11 ms·

Now Blocking 56,037,235 IP Addresses, and Counting

43 points by devinjon 3y ago

chupapimunyenyo 3y ago
I cant read this, im blocked
- ziddoap 3y ago
  Indeed, for whatever reason I appear to be blocked as well. Never even heard of this site, and I am not a bot. Just caught in the crossfire I guess. ¯\_(ツ)_/¯
devinjon 3y ago
"I imagine that if this article makes its way onto Hacker News, I will be criticized."
- Bluecobra 3y ago
  I think it’s a healthy thing to read other’s opinions, even if you disagree with it. HN does a good job of keeping the trolls out. Edit: FWIW I think you’re fighting the good fight and I wish I could implement something similar at work, in where I constantly have bots trying to login to my corporate VPN.
  - WarOnPrivacy 3y ago
    > I think it’s a healthy thing to read other’s opinions, even if you disagree with it. HN does a good job of keeping the trolls out. I like to think world class nits are picked here but you're right - this is one of the more rational neighborhoods. > I wish I could implement something similar at work, in where I constantly have bots trying to login to my corporate VPN. I kludged some bash together to grep the logs and give me IP hit-counts and network owners. I next struggled to make decisions about the IPs - due to 1) the prevalence of residential proxies and 2) learning one of our widgets refreshes 3x/min.
citrin_ru 3y ago
Using http://nginx.org/r/deny http://nginx.org/r/deny is a very inefficient way to block large number of IP/nets. It mentioned right in the documentation: > In case of a lot of rules, the use of the ngx_http_geo_module module variables is preferable.
mergy 3y ago
I block as well with geoblocking and based on source behavior. As long as you understand the limitations, ramifications, and futility in doing this, I have no problems with it as one of the many tactics to defend your footprint and de-noise your logs, etc. It's a never ending endeavor and you will see the shifting attack sources from the baddies along with the games on stub blocks, prefix broker IP block swaps, and more. Just know there is automation and horsepower into that entire attack infrastructure you can't possibly compete with but maybe you can mitigate with the limited time and resources you have and that will be enough to get you through.
hifromwork 3y ago
>It means that more than one percent of the IPv4 real estate on the Internet (and probably much more) is occupied by people and organizations who are either clueless or just do not care how much the rest of us are paying to keep our websites on line There's a significant mental leap here. "I block these IP to conserve my resources, therefore they belong to clueless or malicious organisations". It's wrong in both directions: * I don't think Google, Bing and other crawlers are inherently malicious, and certainly not clueless. Search engines serve a very important role in the internet. Ditto archive.org, and probably dozens of other bots. * IP based blocklists work well for honest bots (not malicious, or at least not illegal). Malicious bot operators just buy SIM cards and use regular mobile internet for the crawling (basically unblockable, because the IP may be renewed every day or every hour). And the really malicious actors use residential proxies, i.e. botnets that proxy traffic through normal users' computers. Anyway I wonder how many of those 56MM IP addresses are regular dynamically allocated consumer grade ISPs. >1-5-2024 For the love of all that is holy, what is this date format.
- WarOnPrivacy 3y ago
  >>1-5-2024 >For the love of all that is holy, what is this date format. Most commonly used in the US. 'Cause we gotta show it like we say it. I think.
  - wccrawford 3y ago
    With dashes? We almost always use slashes here in the US. The only time I see dashes is for ISO format, and that's year-first.
    - D13Fd 3y ago
      Yes. It’s less common but looks unremarkable to me.
    - WarOnPrivacy 3y ago
      Sure. It was the dominant written style throughout the 20th cent and folks kept on with it.
aledalgrande 3y ago
Blocking bots is always going to be an uphill battle. But if the owner is worried about wasting meagre resources, why not serving static HTML files instead of running a PHP server for a simple blog?
turtleyacht 3y ago
> I don't care. Just do your own thing, learn from it, and repeat. That's all we can really want in our projects, on our limited lease on planet Earth. Kudos that you found something to work on.
TekMol 3y ago
What is the issue they are trying to solve? It seems to be a static site. Bots should cause only a neglible amount of traffic per month. My guess would be less than $1. And aren't there free CDNs for static sites these days? I guess you can just push the whole frontent data (html+assets) into a public git repo, put it behind a github page with custom domain and call it a day?
- jdietrich 3y ago
  > What is the issue they are trying to solve? Apparently, he self-hosts on a Raspberry Pi 3B+. I guess you need to block half the internet if your server has about the same performance as a decade-old smartphone. It's charming in a way, like someone who daily-drives a classic car. https://cheapskatesguide.org/articles/self-hosters-nightmare.html https://cheapskatesguide.org/articles/self-hosters-nightmare...
  - Filligree 3y ago
    > After a loss of power, my routers and web server must be restarted in a specific sequence for my home network to function properly. There's a certain charm to Rube Goldberg machines as well.
  - miyuru 3y ago
    LOL, the first comment of that article. "To worldofmathew on Hacker News, While I do sincerely appreciate the fact that you have taken the time to post several cheapskatesguide articles to Hacker News, this may have caused them to designate you as a blog spammer and block your posts. In other words, they may think that you own this site and are posting your own articles. May I suggest that you contact a Hacker News moderator and set him straight before you post another cheapskatesguide article?"
- montroser 3y ago
  No, this craziness(!) here suggests that it's not a static site: > "I added hyphens to the opening and closing PHP statements to prevent my web server from interpreting them as code." It could be a static site, and one could also automate the job that this person spends their precious time doing every day, copying and pasting text (IP addresses) that matches a pattern from one place to another. But they seem content, so...
WarOnPrivacy 3y ago
I imagine that if this article makes its way onto Hacker News, I will be criticized. Maybe they will call me naive or compare me to Don Quixote fighting windmills. Maybe they will call me stupid or paranoid for not using some centralized block list. Maybe they will object to how I characterize those who employ web-crawling robots. Ha. He is so wrong. We're going to criticize his nginx config and his use of PHP. I mean yeah. We'll probably get after that other stuff too but still.
- hsuduebc2 3y ago
  PHP on hackernews also caught my attention.
Fb24k 3y ago
After reading his three part multi-month series about how he can't set up a firewall, I don't think this guy is probably someone who should be providing any useful information on how to use the internet (or anything attached to it).
- justsomehnguy 3y ago
  Now I read it too (well, skimmed). >> -A INPUT -j DROP Yep, it's all is needed to know about this guy.
justsomehnguy 3y ago
> It means that more than one percent of the IPv4 real estate on the Internet (and probably much more) is occupied by people and organizations who are either clueless or just do not care how much the rest of us are paying to keep our websites on line. Oh, tell me, how much? The whopping $5/month? Oh, maybe this is a high load WordPress/like CMS running on LAMP stack... so $8/month? > I wrote the following small PHP script to search though my Nginx configuration file and tally up the number of IP addresses that I am blocking. Holy shit. Blocking bots through nginx configuration, more so, blocking 56M addresses through nginx configuration... Okay, for those of you who never did the thing or have no idea: Just use the firewall (most of the time it is built-in in your OS), use some way to tell the firewall about the 'offenders' (eg fail2ban though there are options) and don't ever block something indefinitely, it's totally meaningless, just use timeouts. If some Bob got his computer infected in 2015 and that computer tried to access /wp-admin.php then there is absolutely no reason to assume what in 2024 the IP address Bob's computer had in 2015 is still 'malicious'. Automatic activity like the scans, bruteforcing and whatever is all about opportunity. They are searching for an easy opportunities to exploit and scanning a server what actively blocks you even for 30m at time is just pointless, there is way, way more opportunities in other places than wasting ~4 weeks trying to scan this server. > I have custom 403 and 404 error pages that explain to those who may care why they are being blocked and how to regain access to the website https://cheapskatesguide.org/custom404really.html https://cheapskatesguide.org/custom404really.html
- Sweepi 3y ago
  Maybe the custom "blocked" 403/404 error page only shows up if your IP is blocked?
  - nijave 3y ago
    They could run a web server (or listener/virtual host) on a separate port and use a firewall to rewrite the destination port for "blocked" requests.
asylteltine 3y ago
I wouldn’t put something on the public internet without geoblocking China, Russia, and the UAE. You should too! Stop their bad behavior by removing them from the internet.
- chupapimunyenyo 3y ago
  This would harm the governments so bad
1970-01-01 3y ago
It's a crazy way of staying online, but it does work, therefore it's not crazy.
egorfine 3y ago
The approach is so wrong on so many levels I'm wondering if that's just a rage clickbait.
WhereIsTheTruth 3y ago
Nobody own their IP, banning IPs means banning innocent people tomorrow You should timeout IP for few days instead
gorkish 3y ago
Someone should tell this guy about bogans so he can block 500 million more IPs. And if you want to do the same? For the love of god get a firewall and subscribe to some RBLs like a sane person.
gmuslera 3y ago
There are some real bad actors behind IP blocks, or hosting providers that have no problem hosting them nor take actions on abuse reports. Referrer spamming, searching for vulnerabilities (some of them with very big URL list to try), misbehaving crawlers, or just plain DoS are some of the ways they may sites, specially the ones serving dynamic content. This space is usually fixed and used by servers, or VPNs exitpoits. Blocking all the blocks associated to their autonomous systems would avoid to put in the rules a lot of /24. But then there are residential IP blocks, specially some with dynamic enough IPs or NATed ISPs. Some people of those blocks may have hostile or clueless behaviour, some may be used as proxy because malware or because they intentionally installed some of the residential proxy servers agents. There you may be blocking legitimate visitors, if a few clients of some ISP are very active you may end blocking a lot of innocent people. And, in this case too, you can target the IP blocks of its autonomous system if you feel that from there you only get bad traffic. But in the end, is your site. you are free to decide to block what you understand that are bad neighbourhoods.
SushiHippie 3y ago
Could it be that the slight delay between opening this page and my browser receiving the first bytes is nginx checking these 50 million IPs? How is this delay so small if there are really 50 million deny statements? Is there a reason why they don't use a firewall?
- nijave 3y ago
  >How is this delay so small if there are really 50 million deny statements? More a testament to the years of optimization nginx has undergone. >Is there a reason why they don't use a firewall? Some other comment said they have a custom error page for blocked requests although I think you could still have the firewall rewrite the destination port and have a listener that serves the custom access denied page there.
6R1M0R4CL3 3y ago
well he should directly block everything and unblock those one by one. this would be more impressive to me :>

43 points by devinjon 3y ago

chupapimunyenyo 3y ago

I cant read this, im blocked

ziddoap 3y ago
Indeed, for whatever reason I appear to be blocked as well. Never even heard of this site, and I am not a bot. Just caught in the crossfire I guess. ¯\_(ツ)_/¯

devinjon 3y ago

"I imagine that if this article makes its way onto Hacker News, I will be criticized."

Bluecobra 3y ago
I think it’s a healthy thing to read other’s opinions, even if you disagree with it. HN does a good job of keeping the trolls out. Edit: FWIW I think you’re fighting the good fight and I wish I could implement something similar at work, in where I constantly have bots trying to login to my corporate VPN.
- WarOnPrivacy 3y ago
  > I think it’s a healthy thing to read other’s opinions, even if you disagree with it. HN does a good job of keeping the trolls out. I like to think world class nits are picked here but you're right - this is one of the more rational neighborhoods. > I wish I could implement something similar at work, in where I constantly have bots trying to login to my corporate VPN. I kludged some bash together to grep the logs and give me IP hit-counts and network owners. I next struggled to make decisions about the IPs - due to 1) the prevalence of residential proxies and 2) learning one of our widgets refreshes 3x/min.

citrin_ru 3y ago

Using http://nginx.org/r/deny http://nginx.org/r/deny is a very inefficient way to block large number of IP/nets. It mentioned right in the documentation: > In case of a lot of rules, the use of the ngx_http_geo_module module variables is preferable.

mergy 3y ago

I block as well with geoblocking and based on source behavior. As long as you understand the limitations, ramifications, and futility in doing this, I have no problems with it as one of the many tactics to defend your footprint and de-noise your logs, etc. It's a never ending endeavor and you will see the shifting attack sources from the baddies along with the games on stub blocks, prefix broker IP block swaps, and more. Just know there is automation and horsepower into that entire attack infrastructure you can't possibly compete with but maybe you can mitigate with the limited time and resources you have and that will be enough to get you through.

hifromwork 3y ago

>It means that more than one percent of the IPv4 real estate on the Internet (and probably much more) is occupied by people and organizations who are either clueless or just do not care how much the rest of us are paying to keep our websites on line There's a significant mental leap here. "I block these IP to conserve my resources, therefore they belong to clueless or malicious organisations". It's wrong in both directions: * I don't think Google, Bing and other crawlers are inherently malicious, and certainly not clueless. Search engines serve a very important role in the internet. Ditto archive.org, and probably dozens of other bots. * IP based blocklists work well for honest bots (not malicious, or at least not illegal). Malicious bot operators just buy SIM cards and use regular mobile internet for the crawling (basically unblockable, because the IP may be renewed every day or every hour). And the really malicious actors use residential proxies, i.e. botnets that proxy traffic through normal users' computers. Anyway I wonder how many of those 56MM IP addresses are regular dynamically allocated consumer grade ISPs. >1-5-2024 For the love of all that is holy, what is this date format.

WarOnPrivacy 3y ago
>>1-5-2024 >For the love of all that is holy, what is this date format. Most commonly used in the US. 'Cause we gotta show it like we say it. I think.
- wccrawford 3y ago
  With dashes? We almost always use slashes here in the US. The only time I see dashes is for ISO format, and that's year-first.
  - D13Fd 3y ago
    Yes. It’s less common but looks unremarkable to me.
  - WarOnPrivacy 3y ago
    Sure. It was the dominant written style throughout the 20th cent and folks kept on with it.

aledalgrande 3y ago

Blocking bots is always going to be an uphill battle. But if the owner is worried about wasting meagre resources, why not serving static HTML files instead of running a PHP server for a simple blog?

turtleyacht 3y ago

> I don't care. Just do your own thing, learn from it, and repeat. That's all we can really want in our projects, on our limited lease on planet Earth. Kudos that you found something to work on.

TekMol 3y ago

What is the issue they are trying to solve? It seems to be a static site. Bots should cause only a neglible amount of traffic per month. My guess would be less than $1. And aren't there free CDNs for static sites these days? I guess you can just push the whole frontent data (html+assets) into a public git repo, put it behind a github page with custom domain and call it a day?

jdietrich 3y ago
> What is the issue they are trying to solve? Apparently, he self-hosts on a Raspberry Pi 3B+. I guess you need to block half the internet if your server has about the same performance as a decade-old smartphone. It's charming in a way, like someone who daily-drives a classic car. https://cheapskatesguide.org/articles/self-hosters-nightmare.html https://cheapskatesguide.org/articles/self-hosters-nightmare...
- Filligree 3y ago
  > After a loss of power, my routers and web server must be restarted in a specific sequence for my home network to function properly. There's a certain charm to Rube Goldberg machines as well.
- miyuru 3y ago
  LOL, the first comment of that article. "To worldofmathew on Hacker News, While I do sincerely appreciate the fact that you have taken the time to post several cheapskatesguide articles to Hacker News, this may have caused them to designate you as a blog spammer and block your posts. In other words, they may think that you own this site and are posting your own articles. May I suggest that you contact a Hacker News moderator and set him straight before you post another cheapskatesguide article?"
montroser 3y ago
No, this craziness(!) here suggests that it's not a static site: > "I added hyphens to the opening and closing PHP statements to prevent my web server from interpreting them as code." It could be a static site, and one could also automate the job that this person spends their precious time doing every day, copying and pasting text (IP addresses) that matches a pattern from one place to another. But they seem content, so...

WarOnPrivacy 3y ago

I imagine that if this article makes its way onto Hacker News, I will be criticized. Maybe they will call me naive or compare me to Don Quixote fighting windmills. Maybe they will call me stupid or paranoid for not using some centralized block list. Maybe they will object to how I characterize those who employ web-crawling robots. Ha. He is so wrong. We're going to criticize his nginx config and his use of PHP. I mean yeah. We'll probably get after that other stuff too but still.

hsuduebc2 3y ago
PHP on hackernews also caught my attention.

Fb24k 3y ago

After reading his three part multi-month series about how he can't set up a firewall, I don't think this guy is probably someone who should be providing any useful information on how to use the internet (or anything attached to it).

justsomehnguy 3y ago
Now I read it too (well, skimmed). >> -A INPUT -j DROP Yep, it's all is needed to know about this guy.

justsomehnguy 3y ago

> It means that more than one percent of the IPv4 real estate on the Internet (and probably much more) is occupied by people and organizations who are either clueless or just do not care how much the rest of us are paying to keep our websites on line. Oh, tell me, how much? The whopping $5/month? Oh, maybe this is a high load WordPress/like CMS running on LAMP stack... so $8/month? > I wrote the following small PHP script to search though my Nginx configuration file and tally up the number of IP addresses that I am blocking. Holy shit. Blocking bots through nginx configuration, more so, blocking 56M addresses through nginx configuration... Okay, for those of you who never did the thing or have no idea: Just use the firewall (most of the time it is built-in in your OS), use some way to tell the firewall about the 'offenders' (eg fail2ban though there are options) and don't ever block something indefinitely, it's totally meaningless, just use timeouts. If some Bob got his computer infected in 2015 and that computer tried to access /wp-admin.php then there is absolutely no reason to assume what in 2024 the IP address Bob's computer had in 2015 is still 'malicious'. Automatic activity like the scans, bruteforcing and whatever is all about opportunity. They are searching for an easy opportunities to exploit and scanning a server what actively blocks you even for 30m at time is just pointless, there is way, way more opportunities in other places than wasting ~4 weeks trying to scan this server. > I have custom 403 and 404 error pages that explain to those who may care why they are being blocked and how to regain access to the website https://cheapskatesguide.org/custom404really.html https://cheapskatesguide.org/custom404really.html

Sweepi 3y ago
Maybe the custom "blocked" 403/404 error page only shows up if your IP is blocked?
- nijave 3y ago
  They could run a web server (or listener/virtual host) on a separate port and use a firewall to rewrite the destination port for "blocked" requests.

asylteltine 3y ago

I wouldn’t put something on the public internet without geoblocking China, Russia, and the UAE. You should too! Stop their bad behavior by removing them from the internet.

chupapimunyenyo 3y ago
This would harm the governments so bad

1970-01-01 3y ago

It's a crazy way of staying online, but it does work, therefore it's not crazy.

egorfine 3y ago

The approach is so wrong on so many levels I'm wondering if that's just a rage clickbait.

WhereIsTheTruth 3y ago

Nobody own their IP, banning IPs means banning innocent people tomorrow You should timeout IP for few days instead

gorkish 3y ago

Someone should tell this guy about bogans so he can block 500 million more IPs. And if you want to do the same? For the love of god get a firewall and subscribe to some RBLs like a sane person.

gmuslera 3y ago

There are some real bad actors behind IP blocks, or hosting providers that have no problem hosting them nor take actions on abuse reports. Referrer spamming, searching for vulnerabilities (some of them with very big URL list to try), misbehaving crawlers, or just plain DoS are some of the ways they may sites, specially the ones serving dynamic content. This space is usually fixed and used by servers, or VPNs exitpoits. Blocking all the blocks associated to their autonomous systems would avoid to put in the rules a lot of /24. But then there are residential IP blocks, specially some with dynamic enough IPs or NATed ISPs. Some people of those blocks may have hostile or clueless behaviour, some may be used as proxy because malware or because they intentionally installed some of the residential proxy servers agents. There you may be blocking legitimate visitors, if a few clients of some ISP are very active you may end blocking a lot of innocent people. And, in this case too, you can target the IP blocks of its autonomous system if you feel that from there you only get bad traffic. But in the end, is your site. you are free to decide to block what you understand that are bad neighbourhoods.

SushiHippie 3y ago

Could it be that the slight delay between opening this page and my browser receiving the first bytes is nginx checking these 50 million IPs? How is this delay so small if there are really 50 million deny statements? Is there a reason why they don't use a firewall?

nijave 3y ago
>How is this delay so small if there are really 50 million deny statements? More a testament to the years of optimization nginx has undergone. >Is there a reason why they don't use a firewall? Some other comment said they have a custom error page for blocked requests although I think you could still have the firewall rewrite the destination port and have a listener that serves the custom access denied page there.

6R1M0R4CL3 3y ago

well he should directly block everything and unblock those one by one. this would be more impressive to me :>