12 ms·

I self-host a (non-critical) mail server and a few other things and occasionally look at live firewall logs, seeing the constant flow of illegitimate traffic hi

I self-host a (non-critical) mail server and a few other things and occasionally look at live firewall logs, seeing the constant flow of illegitimate traffic hitting random ports all over the place, some hitting legitimate service ports but others just probing basically anything and everything. I decided to setup a series of scripts that detect activity on ports that aren't open (and therefore there's no legitimate reason for the traffic to exist) and block those IP addresses from the service ports since the traffic source isn't to be trusted. Something that came out of analysis of the blocked IP addresses was that I discovered a few untrustworthy /24 networks belonging to a bunch of "internet security companies" whose core business seems to depend on flooding the entire IPv4 space with daily scans. Blocking these Internet scanner networks significantly reduced the uninvited activity on my open service ports. And by significantly I mean easily over 50% of unwanted traffic is blocked. Network lists and various scripts to achieve my setup can be found here: https://github.com/UninvitedActivity/UninvitedActivity https://github.com/UninvitedActivity/UninvitedActivity Internet Scanner lists are here: https://github.com/UninvitedActivity/UninvitedActivity/tree/main/02_InternetScanners https://github.com/UninvitedActivity/UninvitedActivity/tree/... Large networks that seem responsible for more than their fair share of uninvited activity are listed here: https://github.com/UninvitedActivity/UninvitedActivity/tree/main/03_UninvitedNetworks https://github.com/UninvitedActivity/UninvitedActivity/tree/... I'm semi-aware of the futility of blocking IP addresses and networks. I do believe, however, that it can significantly reduce the load on the next layers of security that require computation for pattern matching etc. Be aware: there are footguns to be found here.

k8sToGo 2y ago
Have you considered using crowdsec?
- BLKNSLVR 2y ago
  I set it up in a fairly superficial way, and there are only a handful (two or three) rules that can be applied on the free tier, and I'm a tight-ass. It's still running, but it doesn't seem to block much - but that might be because I didn't put enough time into "doing it properly".
- teruakohatu 2y ago
  Are there any downsides to crowdsec?
  - snorremd 2y ago
    You end up sharing signals (IPs) to their crowd-sourced bad IP databases, but only get 3 free IP lists on the free plan. To get some of the bigger IP lists you need an enterprise plan at $2500 a month. Essentially they use the free customers to build the lists that drive their enterprise sales, which is fair enough as you get to use their free dashboard and open source software. But to me it seems they're really only targeting enterprise customers as a business.
    - Philippe_H 2y ago
      Hi all and @snorremd, (Philippe from the CrowdSec team) The $2.5K / month was for enterprise, but we didn't correctly understand the need and converted it to 2 optional prices: $1K for LTS and $1K for support. This will be reflected in an update on our pricing page this week; thanks, everyone, for your patience in this matter. It took us time to segment our four products properly. We wanted to avoid pivoting later, as it happened to so many other open-source tools recently. * The Security Engine (IDS+WAF+IPS) is for everyone. (Free / MIT license, three free blocklists) * Its SaaS companion is made for anyone with a security engine. (Generous free tier, $31/engine/month for pro industrialization features, 3 premium blocklists + all free ones. Volume discounts avail. We'll soon merge SecOPS and enterprise plans, all features at the price of the SecOPS plan) * Blocklists are made for M/L entities to use. (In the range of a few ten of K$ yearly, all blocklists, unlimited) * The Full CTI database is intended to be used by L/XL Corps. (It contains 32 fields about ~25M IP, with industry targeted, country targeted, tech stack targeted, AS and range reputation, etc. Local replication at your place, several updates/day. 10 to 20K$ / month, depending on some parameters) PS: As we did for the Olympic Games 2024, we'll also give away a blocklist for the US presidential election of the most aggressive IP against US assets. With a quarter of a million machines running CS, we have a fairly good overview of this, in real-time. Safer together.
pgraf 2y ago
Just be aware that with your strategy “blocking 50% of unwanted traffic” means blocking non-attack traffic, as these Internet security companies are mostly legitimate. The automated attack traffic that you actually want to block is in the other half and will frequently change IPs.
- BLKNSLVR 2y ago
  > these Internet security companies are mostly legitimate This is both subjective and highly dependent upon the scope of services being run. My setup would probably progressively create more hassle than it saves as on a scale from small business to large business. For the setup I have, I quite specifically want to block their traffic. I'm possibly overly militant about this, but they keep databases of the results of their scans, and their business is selling this information to ... whoever's buying. I don't want my IP addresses, open ports, services or any other details they're able to gather to be in these databases over which I have no control and didn't authorise. To steal an oft-used analogy, they're taking snapshots of all the houses on all the streets and identifying the doors, windows, gates, and having a peek inside, and recording all the results in a database. I believe all of them are illegitimate. They 'do' because they can, and it's profitable. "Making the internet safer" is not their raison d'être. Happy for any else to form their own opinion, but this is my current stance.
  - appstorelottery 2y ago
    Would be cool to have a "don't scan me bro" list of IP's that engage in this that we could share - is there such a thing?
    - BLKNSLVR 2y ago
      The problem is that becomes a concentrator of IPs behind which privacy conscious individuals exist, which probably has higher value to "whoever's buying". It's a conundrum.
      - yesbabyyes 2y ago
cranberryturkey 2y ago
Just install fail2ban.
- WhackyIdeas 2y ago
  For SSH, changing to a random port number resulted in zero connection attempts from bots for months on end. It seems bots just never bother scanning the full 65535 port range.
  - dizhn 2y ago
    For most of my VMs there's no ssh running. I use wireguard to connect to a private IP. I haven't done this on the bare metal yet but I might. Though barring exploits like we had recently nobody is getting into a server with either strong passwords or certificates. Fail2ban in my eyes is a log cleaner. It's not useful for much else.
    - cranberryturkey 2y ago
      it bans the bad ips, isn't that worth running?
      - thfuran 2y ago
        But what does that actually accomplish?
        cranberryturkey 2y ago
        stops the attack from happening from those ips?
  - account42 2y ago
    > the full 65535 port range Note that putting SSH on a high port has security implications.
    - 2y ago
nilsherzig 2y ago
Try running some of your blocked ips through greynoise, they usually have some interesting information about them
- BLKNSLVR 2y ago
  Thanks for the tip. Looks like greynoise use ipinfo.io for IP metadata. I use https://www.abuseipdb.com/ https://www.abuseipdb.com/ for any manual IP address checks, and https://hackertarget.com/as-ip-lookup/ https://hackertarget.com/as-ip-lookup/ for finding what ASN an IP address (range) is a member of. I'll check out greynoise and see what extra info may be provided.
  - reincoder 2y ago
    I (DevRel of IPinfo) run Fail2Ban on a VM as well. Protip use the CLI. - The CLI has the `grepip` command that extracts all the IP addresses from a text. You do not have to parse your logs. - Analyze your data. After you have extracted your IP addresses from your logs, pipe them to the `summarize`, `map`, and `bulk` commands on the CLI. - If you are doing bulk enrichment with the `bulk` command, you can use some kind of CSV query tool like CSVtoolkit, DuckDB, or Python-Pandas. - Look into the ASN data. ASN data is always going to be the more interesting IP metadata for honeypots IPs. Summarize the IP addresses with the `summarize` command; it will give you a high-level report. If you want a web-shareable report, make a POST call to that endpoint. Docs: https://ipinfo.io/tools/summarize-ips https://ipinfo.io/tools/summarize-ips https://github.com/ipinfo/cli https://github.com/ipinfo/cli You can always send your logs to me and ask what I think of them, and if I can find common patterns based on IP metadata. I am running our API and database services 24/7 and enjoy looking at logs. I can suggest firewall configurations based on country and ASN information provided by our free data.
shaky-carrousel 2y ago
Good idea. What I do is, I disallowed password login in my ssh server, and I permanently ban whichever address that tries to log in using a password.
- BLKNSLVR 2y ago
  I use a bastion host on a VPS as the only source IP address allowed to ssh into my systems, so any attempts to connect to ssh (from any IP address other than the bastion) are both blocked and logged into "the list" to be blocked from connecting to any other service ports.
- PokestarFan 2y ago
  I did this but added an "escape hatch" that allowed password logins from the local network only.
TacticalCoder 2y ago
One thing I do is I blocklist entire countries' and regional ISP' CIDR blocks. Believe it or not: straight to firewall DROP. China, North Korea, so many african countries who's only traffic is from scammers, tiny islands in the pacific that are used for nothing but scamming... Straight to DROP. And I do not care about the whining.
- nequo 2y ago
  I assume you don’t host anything that could be useful to the 1.5 to 2 billion people that you’re blocking.
  - luma 2y ago
    Or they host a business site that doesn't do business in those countries and so nothing of value is lost to them. For example, it's literally illegal for me to accept payments from .ru, so why bother wasting their time and my bandwidth?
    - ajsnigrutin 2y ago
      I live in EU,and a bunch of american sites just block the whole EU due to GDPR laws. Then someone in US uses my email by accident to subscribe to some newsletter (not the first time, I also get personal emails for that person, since it's just one letter difference, and i'm guessing it's someone old, considering the emails I get), i try to click "unsubscribe", and it just redirects me to "<site> is unavailable in EU, blah blah" page, without unsubscribing. I make sure to report that site to every goddamn spam list possible.
      - rapind 2y ago
        IMO replying unsubscribe should always work for marketing emails and if it doesn’t then I flag the email as spam. Nope, I’m not going to visit that tracked / info gathering unsubscribe link.
        2y ago
tomxor 2y ago
> and block those IP addresses from the service ports since the traffic source isn't to be trusted Don't get me wrong, I want to do the same, I run a lot of servers and see all the automated nonsense aimed at public servers. However, you should consider the fact that today blocking an IP is akin to blocking a street, a village or sometimes even a town. For ~better or~ worse we now live in the age of CGNAT. If your threat model and use case means you only care about a known subset of users with static IPs who are lucky enough to not share IPs then fair enough; but if you are running services intended for wide spread consumption you are likely blocking legitimate users without even knowing it.
- BLKNSLVR 2y ago
  I have thought about that and, as you say, my use-case is entirely "hobby" so there's nothing I host that's of much interest to others (if things break, which they have, it inconveniences me rather than other people). Having said that, the websites I host are behind Cloudflare and so port 443 allows Cloudflare's ASN, but blocks everything else. This way, any of the IP addresses that are blocked from direct access to port 443 can still access the websites, just through Cloudflare's added layer of protection.
Bengalilol 2y ago
I was about to say out loud that it was a (kind of) relief not finding Google in your lists, then I found https://github.com/UninvitedActivity/UninvitedActivity/blob/main/03_UninvitedNetworks/03_1e100Net.txt https://github.com/UninvitedActivity/UninvitedActivity/blob/...
- BLKNSLVR 2y ago
  I need to check my exact configuration, but whilst I've got 1e100 in a list, I think I've got an exception for it elsewhere. Ie. Whilst it's been detected as uninvited activity, it causes issues when blocked, so it's excluded from the blocking.
tredigi 2y ago
> and block those IP addresses from the service ports since the traffic source isn't to be trusted. This means that you are locking out anybody using a paid VPN service, if any other customer of that same VPN service does any kind of scan.
- BLKNSLVR 2y ago
  Something I didn't mention in my original comment, but have mentioned in another reply somewhere, is that I have the websites running behind Cloudflare, and I allow Cloudflare's ASN into port 443 but block everything else. Essentially outsourcing the security of port 443 to Cloudflare. My use-case is "hobby / enthusiast", so I believe I'm losing nothing and the "world at large" is losing nothing from this setup. Having said that, all policies on this kind of thing need to be strongly thought about in terms of their applicability to the use-case. Were I running a small or even medium business, I'd probably do it exactly the same with maybe a bit more of an eye on what's being blocked and the ownership of the IP addresses, and I'd have some stats to point to on the range of sources of legitimate traffic. It'd have to be a pretty big, international business for it to cause much of an effect (although I'm talking well out of school here because I don't have anything at stake). Flipside, though, I have my outgoing traffic routed through a couple of different exits, and I've had to make specific rules for some websites that block traffic from VPNs and VPSs, which is annoying, so I'm not completely dismissing your point. Lastly, however, at all scales I'd still block the Internet Scanners for reasons I've given elsewhere. Blocking them massively cut down on the uninvited activity - again, it's not about making clean logs, but it really helped clear a lot of the noise.

k8sToGo 2y ago

Have you considered using crowdsec?

BLKNSLVR 2y ago
I set it up in a fairly superficial way, and there are only a handful (two or three) rules that can be applied on the free tier, and I'm a tight-ass. It's still running, but it doesn't seem to block much - but that might be because I didn't put enough time into "doing it properly".
teruakohatu 2y ago
Are there any downsides to crowdsec?
- snorremd 2y ago
  You end up sharing signals (IPs) to their crowd-sourced bad IP databases, but only get 3 free IP lists on the free plan. To get some of the bigger IP lists you need an enterprise plan at $2500 a month. Essentially they use the free customers to build the lists that drive their enterprise sales, which is fair enough as you get to use their free dashboard and open source software. But to me it seems they're really only targeting enterprise customers as a business.
  - Philippe_H 2y ago
    Hi all and @snorremd, (Philippe from the CrowdSec team) The $2.5K / month was for enterprise, but we didn't correctly understand the need and converted it to 2 optional prices: $1K for LTS and $1K for support. This will be reflected in an update on our pricing page this week; thanks, everyone, for your patience in this matter. It took us time to segment our four products properly. We wanted to avoid pivoting later, as it happened to so many other open-source tools recently. * The Security Engine (IDS+WAF+IPS) is for everyone. (Free / MIT license, three free blocklists) * Its SaaS companion is made for anyone with a security engine. (Generous free tier, $31/engine/month for pro industrialization features, 3 premium blocklists + all free ones. Volume discounts avail. We'll soon merge SecOPS and enterprise plans, all features at the price of the SecOPS plan) * Blocklists are made for M/L entities to use. (In the range of a few ten of K$ yearly, all blocklists, unlimited) * The Full CTI database is intended to be used by L/XL Corps. (It contains 32 fields about ~25M IP, with industry targeted, country targeted, tech stack targeted, AS and range reputation, etc. Local replication at your place, several updates/day. 10 to 20K$ / month, depending on some parameters) PS: As we did for the Olympic Games 2024, we'll also give away a blocklist for the US presidential election of the most aggressive IP against US assets. With a quarter of a million machines running CS, we have a fairly good overview of this, in real-time. Safer together.

pgraf 2y ago

Just be aware that with your strategy “blocking 50% of unwanted traffic” means blocking non-attack traffic, as these Internet security companies are mostly legitimate. The automated attack traffic that you actually want to block is in the other half and will frequently change IPs.

BLKNSLVR 2y ago
> these Internet security companies are mostly legitimate This is both subjective and highly dependent upon the scope of services being run. My setup would probably progressively create more hassle than it saves as on a scale from small business to large business. For the setup I have, I quite specifically want to block their traffic. I'm possibly overly militant about this, but they keep databases of the results of their scans, and their business is selling this information to ... whoever's buying. I don't want my IP addresses, open ports, services or any other details they're able to gather to be in these databases over which I have no control and didn't authorise. To steal an oft-used analogy, they're taking snapshots of all the houses on all the streets and identifying the doors, windows, gates, and having a peek inside, and recording all the results in a database. I believe all of them are illegitimate. They 'do' because they can, and it's profitable. "Making the internet safer" is not their raison d'être. Happy for any else to form their own opinion, but this is my current stance.
- appstorelottery 2y ago
  Would be cool to have a "don't scan me bro" list of IP's that engage in this that we could share - is there such a thing?
  - BLKNSLVR 2y ago
    The problem is that becomes a concentrator of IPs behind which privacy conscious individuals exist, which probably has higher value to "whoever's buying". It's a conundrum.
    - yesbabyyes 2y ago

cranberryturkey 2y ago

Just install fail2ban.

WhackyIdeas 2y ago
For SSH, changing to a random port number resulted in zero connection attempts from bots for months on end. It seems bots just never bother scanning the full 65535 port range.
- dizhn 2y ago
  For most of my VMs there's no ssh running. I use wireguard to connect to a private IP. I haven't done this on the bare metal yet but I might. Though barring exploits like we had recently nobody is getting into a server with either strong passwords or certificates. Fail2ban in my eyes is a log cleaner. It's not useful for much else.
  - cranberryturkey 2y ago
    it bans the bad ips, isn't that worth running?
    - thfuran 2y ago
      But what does that actually accomplish?
      - cranberryturkey 2y ago
        stops the attack from happening from those ips?
- account42 2y ago
  > the full 65535 port range Note that putting SSH on a high port has security implications.
  - 2y ago

nilsherzig 2y ago

Try running some of your blocked ips through greynoise, they usually have some interesting information about them

BLKNSLVR 2y ago
Thanks for the tip. Looks like greynoise use ipinfo.io for IP metadata. I use https://www.abuseipdb.com/ https://www.abuseipdb.com/ for any manual IP address checks, and https://hackertarget.com/as-ip-lookup/ https://hackertarget.com/as-ip-lookup/ for finding what ASN an IP address (range) is a member of. I'll check out greynoise and see what extra info may be provided.
- reincoder 2y ago
  I (DevRel of IPinfo) run Fail2Ban on a VM as well. Protip use the CLI. - The CLI has the `grepip` command that extracts all the IP addresses from a text. You do not have to parse your logs. - Analyze your data. After you have extracted your IP addresses from your logs, pipe them to the `summarize`, `map`, and `bulk` commands on the CLI. - If you are doing bulk enrichment with the `bulk` command, you can use some kind of CSV query tool like CSVtoolkit, DuckDB, or Python-Pandas. - Look into the ASN data. ASN data is always going to be the more interesting IP metadata for honeypots IPs. Summarize the IP addresses with the `summarize` command; it will give you a high-level report. If you want a web-shareable report, make a POST call to that endpoint. Docs: https://ipinfo.io/tools/summarize-ips https://ipinfo.io/tools/summarize-ips https://github.com/ipinfo/cli https://github.com/ipinfo/cli You can always send your logs to me and ask what I think of them, and if I can find common patterns based on IP metadata. I am running our API and database services 24/7 and enjoy looking at logs. I can suggest firewall configurations based on country and ASN information provided by our free data.

shaky-carrousel 2y ago

Good idea. What I do is, I disallowed password login in my ssh server, and I permanently ban whichever address that tries to log in using a password.

BLKNSLVR 2y ago
I use a bastion host on a VPS as the only source IP address allowed to ssh into my systems, so any attempts to connect to ssh (from any IP address other than the bastion) are both blocked and logged into "the list" to be blocked from connecting to any other service ports.
PokestarFan 2y ago
I did this but added an "escape hatch" that allowed password logins from the local network only.

TacticalCoder 2y ago

One thing I do is I blocklist entire countries' and regional ISP' CIDR blocks. Believe it or not: straight to firewall DROP. China, North Korea, so many african countries who's only traffic is from scammers, tiny islands in the pacific that are used for nothing but scamming... Straight to DROP. And I do not care about the whining.

nequo 2y ago
I assume you don’t host anything that could be useful to the 1.5 to 2 billion people that you’re blocking.
- luma 2y ago
  Or they host a business site that doesn't do business in those countries and so nothing of value is lost to them. For example, it's literally illegal for me to accept payments from .ru, so why bother wasting their time and my bandwidth?
  - ajsnigrutin 2y ago
    I live in EU,and a bunch of american sites just block the whole EU due to GDPR laws. Then someone in US uses my email by accident to subscribe to some newsletter (not the first time, I also get personal emails for that person, since it's just one letter difference, and i'm guessing it's someone old, considering the emails I get), i try to click "unsubscribe", and it just redirects me to "<site> is unavailable in EU, blah blah" page, without unsubscribing. I make sure to report that site to every goddamn spam list possible.
    - rapind 2y ago
      IMO replying unsubscribe should always work for marketing emails and if it doesn’t then I flag the email as spam. Nope, I’m not going to visit that tracked / info gathering unsubscribe link.
      - 2y ago

tomxor 2y ago

> and block those IP addresses from the service ports since the traffic source isn't to be trusted Don't get me wrong, I want to do the same, I run a lot of servers and see all the automated nonsense aimed at public servers. However, you should consider the fact that today blocking an IP is akin to blocking a street, a village or sometimes even a town. For ~better or~ worse we now live in the age of CGNAT. If your threat model and use case means you only care about a known subset of users with static IPs who are lucky enough to not share IPs then fair enough; but if you are running services intended for wide spread consumption you are likely blocking legitimate users without even knowing it.

BLKNSLVR 2y ago
I have thought about that and, as you say, my use-case is entirely "hobby" so there's nothing I host that's of much interest to others (if things break, which they have, it inconveniences me rather than other people). Having said that, the websites I host are behind Cloudflare and so port 443 allows Cloudflare's ASN, but blocks everything else. This way, any of the IP addresses that are blocked from direct access to port 443 can still access the websites, just through Cloudflare's added layer of protection.

Bengalilol 2y ago

I was about to say out loud that it was a (kind of) relief not finding Google in your lists, then I found https://github.com/UninvitedActivity/UninvitedActivity/blob/main/03_UninvitedNetworks/03_1e100Net.txt https://github.com/UninvitedActivity/UninvitedActivity/blob/...

BLKNSLVR 2y ago
I need to check my exact configuration, but whilst I've got 1e100 in a list, I think I've got an exception for it elsewhere. Ie. Whilst it's been detected as uninvited activity, it causes issues when blocked, so it's excluded from the blocking.

tredigi 2y ago

> and block those IP addresses from the service ports since the traffic source isn't to be trusted. This means that you are locking out anybody using a paid VPN service, if any other customer of that same VPN service does any kind of scan.

BLKNSLVR 2y ago
Something I didn't mention in my original comment, but have mentioned in another reply somewhere, is that I have the websites running behind Cloudflare, and I allow Cloudflare's ASN into port 443 but block everything else. Essentially outsourcing the security of port 443 to Cloudflare. My use-case is "hobby / enthusiast", so I believe I'm losing nothing and the "world at large" is losing nothing from this setup. Having said that, all policies on this kind of thing need to be strongly thought about in terms of their applicability to the use-case. Were I running a small or even medium business, I'd probably do it exactly the same with maybe a bit more of an eye on what's being blocked and the ownership of the IP addresses, and I'd have some stats to point to on the range of sources of legitimate traffic. It'd have to be a pretty big, international business for it to cause much of an effect (although I'm talking well out of school here because I don't have anything at stake). Flipside, though, I have my outgoing traffic routed through a couple of different exits, and I've had to make specific rules for some websites that block traffic from VPNs and VPSs, which is annoying, so I'm not completely dismissing your point. Lastly, however, at all scales I'd still block the Internet Scanners for reasons I've given elsewhere. Blocking them massively cut down on the uninvited activity - again, it's not about making clean logs, but it really helped clear a lot of the noise.