4 ms·

A recent abrupt change in Internet SSH brute force attacks against us

53 points by rolph 3y ago

Jun8 3y ago
This would be a great intro scene in a movie about hackers: the ominous drop off in attacks noticed by a random admin due to all bots gathering their strength to deliver a huge tsunami of attacks agains the US power grid.
- jstarfish 3y ago
  I don't put much stock in attribution, but keeping an eye on what attacks stop when an adversary ends up refocusing their attention due to scandal and/or war is a pretty reliable indicator in my experience (and very difficult to false-flag). Activity we previously attributed to the Russians didn't flicker when Russia invaded Ukraine-- but did stop very quickly into Israel's invasion of Gaza.
  - anonomousename 3y ago
    If true, that seems like some solid evidence. Do you have any links/sources I can read up on? I wasn’t able to find anything.
    - appplication 3y ago
      This type of analysis quickly starts to wade into “intelligence” in the CIA sense of the word. It’s possible to do some of this digging on your own, but for the most part this type of information ends up requiring a lot of inference and synthesis across different sources and collection mechanisms to build a coherent story. If there’s one thing I learned from working in this space, it’s how there’s just massive amounts of “history” playing out every day that will never be written down or acknowledged. In particular, cyber/electronic warfare is a very active space, and quite a few nation states regularly commit what many of us would imagine would be considered acts of war against each other, without a word said to the public.
    - jstarfish 3y ago
      It's not evidence at all, just data (intelligence) that makes me reconsider what I thought I knew about the situation and what else might be related that I didn't consider before. We have a "special" relationship with Israel so I can't go too much into detail, but suffice to say it was password spraying attacks that originated from domestic residential IPs that dropped off. Normally foreign agencies use datacenters and a known set of VPN ASNs. Israel happens to have their own onion routing network in the form of Hola/Luminati, but that isn't a discrete ASN-- it's a botnet of residential proxies. https://news.ycombinator.com/item?id=18161706 https://news.ycombinator.com/item?id=18161706 I don't know if Luminati is even still a thing but this is the sort of footprint I'd expect from it. They'd find residential proxies useful for their astroturfing campaigns so I assume it's still up. Attribution is a game of educated guesses. Now, I'm not implying the Israeli government is the actor here. For all I know it's some bored teenager fucking with us. The timing is what's suspect. Either the operator was compelled to stop when war broke out or the infrastructure they were using was somehow impacted by the Gaza offensive.
  - 15457345234 3y ago
    > Activity we previously attributed to the Russians didn't flicker when Russia invaded Ukraine-- but did stop very quickly into Israel's invasion of Gaza. What's your conclusion here, though? Who do you believe now was the actual bad actor and why did they stop? You're just leaving it open.
    - jstarfish 3y ago
      I kinda have to leave it open. I can't rightfully say for certain it's Mossad. It looks like it's related to Israel's offensive, but even that is subject to interpretation. It remains an open question to me too. The most I'd feel comfortable speculating on is that the attacks were possibly leveraging Israeli infrastructure (Luminati?), which says little about the actors involved. There's a secondary insider event that occurred coincidental to the beginning of the attacks against us that makes me particularly suspicious of our "friends" but I absolutely cannot discuss that here.
  - g-b-r 3y ago
    SSH attacks seem actually to have clearly spiked in the wake of Ukraine's invasion https://www.bsdly.net/~peter/Predicting_developments_in_real_world_conflict_from_patterns_of_failed_logins.html https://www.bsdly.net/~peter/Predicting_developments_in_real...
    - jstarfish 3y ago
      Interesting. We don't run public SSH so it wasn't a consideration.
20after4 3y ago
> "The U.S. Justice Department confirmed on Wednesday that it disrupted a botnet run by a prolific Chinese government hacking operation known as Volt Typhoon." https://therecord.media/china-run-botnet-takedown-fbi-doj-routers https://therecord.media/china-run-botnet-takedown-fbi-doj-ro...
nubinetwork 3y ago
Must be nice, I haven't noticed that much of a change on my end.
anthony_416 3y ago
Back in August of 2023, I would gets *hundreds* of SSH login attempts per minute. Looking at my logs, the numbers have dropping precipitously ever since. As of this month, I only received 17 attempts. All from the same IP range. Can't say I can complain about this.
- awaythrow999 3y ago
  Moving from 22 to a nonstandard port did miracles to avoid this. Really think port knocking could help thwart this entire category of attacks too.
  - anthony_416 3y ago
    Im using a non standard port in the 40,000 range. Changing the port won't necessarily stop the traffic. A lot of these bot nets will portscan an entire ip of anything that has port 80 open to find non standard ports.
    - pitched 3y ago
      It is much more expensive to port scan against UDP than TCP so a nonstandard UDP port would be a lot safer again. OpenVPN is one of the only services that let you do that though.
      - felurx 3y ago
        I'm pretty sure you can put Wireguard on any UDP port you desire, too.
  - scrps 3y ago
    I run ssh on an overlay network so it is never directly exposed to the internet and it is nice to not have to worry about changing ports, fail2ban, knocking. Of course you don't get to do fun analytics on the bots though.
udev4096 3y ago
Why are they not using fail2ban?
- csdreamer7 3y ago
  Where did they say they did not use fail2ban?
  - udev4096 3y ago
    I don't see any mention of it in the post
- rlpb 3y ago
  Here's a good list of reasons: https://www.opencve.io/cve?vendor=fail2ban https://www.opencve.io/cve?vendor=fail2ban Parsing log files not intended to be read by automation (eg. questionable or nonexistent escaping) to then take actions as root is with great remote injection vector is not a good idea.
  - 15457345234 3y ago
    That's a very very small list of vulnerabilities for such a widely used piece of software. Using fail2ban with SSH is very widely done and is thoroughly recommended. It increases security and is objectively a good idea.
    - rlpb 3y ago
      > That's a very very small list of vulnerabilities for such a widely used piece of software. They are schoolboy errors in a relatively trivial piece of software that very well demonstrate the fundamental security flaw in its design. > is thoroughly recommended Recommended by whom? The Internet, or prominent experts in the field?
      - 15457345234 3y ago
        > They are schoolboy errors in a relatively trivial piece of software that very well demonstrate the fundamental security flaw in its design. I read through the CVEs and they're not 'schoolboy errors' at all - in fact, the first one doesn't even refer to fail2ban itself but to a program being called by fail2ban. There's no fundamental security flaw in its design. > Recommended by whom? The Internet, or prominent experts in the field? It's recommended in several linux/unix books - physical books, written by experts - and in many blogs. So, both.
petee 3y ago
I see steady, slow attempts punctuated by some rapid attempts. But I'm still amused with their wasted efforts by not checking error messages; my servers only accept keys
based2 3y ago
https://www.abuseipdb.com/ https://www.abuseipdb.com/ https://www.misp-project.org/ https://www.misp-project.org/ https://www.crowdsec.net/ https://www.crowdsec.net/

53 points by rolph 3y ago

Jun8 3y ago

This would be a great intro scene in a movie about hackers: the ominous drop off in attacks noticed by a random admin due to all bots gathering their strength to deliver a huge tsunami of attacks agains the US power grid.

jstarfish 3y ago
I don't put much stock in attribution, but keeping an eye on what attacks stop when an adversary ends up refocusing their attention due to scandal and/or war is a pretty reliable indicator in my experience (and very difficult to false-flag). Activity we previously attributed to the Russians didn't flicker when Russia invaded Ukraine-- but did stop very quickly into Israel's invasion of Gaza.
- anonomousename 3y ago
  If true, that seems like some solid evidence. Do you have any links/sources I can read up on? I wasn’t able to find anything.
  - appplication 3y ago
    This type of analysis quickly starts to wade into “intelligence” in the CIA sense of the word. It’s possible to do some of this digging on your own, but for the most part this type of information ends up requiring a lot of inference and synthesis across different sources and collection mechanisms to build a coherent story. If there’s one thing I learned from working in this space, it’s how there’s just massive amounts of “history” playing out every day that will never be written down or acknowledged. In particular, cyber/electronic warfare is a very active space, and quite a few nation states regularly commit what many of us would imagine would be considered acts of war against each other, without a word said to the public.
  - jstarfish 3y ago
    It's not evidence at all, just data (intelligence) that makes me reconsider what I thought I knew about the situation and what else might be related that I didn't consider before. We have a "special" relationship with Israel so I can't go too much into detail, but suffice to say it was password spraying attacks that originated from domestic residential IPs that dropped off. Normally foreign agencies use datacenters and a known set of VPN ASNs. Israel happens to have their own onion routing network in the form of Hola/Luminati, but that isn't a discrete ASN-- it's a botnet of residential proxies. https://news.ycombinator.com/item?id=18161706 https://news.ycombinator.com/item?id=18161706 I don't know if Luminati is even still a thing but this is the sort of footprint I'd expect from it. They'd find residential proxies useful for their astroturfing campaigns so I assume it's still up. Attribution is a game of educated guesses. Now, I'm not implying the Israeli government is the actor here. For all I know it's some bored teenager fucking with us. The timing is what's suspect. Either the operator was compelled to stop when war broke out or the infrastructure they were using was somehow impacted by the Gaza offensive.
- 15457345234 3y ago
  > Activity we previously attributed to the Russians didn't flicker when Russia invaded Ukraine-- but did stop very quickly into Israel's invasion of Gaza. What's your conclusion here, though? Who do you believe now was the actual bad actor and why did they stop? You're just leaving it open.
  - jstarfish 3y ago
    I kinda have to leave it open. I can't rightfully say for certain it's Mossad. It looks like it's related to Israel's offensive, but even that is subject to interpretation. It remains an open question to me too. The most I'd feel comfortable speculating on is that the attacks were possibly leveraging Israeli infrastructure (Luminati?), which says little about the actors involved. There's a secondary insider event that occurred coincidental to the beginning of the attacks against us that makes me particularly suspicious of our "friends" but I absolutely cannot discuss that here.
- g-b-r 3y ago
  SSH attacks seem actually to have clearly spiked in the wake of Ukraine's invasion https://www.bsdly.net/~peter/Predicting_developments_in_real_world_conflict_from_patterns_of_failed_logins.html https://www.bsdly.net/~peter/Predicting_developments_in_real...
  - jstarfish 3y ago
    Interesting. We don't run public SSH so it wasn't a consideration.

20after4 3y ago

> "The U.S. Justice Department confirmed on Wednesday that it disrupted a botnet run by a prolific Chinese government hacking operation known as Volt Typhoon." https://therecord.media/china-run-botnet-takedown-fbi-doj-routers https://therecord.media/china-run-botnet-takedown-fbi-doj-ro...

nubinetwork 3y ago

Must be nice, I haven't noticed that much of a change on my end.

anthony_416 3y ago

Back in August of 2023, I would gets *hundreds* of SSH login attempts per minute. Looking at my logs, the numbers have dropping precipitously ever since. As of this month, I only received 17 attempts. All from the same IP range. Can't say I can complain about this.

awaythrow999 3y ago
Moving from 22 to a nonstandard port did miracles to avoid this. Really think port knocking could help thwart this entire category of attacks too.
- anthony_416 3y ago
  Im using a non standard port in the 40,000 range. Changing the port won't necessarily stop the traffic. A lot of these bot nets will portscan an entire ip of anything that has port 80 open to find non standard ports.
  - pitched 3y ago
    It is much more expensive to port scan against UDP than TCP so a nonstandard UDP port would be a lot safer again. OpenVPN is one of the only services that let you do that though.
    - felurx 3y ago
      I'm pretty sure you can put Wireguard on any UDP port you desire, too.
- scrps 3y ago
  I run ssh on an overlay network so it is never directly exposed to the internet and it is nice to not have to worry about changing ports, fail2ban, knocking. Of course you don't get to do fun analytics on the bots though.

udev4096 3y ago

Why are they not using fail2ban?

csdreamer7 3y ago
Where did they say they did not use fail2ban?
- udev4096 3y ago
  I don't see any mention of it in the post
rlpb 3y ago
Here's a good list of reasons: https://www.opencve.io/cve?vendor=fail2ban https://www.opencve.io/cve?vendor=fail2ban Parsing log files not intended to be read by automation (eg. questionable or nonexistent escaping) to then take actions as root is with great remote injection vector is not a good idea.
- 15457345234 3y ago
  That's a very very small list of vulnerabilities for such a widely used piece of software. Using fail2ban with SSH is very widely done and is thoroughly recommended. It increases security and is objectively a good idea.
  - rlpb 3y ago
    > That's a very very small list of vulnerabilities for such a widely used piece of software. They are schoolboy errors in a relatively trivial piece of software that very well demonstrate the fundamental security flaw in its design. > is thoroughly recommended Recommended by whom? The Internet, or prominent experts in the field?
    - 15457345234 3y ago
      > They are schoolboy errors in a relatively trivial piece of software that very well demonstrate the fundamental security flaw in its design. I read through the CVEs and they're not 'schoolboy errors' at all - in fact, the first one doesn't even refer to fail2ban itself but to a program being called by fail2ban. There's no fundamental security flaw in its design. > Recommended by whom? The Internet, or prominent experts in the field? It's recommended in several linux/unix books - physical books, written by experts - and in many blogs. So, both.

petee 3y ago

I see steady, slow attempts punctuated by some rapid attempts. But I'm still amused with their wasted efforts by not checking error messages; my servers only accept keys

based2 3y ago

https://www.abuseipdb.com/ https://www.abuseipdb.com/ https://www.misp-project.org/ https://www.misp-project.org/ https://www.crowdsec.net/ https://www.crowdsec.net/