4 ms·

All of my VPS servers have been slammed in the last 5 days with bruteforce SSH attacks. I've never seen anything this aggressive. One of my servers got PWNed, w

All of my VPS servers have been slammed in the last 5 days with bruteforce SSH attacks. I've never seen anything this aggressive. One of my servers got PWNed, which has never happened to me before. There's definitely some wild, shady shit going on.

pfundstein 7y ago
I highly recommend installing fail2ban to automatically firewall IPs with consecutive failed attempts, or if possible, disable password authentication altogether and use key auth.
- stevendgarcia 7y ago
  Solid advice. I had fail2ban installed and enabled. SSHD Root/password login turned off and only ssh key had access. My firewall was also airtight, or so I thought. Clearly I mucked up a config or setting somewhere because the odds of someone getting past all that are extremely low. One thing I had not prepared for was IP spoofing which I learned can be prevented with a few net.ipv4.conf tweaks. I also just purchased a static IP from my provider so I can lock down ssh access even further. Here's hoping I never have to deal with this headache again! fingers crossed
  - acegopher 7y ago
    What was the vector by which they gained access? I have fail2ban, password login turned off, key access, airtight firewall, etc. and now am worried.
    - cbluth 7y ago
      Me too
    - steve19 7y ago
      If they exploited a service or Web app they might have gotten shell access. The chance of gaining access through ssh with fail2ban is extremely unlikely, unless fail2ban was badly configured.
  - pfundstein 7y ago
    How did you discover the breach, and did you determine the vector? My guess is that it was a pivoted breach from another system on the LAN such as your PC.
    - stevendgarcia 7y ago
      I'm still picking up the pieces but from my logs I can see that hundreds of successive login attempts were made from different IPs, effectively circumventing fail2ban with what I can only assume is some form of automated IP spoofing. I'm hoping that strict ipv4 settings and ssh ip range restrictions will mitigate this in the future. I also used this python script to harden my SSH security with better algorithms. https://github.com/arthepsy/ssh-audit https://github.com/arthepsy/ssh-audit
      - nickphx 7y ago
        No, you were not seeing spoofed traffic. There are that many compromised machines actively scanning.
        stevendgarcia 7y ago
        It's scary to admit this but you are probably right. The first thing these bots do is use server resources to scan ports and brute force their way into other machines. I don't want to think about how many machines are pwned like this. Very sobering!
        snazz 7y ago
        This is also perfectly normal for the Internet, yes? If you have a server with an IPv4 address, expect many attempts per day.
  - OJFord 7y ago
    > One thing I had not prepared for was IP spoofing which I learned can be prevented with a few net.ipv4.conf tweaks. Do you have a handy link for more info about that? `rp_filter`? https://www.slashroot.in/linux-kernel-rpfilter-settings-reverse-path-filtering https://www.slashroot.in/linux-kernel-rpfilter-settings-reve...
  - xorcist 7y ago
    You don't generally "get past" your firewall rules and into your box unless you have accounts that are not password protected. If you really had password logins turned off, you need to identify and isolate how they gained access before you put that box online again. Never "hope" or "cross fingers" that it doesn't happen again. Unless you are an interesting target for some reason, chances are that these attacks are automated and you are running some insecure software somewhere. Start by taking a snapshot of the machine before you do anything else. Go through the logs. Are there any unwanted processes? How were they started? Are there any unwanted binaries in the filesystem? How were they uploaded? Try to find IP addresses that that be tied to any unwanted login, and see search your logs for any previous occurrences. Pay special attention to any web-reachable software you have installed.
  - ghostpepper 7y ago
    Did you have password login disabled for all accounts or just the root account?
- polyglotPirate 7y ago
  Best advice here!
- eridius 7y ago
  I don't trust key-only auth; what if I need to access my machine from a new computer I haven't done this with before? Is there any way to configure SSH to use a custom high-entry password that's different from the user's local password? My local password is something reasonable for me to type regularly (e.g. for sudo prompts), but I'd love to have a super long password just for SSH that I have to copy from my password manager each time.
  - dahfizz 7y ago
    This could be done simply by creating a new user with your super long password. Lock down ssh so you can only log in as this user, and let them `sudo su` into your normal working user.
    - eridius 7y ago
      I mean, yeah, but I don't want to do that. Partially because I don't want the friction, and partially because that won't work with any other tools that tunnel over ssh (e.g. sftp).
      - eitland 7y ago
        There will be friction, but sftp and port tunneling - which are my most used fwatures besides plain ssh - should be possible? I mean either you sftp to a shared folder that can be accessed from your regular user as well or you use a staging area with a cron job (or you load/unload the staging area manually.)
        eridius 7y ago
        If I'm sftp'ing to my server it's because I want to access my files. Not a special shared folder that I then have to separately ssh in, su to the real user, and move into place. I'm not deploying a website so a staging area isn't applicable. This is just a VPS that I use for various purposes.
        7y ago
- avian 7y ago
  +1 for disabling password auth. On the other hand I find fail2ban pretty useless these days. Most attacks I see seem to come from botnets where you only get a few requests from each IP.
- rorykoehler 7y ago
  You can also setup a VPN and limit all port 22 activity to your VPNs IP.
  - stevendgarcia 7y ago
    Yep. I just paid my ISP for a static IP address to accomplish this exact thing.
  - pbhjpbhj 7y ago
    I know it's just an obscurity measure but step one is surely don't expose SSH on 22? Dropped my (home) attacks/scans to zero from hundreds per day.
    - rorykoehler 7y ago
      You could do that but if you become a target due to unexpected viral growth then it's useless.
      - pbhjpbhj 7y ago
        It's more "stops you appearing on Shodan" level. But surely if you can reduce the traffic then you are more likely to notice proper attacks.
        rorykoehler 7y ago
        Shodan is amazing. Creepy too. I used to spend hours trawling video feeds on there. Fascinating.
dharmab 7y ago
You should disable password auth entirely and whitelist the IPs you connect from in your firewall. If you need to conenct from a large range of possible IPs, use a bastion host with 2FA and restricted to IP blocks from countries you actually connect from.

pfundstein 7y ago

I highly recommend installing fail2ban to automatically firewall IPs with consecutive failed attempts, or if possible, disable password authentication altogether and use key auth.

stevendgarcia 7y ago
Solid advice. I had fail2ban installed and enabled. SSHD Root/password login turned off and only ssh key had access. My firewall was also airtight, or so I thought. Clearly I mucked up a config or setting somewhere because the odds of someone getting past all that are extremely low. One thing I had not prepared for was IP spoofing which I learned can be prevented with a few net.ipv4.conf tweaks. I also just purchased a static IP from my provider so I can lock down ssh access even further. Here's hoping I never have to deal with this headache again! fingers crossed
- acegopher 7y ago
  What was the vector by which they gained access? I have fail2ban, password login turned off, key access, airtight firewall, etc. and now am worried.
  - cbluth 7y ago
    Me too
  - steve19 7y ago
    If they exploited a service or Web app they might have gotten shell access. The chance of gaining access through ssh with fail2ban is extremely unlikely, unless fail2ban was badly configured.
- pfundstein 7y ago
  How did you discover the breach, and did you determine the vector? My guess is that it was a pivoted breach from another system on the LAN such as your PC.
  - stevendgarcia 7y ago
    I'm still picking up the pieces but from my logs I can see that hundreds of successive login attempts were made from different IPs, effectively circumventing fail2ban with what I can only assume is some form of automated IP spoofing. I'm hoping that strict ipv4 settings and ssh ip range restrictions will mitigate this in the future. I also used this python script to harden my SSH security with better algorithms. https://github.com/arthepsy/ssh-audit https://github.com/arthepsy/ssh-audit
    - nickphx 7y ago
      No, you were not seeing spoofed traffic. There are that many compromised machines actively scanning.
      - stevendgarcia 7y ago
        It's scary to admit this but you are probably right. The first thing these bots do is use server resources to scan ports and brute force their way into other machines. I don't want to think about how many machines are pwned like this. Very sobering!
        snazz 7y ago
        This is also perfectly normal for the Internet, yes? If you have a server with an IPv4 address, expect many attempts per day.
- OJFord 7y ago
  > One thing I had not prepared for was IP spoofing which I learned can be prevented with a few net.ipv4.conf tweaks. Do you have a handy link for more info about that? `rp_filter`? https://www.slashroot.in/linux-kernel-rpfilter-settings-reverse-path-filtering https://www.slashroot.in/linux-kernel-rpfilter-settings-reve...
- xorcist 7y ago
  You don't generally "get past" your firewall rules and into your box unless you have accounts that are not password protected. If you really had password logins turned off, you need to identify and isolate how they gained access before you put that box online again. Never "hope" or "cross fingers" that it doesn't happen again. Unless you are an interesting target for some reason, chances are that these attacks are automated and you are running some insecure software somewhere. Start by taking a snapshot of the machine before you do anything else. Go through the logs. Are there any unwanted processes? How were they started? Are there any unwanted binaries in the filesystem? How were they uploaded? Try to find IP addresses that that be tied to any unwanted login, and see search your logs for any previous occurrences. Pay special attention to any web-reachable software you have installed.
- ghostpepper 7y ago
  Did you have password login disabled for all accounts or just the root account?
polyglotPirate 7y ago
Best advice here!
eridius 7y ago
I don't trust key-only auth; what if I need to access my machine from a new computer I haven't done this with before? Is there any way to configure SSH to use a custom high-entry password that's different from the user's local password? My local password is something reasonable for me to type regularly (e.g. for sudo prompts), but I'd love to have a super long password just for SSH that I have to copy from my password manager each time.
- dahfizz 7y ago
  This could be done simply by creating a new user with your super long password. Lock down ssh so you can only log in as this user, and let them `sudo su` into your normal working user.
  - eridius 7y ago
    I mean, yeah, but I don't want to do that. Partially because I don't want the friction, and partially because that won't work with any other tools that tunnel over ssh (e.g. sftp).
    - eitland 7y ago
      There will be friction, but sftp and port tunneling - which are my most used fwatures besides plain ssh - should be possible? I mean either you sftp to a shared folder that can be accessed from your regular user as well or you use a staging area with a cron job (or you load/unload the staging area manually.)
      - eridius 7y ago
        If I'm sftp'ing to my server it's because I want to access my files. Not a special shared folder that I then have to separately ssh in, su to the real user, and move into place. I'm not deploying a website so a staging area isn't applicable. This is just a VPS that I use for various purposes.
        7y ago
avian 7y ago
+1 for disabling password auth. On the other hand I find fail2ban pretty useless these days. Most attacks I see seem to come from botnets where you only get a few requests from each IP.
rorykoehler 7y ago
You can also setup a VPN and limit all port 22 activity to your VPNs IP.
- stevendgarcia 7y ago
  Yep. I just paid my ISP for a static IP address to accomplish this exact thing.
- pbhjpbhj 7y ago
  I know it's just an obscurity measure but step one is surely don't expose SSH on 22? Dropped my (home) attacks/scans to zero from hundreds per day.
  - rorykoehler 7y ago
    You could do that but if you become a target due to unexpected viral growth then it's useless.
    - pbhjpbhj 7y ago
      It's more "stops you appearing on Shodan" level. But surely if you can reduce the traffic then you are more likely to notice proper attacks.
      - rorykoehler 7y ago
        Shodan is amazing. Creepy too. I used to spend hours trawling video feeds on there. Fascinating.

dharmab 7y ago

You should disable password auth entirely and whitelist the IPs you connect from in your firewall. If you need to conenct from a large range of possible IPs, use a bastion host with 2FA and restricted to IP blocks from countries you actually connect from.