16 ms·
What SSH Hacking Attempts Look Like
- lol768 8y agoFairly standard stuff, definitely interesting to see all the IoT credentials attempred. I'd recommend SSHGuard over fail2ban though, I seem to remember the version of fail2ban in the Debian repos completely choking on IPv6 and failing open which is obviously undesirable.
- sametmax 8y agoFail2ban failing on ipv6 is strange given it doesn't do anything networky. It just analyses logs and updates iptable.
- iforgotpassword 8y agoCalling "iptables" with a v6 address doesn't sound too clever actually.
- tinix 8y agoInstalling "iptables" also installs ip6tables. https://linux.die.net/man/8/ip6tables https://linux.die.net/man/8/ip6tables
- h1d 8y agoMy experience with SSHGuard was it treated monitoring events (specifically from monit) as failed attempts and blocked further attempts marking the service as unreachable. I'm sure this can be configured but fail2ban does not block such attempts.
- jazoom 8y agoI feel like fail2ban failing open is a good thing. If it fails closed you can't SSH into the server. If it fails open it's just as though you aren't using fail2ban. Actually, my biggest fear with fail2ban is it failing and locking me out of a remote server. If it indeed always fails open then there's no reason for me not to use it. Though, I don't know how it could be guaranteed to always fail open.
- lol768 8y ago> I feel like fail2ban failing open is a good thing. If it fails closed you can't SSH into the server. If it fails open it's just as though you aren't using fail2ban. But it fails open whilst you think there's a layer of protection there. > Actually, my biggest fear with fail2ban is it failing and locking me out of a remote server Most of these tools will let you whitelist an address - and presumably there's usually a way back in via a serial console/IPMI-KVM interface?
- jazoom 8y ago>But it fails open whilst you think there's a layer of protection there. That's pretty easy to test for. >will let you whitelist an address I don't have a static IP address. >usually a way back in via a serial console/IPMI-KVM interface? I haven't been able to do this at my provider when password login is disabled. Maybe I just don't know the way.
- m0nty 8y ago$ grep -c sshd /etc/hosts.deny 1192 $ uptime 11:58:51 up 327 days, 21:33, 1 user, load average: 0.13, 0.11, 0.09 I'm using DenyHosts for this, there are alternatives but this works for me. http://denyhosts.sourceforge.net/ http://denyhosts.sourceforge.net/
- plange 8y agoyou should probably upgrade that kernel
- h1d 8y agoWhat can you tell from the line?
- thatfunkymunki 8y agoHasn't rebooted in a year
- jstarfish 8y agoThat used to be a selling point of Linux versus Windows. Times have changed...
- m0nty 8y agoIf only there were some way to upgrade the kernel without rebooting. What a god-send that would be.
- thenickdude 8y agoThe Spectre/Meltdown patches required a reboot.
- m0nty 8y agoNot all architectures are affected by it.
- vbezhenar 8y agoI never understood why people would use fail2ban or similar approaches. Just set proper password and move on. SSH traffic is tiny, it's not a problem. You'll never be hacked with a proper password. Let them waste their time.
- deleted 8y ago[deleted]
- vxl 8y agoThe failed attempts make it harder to monitor for other attacks because of the noise in log files, network traffic, etc. and if an attacker IP is blocked early they can't try more effective attacks.
- Krasnol 8y agoI've set up a second obstacle by allowing access to the device from a certain range of IPs on the router. Cleaned up the log pretty good.
- seldomcomment 8y agologins can be delayed [0] and TOTP can be used in conjunction with SSH (without google authenticator) [0] https://unix.stackexchange.com/questions/105553/how-to-provide-login-delay-in-ssh#105559 https://unix.stackexchange.com/questions/105553/how-to-provi... edit: quick and dirty notes of my setup [1] [1] https://pastebin.com/yYzSrM61 https://pastebin.com/yYzSrM61
- zAy0LfpBZLC8mAC 8y agoExcept actual attacks don't show up in logs anyway, so it's still pointless? The SSH daemon logs when it successfully rejects an access. A successfully rejected access is inconsequential to your security. If you are using secure passwords or pubkey authentication, it will never log a successful login by an attacker. What remains then is exploitation of the SSH server ... but the SSH server doesn't have a code path that logs "I have been exploited".
- arca_vorago 8y agoMy current take: nftables instead of iptables port-knocking non-standard port key+pass access/auth ip whitelist good logging ED25519 wherever possible!!!
- Artemis2 8y agoI love the idea of Single-Packet Authorization with fwknop instead of port knocking: http://www.cipherdyne.org/fwknop/ http://www.cipherdyne.org/fwknop/
- orev 8y agoI wish fwknop would be added to pfsense.
- AlexCoventry 8y agoHow does fwknop compare to knockknock? https://moxie.org/software/knockknock/ https://moxie.org/software/knockknock/
- michaelrash 8y agoThe design decisions that govern fwknop provide guidance on how fwknop is different from knockknock: http://www.cipherdyne.org/fwknop/docs/fwknop-tutorial.html#design http://www.cipherdyne.org/fwknop/docs/fwknop-tutorial.html#d...
- AlexCoventry 8y agoThanks.
- nerdponx 8y agoIP whitelist is tough when your home network has a dynamic IP.
- purple-again 8y ago
- kuschku 8y agoThis all is easy if you only get a handful of login attempts. But when you get more, and more, it becomes an issue. On one of my servers it got so bad that SSH was at the end constantly using an entire CPU core. I don’t have logs from that server, but here are logs – just the failed attempts – of 2 months from a server that was less severely affected: https://s3.kuschku.de/public/failed_ssh https://s3.kuschku.de/public/failed_ssh [327M] And that was despite already blocking massive areas of IP space already.
- larkeith 8y agoFail2Ban and psad will do wonders for that.
- kuschku 8y agoYeah, I now simply removed 0.0.0.0/0 dport 22 from the whitelist and only added a select few IP ranges. That has calmed the storm significantly, and if an IP from this range (my ISP) tries it anyway I can contact my ISP (a small local ISP) to take that down.
- larkeith 8y agoThat's even better, if you don't need to access from arbitrary IPs.
- jwilk 8y agoThe numbers on https://cdn-images-1.medium.com/max/1600/1*47UCBwMdQGFNtk2aoXB5gA.png https://cdn-images-1.medium.com/max/1600/1*47UCBwMdQGFNtk2ao... look very suspicious.
- ransom1538 8y ago# cat /etc/ssh/sshd_config | grep PasswordAuthentication PasswordAuthentication no ^ it should return that. The end.
- jgtrosh 8y agoIs it worth mentioning useless uses of cat in this day and age?
- sametmax 8y agoPlease educate me
- znpy 8y agogrep PasswordAuthentication /etc/SSH/sshd_config
- gsich 8y agoWhich is worse then using cat, from a user perspective. Imagine if you look for something else, you have to point the cursor in the middle, right after grep and before the file. With cat |grep you press Arrow-up and Alt+Backspace and you are right where you want to be, ready to search for something else. In scripts you can save the extra process though.
- jgtrosh 8y agoYou mean ctrl-w or <ESC>bC? ;-) I would find it okay without cat to start the next line with "grep newPattern " and then fetching the previous filename with alt-. (edit: or !$ of course, but alt-. is more practical) -- not saying it's necessarily always better, but knowing your shell better (and/or using a better one) is often a viable alternative to using another construct. Also, than* (sorry)
- gsich 8y ago
- ryanlol 8y agoI’ve never been able to understand this. Why is it that paranoid nerds need to waste so much time masturbating over failed SSH login attempts? All you need to do is turn off password auth, or perhaps use wireguard.
- znpy 8y agoI am slowly closing all non-vital ports on my router and just letting services accessible from within the VPN. No open port = no hacking attempts. Investing a week of on-and-off studying and tinkering with openvpn is really paying off.
- zAy0LfpBZLC8mAC 8y ago> No open port = no hacking attempts. That's just obviously nonsense? Closing the port does not change anything about the attempts. Nor about the success rate of the attempts, if you aren't being an idiot with insecure passwords.
- kdelok 8y agoI think the person above was maybe being too imprecise with their language. Obviously it doesn't affect the frequency of portscans and the like, but whitelisting ports is a reasonable approach to further mitigating your risks.
- zAy0LfpBZLC8mAC 8y agoWell, potentially. But then, there is no fundamental difference between a service rejecting unauthorized connections and a firewall rejecting unauthorized connections. If your service is already rejecting unauthorized connections, you don't gain anything by also rejecting the same connections at the firewall, and that is why "No open port = no hacking attempts" is ultimately nonsense: It doesn't change anything about the attempts, and chances are it doesn't fundamentally change anything about the rejections either. Also, adding a VPN exposes the VPN service to the internet, which thus adds attack surface in a different place. Which might be worth it if you need remote access to otherwise vulnerable services. But the simplistic view of "rejecting connections at the firewall" == "no more hacking attempts!11" is just that: simplistic.
- fgonzag 8y agoThat's not how it works. If a vulnerability is found in the service, and you don't need external access, blocking it from the firewall/router essentially blocks the vulnerability. In your scenario, with a public facing service, you will get exploited by a 0 day.
- gsich 8y agoChanging the SSH port proved to be most successful. Yes, of course you can find the port with a port scan. But it keeps the logs clean. And I don't want to waste CPU cycles on some brute-forcing idiots.
- VilniusHacker 8y agoI do that. Security professionals will whine "security by obscurity is not security" but the fact is it cuts down these attempts by a huge factor.
- gregmac 8y ago(I'm primarly a developer, not sysadmin) Is using non standard ports a reasonable approach? On one hand, it's kind of a security-by-obscurity measure, and it's also a (very minor) inconvenience to real users. However if it's not being used in place of other reasonable security measures, I'm not really sure what's bad about it, but it does feel a bit janky to me. Same goes for other non-public services such as VPN. What ports should be used? I think clearly you don't want to use other well known services (eg, running ssh on port 25 is just going to fill your logs with failed connection errors). Should you use ports >1024? >10000?
- orev 8y agoI’m so tired of hearing people quote “security by obscurity” in relation to moving ssh ports. It is NOT security by obscurity (SBO). SBO is taking an insecure thing and trying to hide it, and relying on the hiddenness as the ONLY means of security. SSH is already highly secure, so hiding it has nothing to do with the security. You can debate whether it’s worth the inconvenience vs reducing log spam, but SBO does not come into it at all.
- xur17 8y agoI typically just pick a high numbered port. > 10k would work. I see it as a way to reduce the number of attacks, which is helpful, and may end up being more secure, but I wouldn't count on that alone. Against botnet attacks it will likely help, but against someone specifically targeting you, it likely will only slow them down. Basically it boils down to 'why not?' for me.
- jlgaddis 8y agoSomeday I'm going to put all the different pieces together and have things like SMTP & SSH auth failures, hosts hitting my spamtraps, bots scanning for open 23/TCP, and so on, to where I end up with one central host collecting all those events and injecting /32 routes for those IPs towards null0 in my core.
- jacquesm 8y agoA friend of mine built this complete with a whole pile of honeypots and lots of companies use it. https://www.blockedservers.com/ https://www.blockedservers.com/ Super useful.
- codetrotter 8y ago> It absolutely shocked me that it is worth the time of these drive-by attacks to try combinations like root/password, root/root, or root/admin. Apparently they have enough successes using those incredibly insecure combinations that it is worth their time. It is fairly evident to anyone that has looked at the traffic that a honeypot gets that most of the activity is automated. Most of it is purely botnets, some of it is automated password guessing followed by manual login. I think the amount of people manually trying passwords is insignificantly small.
- highesttide 8y agoI think they mean "time" for the botnet in this case
- ejrv 8y agoOne thing I've found amusing is you can cut down SSH hacking attempts to almost zero (at the cost of compatibility with legacy systems) by using very progressive encryption. If you force use of ChaCha20/Poly1305, ED25519 and so forth in sshd_config, you'll see these attempts almost disappear. There's probably an observation to be made about just how overwhelmingly many of these attempts come from fairly old software setups.
- maxk42 8y agoGood info! Got any good resources on configuring sshd algos?
- nirv 8y agoI highly recommend Mozilla's OpenSSH configuration guideline[1]. [1] https://infosec.mozilla.org/guidelines/openssh#Modern https://infosec.mozilla.org/guidelines/openssh#Modern
- ejrv 8y agoNirv's recommendation below is spot on, but here's a ready-made version as well: https://github.com/ejrv/linuxconfigs/blob/master/etc/ssh/sshd_config https://github.com/ejrv/linuxconfigs/blob/master/etc/ssh/ssh... It's basically a clone of Mozilla's, except enforcing only the very strictest encryption. You can just lift the first 16 lines out of that and merge them into your sshd_config if you don't want to change your existing sshd_config too much.
- ethbro 8y agoOr the ratio of script kiddies to developers.
- anonu 8y agoYou mean the ssh handshake will fail... It won't stop an attempt. They just won't get to a password prompt. Right?
- asclepi 8y agoI did that and it bit back when it prevented Transmit from connecting to my servers, with only a generic "Unable to connect" message popping up. This was fixed fairly recently[0], but it's a reminder that using a restricted set of ciphers may not only prevent illegitimate connections from going through. I can confirm from the same experience, however, that it prevents most hacking attempts from ever going to the authentication stage. [0] https://library.panic.com/releasenotes/transmit5/#5-1 https://library.panic.com/releasenotes/transmit5/#5-1
- tempfs 8y agoSome notes from an InfoSec person. - Have a stand alone SSH server that is something like a R-Pi[type B] running an OS that gets patches regularly via unattended-upgrades and reboots itself at least once a week.[could also be a minimal VM like AlpineOS if you need Gbps+ line speed] - Have this R-Pi and your network gear plugged into a UPS that can withstand at least a couple of hours of power outage. - Use non-standard ports on your perimeter FW for forwarding to the R-Pi. - Run fail2ban or something similar. Set a bantime of at least an hour[3600 sec] after no more than 5 attempts in 600 sec. This is mostly to discourage anyone who stumbles across your perimeter listening port.[which enough will] Password login will be disabled anyway, but a bot might not check for allowed auth types.[nmap -Pn -p 22 --script ssh-auth-methods <target>] - No passwords allowed in /etc/ssh/sshd_config[PasswordAuthentication no]. Use public key based authentication.[PubkeyAuthentication yes] Put the public SSH key[id_rsa.pub] of any machine that you wish to authorize into ~/.ssh/authorized_keys on your R-Pi. This way you can authorize and de-authorize remote machines[with particular users] at will. - add 'AllowUsers xxxx yyyy' to your /etc/ssh/sshd_config - No root logins allowed![PermitRootLogin no] - Maybe limit via perimeter or on the R-Pi which source IP ranges are allowed to access to SSH.[whitelisting] - Periodically review your logs of perimeter FW, R-Pi FW and /var/log/auth.log to see what's going on. It should be pretty quiet, but look at them anyway once a week. - Sleep well knowing that you have a simple, layered and robust defense strategy that also has power-event survivability. Others have mentioned port-knocking which is a cool trick but not something that I typically use in an actual daily defense strategy because I'm not sure how much value it really adds. Don't use your perimeter device to host SSH services. It will not get patches fast enough when vulns come up.
- rsync 8y ago"Others have mentioned port-knocking which is a cool trick but not something that I typically use in an actual daily defense strategy because I'm not sure how much value it really adds." I don't know how much it adds, but it is non-zero. The knockd daemon is rock solid[1] and your ssh port traffic goes down to zero (other than your own use). Port knocking has no place in security by itself but I think it's a wonderful addition to a layered defense - my favorite one, in fact.
- ktpsns 8y agoI wonder why the author set up a dedicated SSH python server which mimics some unix userspace as a honeypot. Why not a VM with a real standard widespread linux distribution to see what attackers actually do? Obviously their codes are not very sophisticated that they stop when "uname" fails with certain arguments.
- iforgotpassword 8y agoBecause it's much easier to set up? It's not breaking after any session no matter what the attacker does. Every session is isolated if several run concurrently. Getting this with a VM involves quite some effort. Then you need to make sure the attacker cannot use any commands to cause external damage but still make them seem to work. But the latter is also a problem with the simulated environment. It's quite easy to detect if you're not just blindly sending automated keystrokes without checking the feedback.
- ktpsns 8y agoObjectively there should be no difference between a honeypot VM and an actual "productive" VM. I acknowledge that it is much harder to setup, maintain and supervise a network of honeypots VMs instead of a dedicated isolated honeypot server process which just simulates a system. However, as mentioned in other comments here, there is no gain of knowledge if you don't simulate a whole system but only parts of it.
- f311a 8y agoThey stop because they know that it's a honeypot. I guess it's a simple check for popular honeypots which emulate linux commands.
- deftnerd 8y agoMy servers generally have an IPv4 address and a whole IPv6 subnet. I like to disable SSH except on a specific IPv6 address. Many botnets just focus on IPv4 and there is a lot more territory to scan on IPv6. I usually couple that with moving to a non-standard port, enforcing modern encryption (ed25519 or ChaCha20, etcc), fail2ban, require SSH keys, etc
- gist 8y agoI think that by using obviously easy passwords anyone who really is looking for something is going to go to the next target. After all impossible to believe anything of value is behind 1234567 or changeme as a password. Even 'baseball' makes slightly more sense. Besides someone running an ssh server is not exactly mr computer newbie. Maybe a desktop you pick out of the trash might contain something interesting if that was the password. In a classic CIA honeypot I would imagine the women are carefully matched to the target. If you are "James Bond" you get a better looking woman to try and trap you. If you are "Wallace Shawn" (in looks) you would probably figure out really quickly it's a trap if the woman looked like Angelina Jolie.
- fareesh 8y agoHow configurable is fail2ban? Is it service based? Am I in danger of locking out a legitimate user like someone who makes 25 ajax requests within the span of 3 seconds? Can I make it work for authenticated HTTP endpoints as well?
- WillieStevenson 8y agoLong ago I made the same thing. https://livesshattack.net https://livesshattack.net
- daveguy 8y agoI wrote a little utility to keep ssh ports off unless a totp packet is sent to a UDP port. I haven't worked on it much since the initial proof of concept. There are several things that could be done to improve it (open only for the address that sent the packet, symmetric key for safer secret distribution, etc). Shared key would involve encrypting time and only open if the decrypted time matches current. The totp packet is a full hash of time + shared code so brute force is completely infeasible. It's currently in python, it would be much better in something like go with daemon monitoring. It is very simple, and it practically eliminates attempts. If it became widely known there would probably be attempts on the udp port. An additional improvement would be randomize the listening port based on the time+hash: https://dhj.io/posts/2017/05/16/totp-knock/ https://dhj.io/posts/2017/05/16/totp-knock/
- finnthehuman 8y agoIn case you were unaware and didn't just want to write your own, fwknop is an off the shelf tool to do the same. http://www.cipherdyne.org/fwknop/ http://www.cipherdyne.org/fwknop/
- daveguy 8y agoI appreciate the reference. I looked at that before starting and reference it in the blog post (but should probably include a direct link to the software rather than the original concept post and article that include links). The similarities are they both use a single packet and they are both an improvement on port knocking. This is a simpler solution (pairing UDP with TOTP). No TCP connection is required. Making it ip address specific would prevent replay attacks (although even without address specific the replay would only be valid for ~30s). Another benefit is that a port scan will reveal no ports open (because it is a one-way UDP based protocol). fwknop is definitely a polished solution where this is just a proof of concept for the TOTP/UDP as a shared key knock. Thank you for the feedback! EDIT: I updated the post to include a direct link to fwknop rather than just links to the posts/articles about it.
- kamaraju 8y agoI found fail2ban is necessary but not sufficient. It requires some initial number of failed attempts before the attacker is blocked. To prevent even those initial attempts, I use multiple layers of defenses such as country based blocking or blacklist based blocking. If an attacker slips through despite all this, I blacklist them manually in /etc/hosts.deny. If there is any interest, my scripts are in https://github.com/KamarajuKusumanchi/hosts.deny/ https://github.com/KamarajuKusumanchi/hosts.deny/