12 ms·
An FBI agent from the Cyber Crimes division gave a talk while I was in college (>10 years ago). He was interested in brute force attacks against SSH daemons and
by nmaludy 6y ago
An FBI agent from the Cyber Crimes division gave a talk while I was in college (>10 years ago). He was interested in brute force attacks against SSH daemons and created a couple hypotheses around number of logins and common passwords. To test this he setup two Honey Pot to record all of the username/passwords. The first one listened on standard SSH port 22, the other listened on a random high-numbered port. He left both of these running for ~6 months.
Results:
The honey pot listening on standard port 22 received 1,000s of login attempts (sorry, don't remember the exact number). The honey pot listening on the random high-numbered port received exactly 0.
I know this is just an anecdote and it might not necessarily be true today, but this experiment always sticks in my head. At least the guy used the scientific method: created a hypothesis, conducted the experiment, analyzed his results.
- ryanlol 6y agoIt’s an useless anecdote because SSH bruteforce attempts are not a threat and cost you nothing.
- elmo2you 6y ago> It’s an useless anecdote because SSH bruteforce attempts are not a threat and cost you nothing. I can say from personal experience that this anecdote is both accurate (20 years ago and up till today) and meaningful. No idea where you get this notion that brute force attack are no threat or without cost. They certainly do pose a security risk (takes only one insufficiently trained employee/intern for a potential breach) and they certainly come at a cost (way beyond dirty just logs).
- fmajid 6y agoBotnets and fast networking stacks like DPDK have made port scanning the entire Internet a much more viable proposition than 20 years ago. Depending on your sshd settings you can be effectively locked out of your machine by a brute-force attack. Running on IPv6 and/or having a secondary sshd instance that only accepts connections from whitelisted IPs is cheap insurance.
- mercer 6y agoThat doesn't invalidate the observation (which I share) that these attempts are almost 0 when using a different port. It reduces logspam and if I start getting lots of brute force attempts on my non-standard port, this is useful and meaningful information (someone cares enough to do this).
- elmo2you 6y ago> Botnets and fast networking stacks like DPDK have made port scanning the entire Internet a much more viable proposition than 20 years ago True indeed, yet even today I have seen little evidence of scanning beyond standard ports (pretty much the same as in the past). Criminals are opportunistic by default and tend to go for low hanging fruit (standard ports, with standard server config). I certainly did see in increase on standard ports. Even while full range scanning has become more feasible, I have not seen much evidence of its use.
- iso1631 6y ago> takes only one insufficiently trained employee/intern for a potential breach How? If they've leaked their key, why do you assume the port hasn't leaked too? On the other hand if they haven't leaked their key how would they get in? Or are you allowing password authentication like it's 1999?
- elmo2you 6y ago> Or are you allowing password authentication like it's 1999? That is assuming you have such authority or technical means. If you're maintaining systems for a company, there's a good change that the product vendor simply won't allow fucking around with their system like that (ergo: yes, in practice you are indeed stuck with your 1999 authentication). I'm not saying that it is good security (that's why layers security is often paramount), but it is situation I've encountered more than a few times. Great for you, if you are GOD on all the systems you work with. Even then, your client/employer might simply tell you to stuff your objections and accept the bad authentication policy, because to them the risks are simply not worth the business disruption. I totally agree that is a flawed argument. But decisions usually aren't always (if ever) called on valid arguments. Good for you, if you are in a position where you never had to deal with such real life situations.
- deckard1 6y agoNot sure what you're arguing here. You either have control over sshd or you don't. Or are you really suggesting you can change the port of sshd but aren't allowed to disable password auth? I'm a software engineer, so if my company gets hacked via ssh that's really not my problem. Worrying about such things would make me a busybody. But if you're a system admin and can't properly do your job, then I would seriously start looking for a new place to work. They will get hacked and you will be the guy that gets blamed.
- elmo2you 6y ago> Not sure what you're arguing here. You either have control over sshd or you don't. Or are you really suggesting you can change the port of sshd but aren't allowed to disable password auth? First, you'll have to separate two things here. One is the technical ability to control sshd, the second whether a company will allow you to tinker with the auth policy (whether that is password login, password login with only strong passwords, or rsa/ecdsa key access only). The latter has nothing to do with control and only with what decision makers allow you to do (that sometimes is a large product vendor, not allowing anything beyond what they ship). If you work in a place where you have full control over the systems you work on, great for you. I can ensure you that it is not the norm (unless we're talking about hobby projects or projects with exclusive personal ownership). As for the technical aspect, keep in mind that changing the public facing ssh port might not even be done on the host itself, but e.g. in port forwarding table in a router/firewall. This might not even always happen because it's technically impossible to do it on the box itself. I'm pretty certain that tinkering with a box is regularly discouraged (especially if it is managed by some orchestration or vendor specific control/update tool), while effectively the same can be done by changing a router/firewall. There's a lot more things to be said about that, but please take it from me that hacking around in a systems you have not build yourself isn't always a bright idea (and it happens to be a very common situation). > But if you're a system admin and can't properly do your job, then I would seriously start looking for a new place to work. That's an interesting theory, but frankly not how I think the real world (usually) works. As a system admin you are there to solve problems for a client or employer. You can (and should) of course always warn for potential dangers, but refusing work or quitting a job/assingment because you're not getting full control over a system .. good luck with that. It is simply not an acceptable position in many situation. You must be in really high demand if you want to pull stunts like those and still have any work after a while. Maybe it works different in software engineering land, but I highly doubt it. When was the last time you quit a job, because you preferred a different library or framework over the one your superiors/client dictated? Please don't get me wrong. On a personal level I'm very principled about what I choose to work on or with (and what I refuse to take part of). But at the end of the day we are professionals, here to solve problems. If we can and a client/employers is willing to accept the risks of an imperfect solution that fits in their requirements, it ultimately is their call and responsibility. All within reason, of course.
- selfhoster11 6y agoThey cause some real work because of the log noise they create. It's easier to see targeted SSH attacks if all the undirected attacks are filtered away.
- gruez 6y agoWhat would be indications of "targeted" ssh attacks, and what can you do in response?
- sascha_sl 6y agoThe fact that someone bothered to scan the entire range (or find your port at random) might indicate that they're specifically targeting you, and just being aware of that is an upside.
- gruez 6y ago>and just being aware of that is an upside. but what can you realistically do with that knowledge?
- BLKNSLVR 6y agoInfinitely more than you can do without that knowledge
- ryanlol 6y agoGive some examples then. Knowing that someone is targetting you shouldn’t change anything, you should be ready regardless.
- sascha_sl 6y agoIt shouldn't, but it does. Many smaller companies driven by business people, where maybe tech is just seen as a necessity on the side need a narrative like "people are trying to get in and if they do it's going to be a disaster" to take security seriously. Then or at the point where the disaster strikes. I'm not really sure why this point was voted down below either; just because you work for someone who takes security seriously (at least to the point where it's insurance-satisfyingly safe) does not mean everyone does. Years ago I worked at a small agency and every bit of time I spent had to be justified and produce tangible/visible results. "But is anyone really going to try to hack this local business" was a question I actually had to answer, since most other employees were creatives.
- icedchai 6y agoThey fill the logs with noise. That's "something." I finally set up fail2ban a while back... it works wonders.
- gruez 6y agoDo people even read the logs? What value does it add compared to just monitoring successful logins?
- cedilla 6y agoThere's a lot of value. For example, if you see failed logins against random user names like "dbadmin" or "root" it's likely just random scanning, but what if suddenly lots and lots of valid user names appear?
- gruez 6y ago>but what if suddenly lots and lots of valid user names appear? then what are you going to do?
- sokoloff 6y agoAt a minimum, spend some of my limited time and attention on this issue rather than the 100s of other things that might be clamoring for my time.
- wbl 6y agoWhat are you going to do to solve that issue?
- cedilla 6y agoWell that would highly depend on what I'm seeing. If it's a single user there might be an attack on the way against that user. If it's multiple users, there might have been a compromise of some credentials. It's definitely something you need to investigate.
- Smar 6y agoI get thousands of tries in a day to a newly installed host using IPV4 nowadays.
- teilo 6y agoI gave up obscuring years ago, and just use fail2ban.
- bronson 6y agoNow that botnets are brute forcing from thousands of unique ips, fail2ban doesn’t offer much on my servers anymore. I‘ve stopped installing it.
- hn_acc_2 6y agoJust curious, what problems does fail2ban suffer with thousands of unique ips? (A crowded iptables I guess...) I still use it with a super oppressive jail time and few retries, with a few whitelisted IPs and it seems to work ok.
- jldugger 6y agoI think the concern is a botnet with n IPs is that fail2ban tracks individual IPs, so if you have any kind of grace period before bannination, they get a linear speedup of n, and if there's an expiration period, get to try n times harder than a single bored script kiddie. Worse, from an economic perspective, theres enough hosts listening on port 22 that a bot can try instead while they wait for timeouts, so you're not really imposing a cost on them. If you view running a botnet as a form of multi-armed bandit problem, the best you can really do is limit the economic value by slowing them down a tad versus their many, many other options.
- mobilio 6y agoF2B support subnets: https://github.com/fail2ban/fail2ban/issues/2261 https://github.com/fail2ban/fail2ban/issues/2261
- rtkwe 6y agoI think they're saying it doesn't stop brute force attacks because the botnet will just try with another IP.
- LinuxBender 6y agoThis is my observations as well. For 20+ years I have run ssh on a high port, with exception to my sftp server. The sftp server is hit every day, all day. I have received 0 hits to my ssh port on all my other servers. Even if they hit that port, they would not see anything, as I use a poor-maps port knocking using iptables string matching, but I would still see the attempts in the iptables counters and they are always 0. FWIW, when I chose my port, I looked at port scanning statistics back in the day, looking for the least scanned ports. It appears those stats have held true for a couple decades at least.
- Twirrim 6y agoAnecdotally, I have a machine with an exposed SSH, on a high number port. I get brute force attempts on a regular basis against it, just way less than when I run it on the standard port number. Security by obscurity is just one part of the steps I take with that machine. Using a high port number is dead simple and easily handled client side too, so I just do it.
- cheschire 6y ago> just one part of the steps I take with that machine. You may be interested to know this is called "Defense in depth". https://en.wikipedia.org/wiki/Defense_in_depth_%28computing%29 https://en.wikipedia.org/wiki/Defense_in_depth_%28computing%...
- babadaba 6y agoOr you could say, security by obscurity is one of the layers of their defence in depth strategy. Edit: I believe you are implying that they used “security by obscurity” incorrecty, which I don’t believe they did. If I read that wrong, my bad!
- sllabres 6y agoThe articles summarizes > It’s where you keep the mechanism secret, not the key. I think this can be, as you write, defense in depth if the secret of the mechanism is not the only defense. As example the block cipher for the Common Scrambling Algorithm https://en.wikipedia.org/wiki/Common_Scrambling_Algorithm https://en.wikipedia.org/wiki/Common_Scrambling_Algorithm has been secret. As it seems that has delayed the analysis of the system for about 8 years, but not damaged the procedure.
- a1369209993 6y agoTechnically defense in depth refers to multiple effective security measures (like cryptographic login), so security by obscurity isn't actually part of it. (Moving SSH port plus something like fail2ban could be considered defense-in-depth against the incidental DDOS-like issues, though.)
- 6y ago
- vbezhenar 6y agoI recently tried to change ssh port to remove log noise. Well, it certainly helped a little bit, but bots quickly found out new port and started to brute force it, so in the end it did not help, just reduced noise. And as I don't see much difference between 100000 attempts and 1000 attempts, I decided to return it back. I don't care about brute force anyway, my passwords are not "root:root".
- syshum 6y agoroot:root would be crazy, I would only use root:toor5 then I am secure for sure... /s //Side note, you should never allow login as root, and you should not really allow remote password login at either, only keys
- yjftsjthsd-h 6y ago> root:root would be crazy, I would only use root:toor5 then I am secure for sure... That's a minimal (read: insufficient) improvement, but if it were behind a real layer of security then it would strictly be an improvement. > Side note, you should never allow login as root, and you should not really allow remote password login at either, only keys I'm on board with keys, but why not allow root? If it's secured by a key, what's the harm?
- zerkten 6y agoWhen you have a large base of installations in a big organization, this can make a difference in practice because your incident responders have to sift through less data. This makes much less of a difference when you have great log management and SIEM systems in place. Many places don't, and some hygiene can make a difference at times. When I see this in practice, the first thing I check is how auth is being done and the overall security of the host. Then, I look for how they are doing SIEM because cleaner logs is a common reason and they'd be better off with a more proactive management approach.
- gen220 6y agoThis made me think of a fun toy-idea. Let the server and client share a secret. Use that secret to encrypt the UTC date (2020-09-21), and sample some decimals from the first few bits (adding 100 or so, to avoid low-ports). You could use that mechanism to rotate ports every 24 hours. This way, the bots wouldn't be able to learn the ssh port for more than 24 hours, without the shared secret. Sounds like fun, or an easy way to lock yourself out of a box by mistake, depending on your perspective. :)
- heavenlyblue 6y ago> I know this is just an anecdote and it might not necessarily be true today, but this experiment always sticks in my head. At least the guy used the scientific method: created a hypothesis, conducted the experiment, analyzed his results. I don't need a research to see the difference in how much logging journalctl generates immediately after I disable port 22.
- doggydogs94 6y agoDon’t disable port 22; just neuter it. If port 22 is disabled, the attacker will look elsewhere in your system.
- vlod 6y agoDo you have a suggestion on how to do this?
- ogre_codes 6y agoAn SSH Tarpit is a good way: https://nullprogram.com/blog/2019/03/22/ https://nullprogram.com/blog/2019/03/22/
- vlod 6y agothanks!
- giovannibajo1 6y agohttps://github.com/cowrie/cowrie https://github.com/cowrie/cowrie
- tensor 6y agoThis is an interesting point. Imagine if you put a fake SSH agent on 22, it responds just like SSH but never allows a login. Would it make it even less likely that someone would bother trying another port?
- aikinai 6y agoBy the way, does anyone here know how to actually check sshd logs on MacOS?
- solarengineer 6y agoThere are a number of responses here: https://stackoverflow.com/questions/43382825/where-to-find-sshd-logs-on-macos-sierra/54148031#54148031 https://stackoverflow.com/questions/43382825/where-to-find-s...
- wvenable 6y agoWhat I've found these days monitoring my own network is that there is now 2 waves -- a port scan and then the attack. If I change a port for anything to another random port I won't get any login attempts for a few days but eventually I start getting hit again. I can repeat this over and over. I imagine what is happening is that the bad actors are scanning for open ports and they feed that periodically to another process that attempts logins.
- Retr0spectrum 6y agoThe second wave is likely when public port scanning services such as shodan re-scan your host. (I wonder how hard it would be to fingerprint and subsequently blackhole shodan et al's scanning traffic)
- fgonzag 6y agoLike an IDS or IPS? Snort is quite decent at that.
- beams_of_light 6y agoI have the same experience with non-standard port usage, and I think it's a very reasonable thing to do, while also caring for the security of the service behind that socket. SecOps will thank you for not having to wade through log spam in the endeavor of preventing attacks.
- throwawaynothx 6y agoDo you know if he ran the honey pots on the same server? if so that's why it received 0. the bots hit 22 and stopped.
- nmaludy 6y agoDifferent servers.
- dwild 6y agoIf you expect to be hit by that kind of attack (simple combination of username/password), then you should protect yourself from that kind of attack. It's never been easier nowadays to do this. You may answer that you could still miss a few of theses simple password, that your solution would be more effective, sure, but then you use security by obscurity to protect yourself. By the way, security by obscurity does works, it's not bad per say, just as that FBI agent just proved, it does have an effect. If it didn't, there wouldn't be so many case where it was used. The issue with security by obscurity is when you rely on that to protect from vulnerabilities and then ignore them. It only lower the likeliness of getting attack, it doesn't make attack less effective, it doesn't protect from any vulnerabilities. Sadly, too many time, we just ignore it, hide everything and hope to avoid targeted attack which would foil that obscurity pretty quickly. This is when it get bad.
- thefreeman 6y agoThe entire point of everyone who rants against non standard ssh ports is that it adds more noise to the signal of "You should disable password login entirely and only use public / private key authentication." And reading many of the responses in this thread it makes sense to me. So many people in this thread talking about how many less failed password login attempts they get when they change the port, indicating they allow password based login in the first place.
- GekkePrutser 6y agoEven with key auth only, which I have, you're still getting many attempts, wasting CPU cycles. Most scans are smart enough to give up as soon as they see publickey as the only method, but some keep trying to connect hundreds of times.