10 ms·
Learn from your attackers – SSH HoneyPot
- otakucode 9y agoIf one were to run a honeypot like this and take every IP which connects and attempts a login and immediately ban it from your network, then if more than 1% of the IP range they are in has been banned, ban the entire range... what would the expected outcome be for a typical residential user?
- ryanlol 9y ago>what would the expected outcome be for a typical residential user? Wasted time for precisely zero benefit. There are lots of hobbyists that'll tell you otherwise, but in reality you should be using key auth and worrying about better things.
- merb 9y agoin my first years I've setup a ssh server behind a nat'd network and installed fail2ban. the outcome was that nobody could access the ssh server... probably the same thing would happen, on ssh I've seen so many login attemts even on the smallest VPS I've ever gotten, it's just ridicoulus and important to use non password authentication with ssh (i.e. rsa keys probably 4096b).
- logicallee 9y agoEDIT: I'm withdrawing the question per the accepted answer below. - original comment: >it's just ridiculous and important to use non password authentication with ssh Does this follow from the assumption (that you did not state) "since I won't bother to create a high-entropy password"? Granted its a fairly safe assumption if you're not trying too hard but SSH supports passwords of arbitrary length so I'm curious if there's any other reason, besides not taking the time to create a high-entropy password. ----- Accepted answer below (esp "Because your users cannot be trusted. I run servers which coworkers, all developers, get access to.")
- tokenizerrr 9y agoBecause your users cannot be trusted. I run servers which coworkers, all developers, get access to. It's either ssh keys or 2 factor authentication because most don't give a fuck about security and will insist using garbage passwords if not prevented from doing so (and then they will complain), even though the use of a password manager is mandatory.
- logicallee 9y agoPerfect answer actually. Accepted and I've updated my original comment.
- jlgaddis 9y ago> ... will insist using garbage passwords if not prevented from doing so ... /etc/security/pwquality.conf is how I prevent co-workers from doing so (on RHEL and derivatives).
- tokenizerrr 9y agoGood to know. We do our user provisioning through an internal portal which enforces everything. The ssh servers als gets configured to disable password login so the only thing passwords are used for is sudo. All web stuff is behind a SSO gateway with password+2fa.
- Cozumel 9y agoUsing passwords for ssh is just a bad idea, high entropy or not. Keys are the only way to go. The amount of login attempts you get otherwise is just off the scale, on one server there were over 20,000+ failed attempts from my last login in the space of just 10 minutes, all it takes is one lucky guess.
- spiritguy 9y agoOne thing i was surprised to learn was that whatever you type as your password is actually transmitted to the server for validation! So if you typed the password for the wrong server in.. then you've just leaked that other password to the server you were trying to connect to! uh oh! if you use keys, this doesn't happen.
- callesgg 9y agoJust alter the ssh port, i have never received a ssh connection that i can not personally account for.
- secfirstmd 9y agoYep, big advocate of that.
- pilsetnieks 9y agoSecurity through obscurity is not the solution, though. Sooner or later someone does a port scan and finds that port. I've found that it's more efficient just to whitelist the IPs or subnets you might need and denying the rest. Edit: I didn't mean changing the port is a bad thing. You can and should do it, and it will help a bit - just that it shouldn't be the only thing to rely on for your SSH security.
- avtar 9y ago> Security through obscurity is not the solution, though. It's not the only solution but something that can be used to at least drastically reduce the amount of noise in logs. > I've found that it's more efficient just to whitelist the IPs or subnets you might need and denying the rest. That does sound more efficient but what if someone connecting doesn't have a static IP?
- pilsetnieks 9y ago> what if someone connecting doesn't have a static IP? Whitelist your ISPs subnet then, at the very least - there's much lower probability of an attacker coming from just your ISPs, and, of course, use other measures too - I didn't mean that to be the only solution for the problem.
- scaryclam 9y agoWe use a VPN for this reason. It works pretty well and means we can lock things down to a single IP while allowing staff to still get on from home.
- Karunamon 9y agoOnly exception I can think of is SSHing into things where you’re not allowed to plug stuff in or put files on, like a public terminal of some kind. Password plus a second factor would be more usable there.
- noir_lord 9y agoLiterally the first thing I do on any new machine is disable password logins, disable root logins and then punt the SSH port to the top of the range (usually 47999). Every remote machine gets it's own keypair as well and links between machines also get their own key-pairs (key handling is PITA, I've yet to find a method I really like), Keys for work stuff are back up on multiple LUKS memory sticks (two in separate safes, two offsite). I'm fortunate in that work machines are in the pet category not cattle category so we don't need to change keys that frequently (though I will rotate them out yearly anyway).
- jaytaylor 9y agoFriendly advice.. It might be better not to disclose this level of detail about your professional security practices, especially when there is a trail of breadcrumbs in the HN profile on your account..
- noir_lord 9y agoShrug, given how hostile the internet is I fail to see how this increases the risk at all.
- rimliu 9y agoYeah, I am not so sure how knowing that keys are rotated helps the wannabe attacker. The only "useful" bit would be to look for SSH port up in the range, not much help though.
- noir_lord 9y agoMy attitude is short of you having the SSH keys if I can't tell my entire setup and still be secure then I'm relying on obscurity not security.
- z29LiTp5qUC30n 9y agoIf you do it right it doesn't matter. The default ip(6)tables rule: drop all packets For example to access my least secured public facing server you need to do the following: 17 port port-knocking to enable crypto port knocking Which enables ssh Which requires a 16K RSA key and TOTP The only accounts that can be logged in have 18 character randomly generated usernames and 255 character passwords. The login accounts have no access to administrative functions. So you'll need to su to an administrative account and guess the 255 character password or have a zero day exploit that can bypass SELinux and cryptographically signed white-listing of binaries.
- z3t4 9y agoDon't forget about ipv6 if you have that enabled.
- bcoates 9y agoYou'd be unreachable from cloud providers, major public hotspots, residential ISPs, and countries that use carrier grade NAT to stick all their traffic on one IP or one range.
- otakucode 9y agoHmm, I hadn't thought about cloud providers. Public hotspots, residential ISPs, and countries behind firewalls aren't really something I expect most home users really need their home network to be accessed by. But blocking connections from cloud providers would be unworkable. I suppose lots of automated SSH attacks would be coming from cloud providers... maybe there are few enough they could be whitelisted?
- artursapek 9y agoWith cloud providers it's likely a "bad" IP will eventually get re-assigned to a "good" customer whom you don't want to block. That's why an exponentially increasing ban is generally a good idea, I think. The more abuse you get from a given IP address the longer you ban it each time. If it's use for a one-time attack and thrown away, you forgive it relatively quickly.
- user5994461 9y agoFrom experience, there are two kind of cloud providers. The normal ones like AWS, OVH and Digital Ocean. And the shady ones. The first one don't attack much of anything, there is the occasional spider that tries to index your website (we had data worth scraping). The later can be banned by entire AS without issues.
- justinsaccount 9y agoI do this sort of thing, but I'm listening on a few /16s. 1% is a pretty low threshold. There are some v4 networks out there that are complete garbage.. If I was going to do something like that I'd start at closer to 90%.. 230 hosts on a /24. $ select subnet, count(distinct(cidr)) as unique_sources from (select set_masklen(cidr,16) as subnet, cidr from stuff where why like 'SSH%' and added > '2017-08-01') as foo group by subnet order by unique_sources desc limit 20; subnet | unique_sources ----------------+---------------- 181.211.0.0/16 | 11688 190.214.0.0/16 | 8486 31.162.0.0/16 | 8454 181.196.0.0/16 | 7994 181.113.0.0/16 | 7892 188.16.0.0/16 | 7294 94.51.0.0/16 | 6384 188.19.0.0/16 | 6077 31.163.0.0/16 | 5905 178.47.0.0/16 | 5788 201.178.0.0/16 | 5620 190.48.0.0/16 | 5179 188.17.0.0/16 | 4893 201.179.0.0/16 | 4812 188.18.0.0/16 | 4266 186.178.0.0/16 | 4208 5.141.0.0/16 | 4203 186.129.0.0/16 | 3858 181.112.0.0/16 | 3836 190.174.0.0/16 | 3836 I think some of those networks are using CGN and have a much smaller number of actually compromised hosts.. ISPs generally just don't give a shit about security.
- bcoates 9y agoYou may as well just block LACNIC IPs wholesale and save the trouble at this point
- otakucode 9y agoI don't see why a residential user should be worried about residential users from other ISPs being able to reach their machine usually. I suppose it would be problematic for torrenting and maybe gaming (depending on architecture of the game). But I imagine my grandmother couldn't care less if some other grandmother couldn't connect to her network directly. 90% seems really high... you'd really wait until 230 of 255 possible hosts have attempted a breakin before deciding they were on a network too dangerous to preserve your accessibility from? Are there a lot of networks where 90% of the boxes are launching attacks, but 10% have legitimate need to connect to your personal home machine?
- 9y ago
- thaumaturgy 9y agoI've been doing something similar to this for years on a mail server that hosts all of my mail as well as a number of customers. Contrary to the other replies, it's worked out really well, and you can rely on customers to complain pretty quickly if anything's not working right. The bulk of the abuse comes from Russian and Chinese networks, with Brazil working hard to catch up. Those can be pretty well perma-banned without consequence. Under "folks that should be fined into oblivion", there's cogentco.com, quadranet.com, colocrossing.com, level3.com, vpls.com, softlayer.com, hostingsolutionsinternational.com, datashack.net, singlehop.com, actonsoftware.com, and a whole raft of others. The secret sauce is a carefully tuned ban decay function that scales up with the number of abuses. Occasional hits get a network banned for an hour or less, subsequent hits get them banned a little longer, and then there's a fine line where the function goes exponential, all the way up to 6-month-plus ban periods for really big nuisances. Otherwise it's just fail2ban on some logs and some home-brew code that does whois lookups on nuisance IPs and some data mining on the whois response.
- noir_lord 9y agoI can't ban China as our suppliers at work are partly -Chinese, it's a royal PITA, I can't even whitelist them as they have to bound around VPN's just to get around the Great Firewall of China on their end.
- yorwba 9y agoIf your suppliers use VPNs anyway, you can absolutely ban Chinese IPs without much consequence for them. However, if it's for a user-facing site, please don't just ban all of China. After relocating, I was quite annoyed by the number of websites that are unavailable not because they are blocked by the Great Firewall, but because they banned all of China and VPNs to stop the abuse they saw from some IPs in those ranges.
- noir_lord 9y agoIf they consistently used VPN's I would but it seems to be intermittent since the GFWoC is a fickle thing.
- acranox 9y agoI did this with fail2ban. Initially just blocking the IP, and then sweeping back through occasionally and blocking the whole subnet when I had a few offenders in the subnet. There was no negative impact for me.
- emveeoh 9y ago+1 fail2ban (properly configured, of course)
- samstave 9y agoDOS against a range provided by an ISP assuming you could spoof the source over and over, you could have valid ranges auto-blocked for valid services where you DOS a range of IPs from any given ISP
- bitbang 9y agoFwknop is a much better solution than whack-a-mole blacklisting or even Fail2Ban. https://github.com/mrash/fwknop https://github.com/mrash/fwknop
- X86BSD 9y agoI've been using something similar for a while now. I use pam_jail on freebsd to drop the ankle biters using common ssh login attempts like test, ubuntu, oracle etc into a FreeBSD jail where I watch what the do and get a copy of all their tools. I rate limit the outgoing traffic from that jail to something painfully slow to prevent them from causing any major issues. But being able to fire up 'watch' on freebsd and snoop the tty they are on in the jail is awesome for forensics. It's secure, they can't break out of the jail. It's rate limited to prevent them causing much damage to anyone. It's easy to observe every thing they type and do in the jail from the host.
- corybrown 9y agoWhat have you seen them do?
- samstave 9y agoAttempt to log in it would seem...
- ConfucianNardin 9y agoMight have changed by now, but what they used to do was usually: * Download a large file (e.g. XP SP3) from Microsofts servers to test bandwidth * Download IP scanners and trying to run them (in order to find new hosts to attack) * Download botnet code that connects to some C2 server Tools they download were often downloaded as source, which they then tried to compile in order to run it. Searching for e.g. 'youtube kippo' will get you videos of honeypot sessions.
- snuxoll 9y agoThat's...pretty awesome. Have any guides/blog posts/whatever on your setup?
- X86BSD 9y agoI don't. When I was setting this up I didn't find any guides or blogs about doing this so I thought it was not of interest to anyone. So I just went about implementing it a piece at a time. Installing pam_jail, then getting /etc/pam.d/sshd configured to use it (which is a one-line addition, very easy). Now by default pam_jail simply puts the user into a jail of their home directory. This is where you can go a few routes with this. If simply jailing the user on the host OS to their home directory is all you are interested in doing you can stop there. Configure FreeBSD to not allow users to see other processes, user processes and group processes not owned by them, and they won't know anyone else is on the system. security.bsd.see_other_gids security.bsd.see_other_uids The above are the two sysctl's you want to enable. A simple ps will show you all the shells that are in a jail. Tue 19 7:19PM priyanka.setecastronomy ~> ps -x PID TT STAT TIME COMMAND 30308 - S 0:00.05 sshd: oracle@pts/0 (sshd) Now you have the tty he/she is on (pts/0). There are many ways to get that including ps so use whatever method you prefer to get the tty. Fire up watch to snoop on the tty. Tue 19 7:48PM priyanka.setecastronomy ~> doas watch pts/0 And boom, you can watch the ankle biter do his/her thing. You can do this on the host as described, because that is how pam_jail works. But if you prefer for some reason to make an actual jail with VNET etc and do this simply forward ssh on the host to the jail's IP. If you do rate limiting you will already be configuring pf or ipfw anyway so an extra line to forward 22 to the jail host is not big deal. Yeah this is a crappy write-up but it's spur of the moment and in a HN reply post so it's worth the price paid. I'm happy to answer questions or help you if you get stuck anyway I can just message me. It's pretty simple to do. If this is in a corporate setting I would consult with legal before doing this. The world is a whacky place these days, and if the ankle biter does something evil from your honeypot jail you may be liable.
- knoxa2511 9y agoReminds me of this Fishing for Hackers post https://sysdig.com/blog/fishing-for-hackers/ https://sysdig.com/blog/fishing-for-hackers/
- shpx 9y agohttps://xkcd.com/350/ https://xkcd.com/350/
- Myrth 9y agoI almost closed the page on mobile because I thought it's empty or broken...
- ryanlol 9y agoIt's not any less broken on desktops.
- linsomniac 9y agoAside: I used to run a small ISP, a 200-300 dedicated+virtual machines. We set up our router to alert us if outbound SSH connections from a host went above a certain threshold, which was a super reliable way of detecting if a host was compromised. I think we had a near 100% success rate, because once a host is compromised they use it to start trying to compromise other hosts. But, we also had every customer on a VLAN, limited to only being able to send traffic from their IPs, and also blocking incoming and outgoing bogon traffic. Years ago I attended a presentation by Evi Nemeth (RIP) related to CAIDA and one thing they found in auditing "backbone" traffic was that some huge percentage of it was bogon traffic (I don't recall the exact number, but lets say 10% +/- 6%). Nobody wanted to filter that traffic because the pipes were less expensive than the routers to handle filtering packets at high pps rates.
- fujiters 9y agoYou'll never really know your success rate though. You could have had machines compromised for years with small amounts of traffic.
- tobltobs 9y agoI guess he doesn't mean that he detected all infected machines but that near 100% of machines which triggered the alarm were indeed infected.
- antoinealb 9y agoThat's a bad metric though. You could miss a lot of infected hosts.
- bauerd 9y agoOutbound traffic probably wasn't their only heuristic
- marcosdumay 9y agoThat is the single most important metric when you want to create an alert.
- tr1ck5t3r 9y agoBlocking IP addresses/blocks doesnt matter if your ISP has been hacked and attacks from pseudo ip locations are being injected into the communication, easily done with a switch. This way any attack can be made to look like its come from some other part of the world, but as its packet data intended for your server's you are none the wiser. One advantage of this for the attacker is they can use your own systems to block you from genuine locations you want to visit. "However, configuration is sadly not complete, unfortunately SSH servers do not usually listen on port 2220 and you may have noticed that when you are connected to the Honey Pot you are unable to do anything with the internet, try a ping to a website." So if the hacker's ping one of their other botnet's to check its genuine, this honeypot is identified straight away as no ping ever gets out. People have to consider everything online is hostile and you cant trust anything if your ISP has been hacked, this is one of the things Snowden highlighted the 5Eyes has been able to do with ISP's like the main Belgium telecoms company.
- spc476 9y agoOver a decade ago, I set up a tarpit [1] (I worked at a small web hosting company) on an unused machine to catch network scans. I had the idea of using the information gleaned from that to block IP addresses at the router level, but never did get around to it. [1] http://boston.conman.org/2006/01/07.1 http://boston.conman.org/2006/01/07.1
- throwaway613834 9y agoQuestion: I've had an SSH server open to the world for months on a nonstandard port. I've never gotten any attacks logged on fail2ban, to the point where I suspect fail2ban is either not logging the attacks or not operating correctly... even though it logs every login I'm aware of just fine, so it really seems like there really haven't been any attacks. I find this so unlikely. Is this normal? Does no one scan high ports ever? Are there any more plausible explanations?
- perpetualcrayon 9y agoThat sounds like a much better solution than I've used in the past. By simply restricting total # of connections / IP to one my attacks dropped dramatically, but not gone completely. I think it's just a matter of numbers. The IP search space is already pretty large with IPv4 on just a single port. With IPv6 it'll be more or less impossible for attackers to make any substantial progress by just searching for IP addresses to attack, let alone an IPv6 address on a non-standard port. After reading this I'm likely going to be upgrading my technique. SSH on IPv6 only (eventually shutting down IPv4 altogether, can't wait), listening on a non-standard port, limit 1 connection per IP.
- throwaway613834 9y agoYeah, definitely listen on a nonstandard port. I'm not sure how to limit to 1 connection per IP but it sounds rather limiting, so I wouldn't do it... I often have multiple SSH sessions from my own IP.
- perpetualcrayon 9y agoI just run a 'screen' session to open 10 bash sessions simultaneously when I want to do anything substantial. That has never felt restrictive to me.
- throwaway613834 9y agoCan you see two screens simultaneously with screen though?
- smokescreentech 9y agoGreat post. Honeypots behind the firewall are also a great way to pick up lateral movement and privilege escalation. There's a ton of useful open-source tooling one can play with in the space. Here's a list: - https://www.smokescreen.io/practical-honeypots-a-list-of-open-source-deception-tools-that-detect-threats-for-free/ https://www.smokescreen.io/practical-honeypots-a-list-of-ope... Most of the projects are also small enough that you can easily contribute back to them especially in terms of reducing the ability to fingerprint the honeypots. Somewhat timely, The Grugq just wrote up a nice article on Counterintelligence for Cyber Defence: - https://medium.com/@thegrugq/counterintelligence-for-cyber-defence-97d33503064d https://medium.com/@thegrugq/counterintelligence-for-cyber-d...
- YouKnowBetter 9y agoI just harden my sshd config based on https://github.com/arthepsy/ssh-audit https://github.com/arthepsy/ssh-audit and ALL bots will fail since they do not support the more current config os key-exchange, host-key, encryption and message authentication code algorithms. To make it even more interesting: one can actually see which bot is trying by the way they fail to handshake. I like a litte noice on my ports so leave ssh on its default port, accept authentication only with a certificate and be done. Works like a charm. And yes it makes my sshd stand out like a sour thumb since 99.9% (guestimation) of you folks don't tighten up your ssh configs :)