16 ms·
Best Practices for Securing SSH
- deleted 5y ago[deleted]
- theandrewbailey 5y agoIs fail2ban still a good idea? Any reason it's not mentioned here?
- oakwhiz 5y agoI think that if you follow the recommendation to disable password-based authentication, then fail2ban is downgraded from a near-requirement to a defense-in-depth tactic. It's not nearly as important to restrict the number of retries if asymmetric key-based authentication is used, because there is a much larger keyspace to search through than if passwords were used - assuming that the cryptography works as intended.
- tyingq 5y agoI'm a fan of pam_shield, because it's tied right to the auth part versus tailing log files. And you can trigger whatever action you want. I believe the default is just null routing. https://github.com/jtniehof/pam_shield https://github.com/jtniehof/pam_shield There are other, similar pam modules if this one isn't your cup of tea. There's pam_tally2, and others.
- traceroute66 5y agoNot a good idea for two reasons: 1) The very first configuration change anybody should be making to an SSH server should be disabling password-based authentication. Once you've done that you've rendered fail2ban obsolete, because the only real way in for attackers from that point in is via software security vulnerabilities, and fail2ban can't help you with that, that's your job to keep yourself patched up. 2) In an IPv6 world, fail2ban is pointless. The ranges are so vast.
- fulafel 5y agoIt's a way to lock you out if the adversary can source or spoof connections from your usual access network.
- VTimofeenko 5y agoFail2ban has a per-jail ignoreip to not ban specific IPs: https://www.fail2ban.org/wiki/index.php/Whitelist https://www.fail2ban.org/wiki/index.php/Whitelist
- deleted 5y ago[deleted]
- BeefWellington 5y agoAs others noted, probably because once you disable password authentication you're cutting out the ability for most bruteforce attempts to work. However, I still think it's valuable for the following reasons: 1) It can slow stupid attackers down, e.g.: those that don't abort a bruteforce attempt when they see password authentication is disabled. Judging by my logs there's lots of those still out there. 2) It keeps the SSH logs less cluttered. 3) You can use it to build a set of hosts to consider blocking permanently.
- uniqueuid 5y agoAs sad as it makes me, blocking large parts of the world that you don't expect to connect from via a list of CIDR blocks is an incredibly effective way to secure anything and reduce logspam. I personally use nft blackhole [1], which I can recommend for its ease of use. [1] https://github.com/tomasz-c/nft-blackhole https://github.com/tomasz-c/nft-blackhole
- traceroute66 5y agoAbsolutely pointless for two major reasons: 1) Many attackers "hide" behind cloud services. So whatcha gonna do when your primary attack vector is AWS EC2 instances ? Block AWS CIDR ranges ? 2) With IPv4 exhaustion increasing numbers of people will be using IPv4 allocations across geographic boundaries.
- uniqueuid 5y agoLooking at my case, I wouldn't call this pointles, since it removed > 95% of login attempts. Sure, IP address block fragmentation will remove some value, but you'd still kill a lot of juicy compromisable home IP addresses in those countries. Regarding cloud providers - well, I hope they have some automated systems to clamp down, since it's against their own interest to serve as vectors.
- traceroute66 5y ago> since it removed > 95% of login attempts If you use certificate authentication and disable password authentication then you've killed off 100% of unauthorised attempts right off the bat. There really isn't much need to do more than that, its the world's easiest security fix.
- uniqueuid 5y agoSure, I don't think we actually disagree too much. Leaving password auth on is simply negligent. That said, blocking all countries that you don't expect to talk to is the world's second easiest security fix, and protects other processes you might have running, and other unknown vectors that might be worse, such as heartbleed.
- ncmncm 5y agoThe article fails to recommend turning off "KbdInteractiveAuthentication", which used to be called "ChallengeResponseAuthentication", which is another password protocol.
- CaliforniaKarl 5y agoThat's because most two-factor authentication methods rely on keyboard-interactive auth to prompt the user. After using a different auth method (such as public-key), the SSH daemon then presents the need for additional authentication, with keyboard-interactive as the available option. This works by specifying `keyboard-interactive:pam` as the authentication method, and modifying SSH's PAM stack to call whatever PAM module handles the second factor. Notably, any PAM module which asks for a password (for example, "pam_unix") is removed from SSH's PAM stack.
- tialaramex 5y agoIf you can, you should do this via FIDO Security Keys instead. You do need a modern SSH (client and server) and of course money to buy keys for whoever is authenticated, but for employee systems in particular that's a very small expenditure and is re-usable for other problems (e.g. web site authentication with WebAuthn). With Security Keys you can get a physical device you've decided you trust to authenticate that its authorised user is present remotely. Although the keys aren't very bright, they do understand a handful of bitflags they're signing, and two of those bitflags are "User Present" and "User Verified", by requiring "User Verified" the physical device you trust must have verified the authorised user (e.g. local PIN, fingerprint sensor) before signing the message. This approach is more robust, because it's all happening in the SSH public key authentication layer, not in ad-hoc PAM code, and it's simpler because there is no "second factor" data living on your SSH servers, the second factor is a problem for the authenticator only, yet it also re-uses an authenticator your employees can use to e.g. authenticate to your local gitlab install, or your Google docs, or even to decrypt their laptops on startup.
- oofbey 5y agoThe article says to use two-factor-auth (obviously a good idea) but says nothing about HOW you add 2FA to SSH. Does anybody have pointers? I'd love to add 2FA to my bastion hosts, but don't want to put a ton of effort into doing so.
- lnxg33k1 5y agoUsually with authentication on linux, pam is a good starting point https://duo.com/docs/duounix https://duo.com/docs/duounix
- VTimofeenko 5y agoI set up something like this [1] in the past. It relies on pam letting in the user if and only if that user provides the generated number. Do note that messing up pam is easy and may lock you out, so have a backup/snapshot, rescue shell or an open session when you are configuring it. [1]: https://ubuntu.com/tutorials/configure-ssh-2fa#1-overview https://ubuntu.com/tutorials/configure-ssh-2fa#1-overview
- seldomcomment 5y agoyou can do so also without google authenticator in a previous HN post I've copied my quick 'n dirty notes [1]: https://pastebin.com/yYzSrM61 https://pastebin.com/yYzSrM61
- beermonster 5y agoSince OpenSSH >=8.2 you can use FIDO2 and security keys. See https://www.stavros.io/posts/u2f-fido2-with-ssh/ https://www.stavros.io/posts/u2f-fido2-with-ssh/
- deleted 5y ago[deleted]
- R0b0t1 5y agoIt depends what you mean by 2FA. Older definitions allow password protected keyfiles. If you mean active 2FA, you need a PAM plugin and usually a provider, but you can host your own. Look at e.g. Yubikey.
- pphysch 5y agoAnother option is not actually expose SSH at all and proxy shells through a web server via WebSockets, fronted by xterm.js or hterm.js. There are some limitations here (like ctrl-W will get captured by the browser rather than the shell) but it is relatively easy to implement and fits a lot of use cases without the nightmare of fitting Linux PAM to your organization's evolving IAM needs. Won't work for everyone, but definitely something to consider if you are offering "shell-as-a-service" internally or externally.
- R0b0t1 5y agoSuperficially this sounds good but if you don't trust the security of SSH you need to ask why you trust TLS any more.
- pphysch 5y agoIt's not about trusting the security of SSH vs TLS, but rather the ergonomics of deploying either technology. All the Big Cloud providers offer both, last time I checked. Point is, you can offer a secure & functional remote shell without touching Linux PAM or the Linux authx stack beyond `useradd` and a locked-down `sshd_config`. Orgs that already offer web services over HTTPS may find this route desirable.
- DarylZero 5y ago> without touching Linux PAM or the Linux authx stack ..falsely implying you'd have to "touch" this stuff to set up OpenSSH on port 22.
- aborsy 5y agoIf public key authentication is used with secret key in a hardware key/TPM/secure enclave, most other suggestions made don’t help further. Fail2ban is certainly not needed (unless there is potential that some users may use very weak passwords, which password policy shouldn’t permit that, or logs are preferred to be cleaner). Firewalls, public key authentication (verify host keys, also rotate), hardware keys, using SSH over Wireguard, and a secure bastion host provide real security. Preventing SSH agent and X11 forwarding is good too.
- gjs278 5y ago
- rgun 5y agoCan you please elaborate on why preventing ssh agent is good?
- deleted 5y ago[deleted]
- tptacek 5y agoForwarding your agent exposes your authentication secrets to the machines you're connecting to.
- tedunangst 5y agoA bit late, but I feel it's important to clarify it exposes something like "authentication capability" not the actual secrets. It's temporally bounded.
- tptacek 5y agoYeah, I had the same itchy thought when I wrote this; I decided to keep it simple.
- bennyp101 5y agoI usually set up a bastion host that has Tailscale installed on it, with my private key stored on a yubikey. That way you need to be on the Tailscale network, and have my Yubikey/PIN - makes it nice and easy for me to get on from pretty much anywhere if I need to.
- beermonster 5y agoCan also audit using ssh-audit [1][2] [1] https://man.archlinux.org/man/community/ssh-audit/ssh-audit.1.en https://man.archlinux.org/man/community/ssh-audit/ssh-audit.... [2] https://www.ssh-audit.com/ https://www.ssh-audit.com/
- throwJan22 5y agoI try to have IP6 only addresses as it helps with port scanning. I'd think it'd be more common.
- chasil 5y agoRotate your keys. https://www.linuxjournal.com/content/ssh-key-rotation-posix-shell-sunset-nears-elderly-keys https://www.linuxjournal.com/content/ssh-key-rotation-posix-...
- AtlasBarfed 5y agoAt work I have an access tool for writing automation on groups of servers. Basically orchestration without a server. I used to be SSH only, but the framework is built around simply delivering CLI commands and enabling file transfer. So I abstracted the command request/response and now I can do it over AWS-SSM, or docker run, kubectl, salt daemon, teleport, or even AWS-SSM to a "bastion" and then ssh from there. AWS-SSM is basically a polling mechanism, you can easily roll one of your own. What I don't like is two factor authentications that require manual steps. Then you can't automate anything.
- terom 5y agoAnsible connection plugins provide the same, e.g. https://docs.ansible.com/ansible/latest/collections/community/aws/aws_ssm_connection.html https://docs.ansible.com/ansible/latest/collections/communit...
- AtlasBarfed 5y agointeresting, thanks, good to know. The connection stuff is pretty annoying to maintain.
- GEBBL 5y agoImplement port knocking
- dtgriscom 5y agoI've long thought that this would be a great idea, but I've never spent the time to set it up. Any tool recommendations (for Ubuntu)?
- compumike 5y agosudo apt install knockd
- anjbe 5y agoWhy? Putting SSH behind a WireGuard VPN provides stronger security guarantees than port knocking and is built into the OS on both my server and my clients.
- falsenapkin 5y agoWhat happened to single packet authentication? As someone who has casually run hosts, I've been disabling password and setting up keys for most of the 10+ years. Always been curious about SPA though, it seems like a decent way to protect a service, better than firewalling IP ranges and changing ports, no?
- scoodah 5y agoA client I work with uses Appgate SDP which uses SPA to do exactly that. I haven’t seen SPA come up often other than that. For this particular client it was a hard requirement for them to even consider allowing access from public internet.
- mikesabbagh 5y ago> Implement firewalls This is the most important by faaaar. Do this experiment: Start a vm on any cloud with an open port 22, and watch the logs of sshd service. You will be amazed at the number of requests with bad credentials that will hit your machine within minutes. I watched logs to different ports, and ssh wins first place
- mattrighetti 5y agoProbably not as effective as firewalls but I use Fail2Ban to block some of those random requests IP on my local server exposed to the internet
- JimDabell 5y ago> You will be amazed at the number of requests with bad credentials that will hit your machine within minutes. Why are you worried about requests with bad credentials?
- fmajid 5y agoNot mikesabbagh, but depending on your sshd_config's MaxStartups you can easily be locked out of SSH access if the probers have too many active unauthenticated sessions holding up your own from connecting. Having a backup sshd behind something like Wireguard is an inexpensive insurance against this kind of DoS.
- jjav 5y agoYou shouldn't be amazed. Assume that any available service will be probed continuously. There's no harm in that. Sure the log noise can be annoying but be in peace with it. If there are no weak credentials which can allow access, there's no issue. The "if" is important, but tractable.
- patrakov 5y agoThere _is_ harm. At one of the places where I worked previously, we used a few dedicated servers in different cities, and periodically synchronized data from the central one to all others using rsync-over-ssh. Sometimes we got a warning from rsync that the connection was unexpectedly closed. We have traced this warning to SSH credential bruteforcers (yes, completely futile) that exhausted MaxStartups. So, we installed fail2ban.
- angry_octet 5y agoLovely that they are propagating the folklore security of using a non-standard port. And yet they also discuss using a well-known bastion host.
- XorNot 5y agoFor security it's worthless, but for reducing log noise it's not a bad idea if your number of users is low.
- yeetaccount4 5y agoSame with port knocking. It has a practical purpose, but idiots will have it lull them into a false sense of security.
- angry_octet 5y agoI've tried it, and the number of failed logins is still significant, and it can only have gotten worse now, given the ease of scanning the entire IPv4 range. If you only have {2fa,key,certificate} auth the number of alerts you should have from SSHD itself is (almost) zero, failed logins are (almost) _all_ _noise_. Higher level systems that monitor origin/destination/heuristics of successful logins are where its at.
- BenjiWiebe 5y agoI've got a web server with a popular vps company. With ssh on port 22, I (naturally) got lots of failed login attempts. I moved it to another port...0 attempts in the last 7 days (btmp was rotated then). It shuts down log spam 100% for me.
- throw7 5y agoThey really push ssh certificate auth vs key auth. Each has their tradeoffs. If you're going to go through implementing a pki for ssh, I'd throw in something like kerberos also to look into.
- anjbe 5y agoI switched to SSH certificates on all my personal machines nearly four years ago. Compared to plain keys, there are three main differences I’ve noticed in practice. First, I started giving certificates expiration dates, so a compromised key would only be valid for a short time even if I were unable to revoke it manually right away. It provides some confidence that my keys weren’t exfiltrated once and subsequently used behind my back for years and years. Second, when generating a new client keypair, I only have to copy the public key to my certificate signing machine (one copy) rather than every host I plan to log into (many copies). This more than makes up for the minor certificate configuration necessary on new clients. Third, host authenticity warnings are now a thing of the past. Since I switched to certificates, they only appear when something’s actually misconfigured, never simply because I connected to a new host. As a result, I’ve lost the habit of blindly accepting the fingerprint (and in fact I’ve now turned on strict host key checking in the config file so accepting it isn’t possible). Of course, you don’t need certificates for that… if you’re diligent at checking fingerprints. I tried to be, but sometimes I fell short. Not anymore.
- gnufx 5y agoI'm baffled why organizations don't use Kerberos, especially when they already have it (Active Directory).
- ed25519FUUU 5y agoThis advice is not very good. Probably the best thing in this list is using the “AllowUsers” directive (which makes limiting root access a moot point) and using a good, strong key. There’s not much benefit to using a cert over a password-protected ssh private key.
- pigbearpig 5y agoThis sure looks like advertisement for teleport vs any actual advice. A real post would actually tell you how to do these things. Compare it to something like https://www.howtogeek.com/443156/the-best-ways-to-secure-your-ssh-server/ https://www.howtogeek.com/443156/the-best-ways-to-secure-you...
- KennyBlanken 5y agoIt's spam, and I can't believe it got upvoted as much as it did. Any junior sysadmin should consider this "yeah, duh" levels of advice.
- gtm1260 5y agoIs there any simple, secure by default alternative to ssh?
- jesterson 5y agoSSH blocked by firewall for any connections except incoming from VPN IP address. VPN server is mine and located on separate server. This has proven to be working solution for years. Any monstrous "security" constructions with keeping it open or partially open will backfire once one attack vector will be discovered.
- gnufx 5y agoAround the time it was released I tried to interest people in Kuhn's https://www.cl.cam.ac.uk/~mgk25/otpw.html https://www.cl.cam.ac.uk/~mgk25/otpw.html for those travelling and wanting SSH access to the site from potentially-dodgy systems. Still, without your own device you may not be able even to use hardware keys. If the client is untrustworthy, you do still might worry about the channel being open from it to the other end.
- jmnicolas 5y ago> Disable password-based auth What happens if I lose my ssh key? I like passwords because I can remember them so I don't have to put my key on someone else's computer (cloud). So on my server I have one user (non root) with a long password so I don't have access to my keys I can still login.
- fmajid 5y agoAn advertorial puff piece that fails to mention that OpenSSH since 8.2 has had the ability to use FIDO U2F tokens as private keys.
- teitoklien 5y agoI wrote a simple bash script which fetches my current ipv4 address , and then uses aws cli to add that ip address to my whitelist for the ssh port on all my instances. I have a cron job which autoclears all the whitelisted ip addresses at the end of the day. If youre a team, you can always make a similar script and share it with everyone, since aws cli is configured with your team members iam access, you can be assured that they can only whitelist themselves on instances which they have access to over iam. If you dont use aws, just expose an api on your server, protect the endpoint with an api key and use that endpoint to send the whitelisted ip to update your iptables(/whatever firewall you're using). If all of this sounds really complicated to you, you can always just setup wireguard on one of your machines, then make all your team members connect to that vpn, and only whitelist the ip address of that machine across all your instances. That way only people who can authenticate with your vpn can even access your ssh ports.
- kogepathic 5y agoFWIW, Teleport are also the sponsor of Last Week in AWS Security (2022-01-06). It seems someone was told to go market Teleport and posting to HN is free (unlike newsletter sponsorships).
- ArenaSource 5y agoSSH over Tor hidden service