15 ms·

How to SSH Properly

584 points by old-gregg 7y ago

exabrial 7y ago
As an alternative to ssh certs, we lookup keys in an LDAP Cluster. Each approach has pros/cons. SSH certificates are decentralized and resilient against failure, but revocation of a breached certificate isn't instantaneous. Keys-in-LDAP offers less resiliency, but offers instant key revocation. It's also simpler and offers a familiar experience to most devs. The most important thing is to look at your attack surface realistically and plan accordingly. Just blinding putting in an SSH cert authority of an unsecured-non-redundant ldap cluster is worse than no key management at all.
- closeparen 7y ago
  1) Programmatically issue certificates with short lifetimes after authenticating against your IdP. This way if a user is deactivated in LDAP their access will expire on its own after a few hours. 2) Firewall off all but a few bastion hosts and ProxyCommand through those. Immediate revocation can be assured by updating the CRLs on those specific hosts.
  - Spivak 7y ago
    Buy why go through the extra effort compared to just having your servers ask the IdP for the user's keys?
    - cbhl 7y ago
      If your IdP goes down, how will you SSH in to fix it?
      - exabrial 7y ago
        Currently we use the root "physical" console. Generally we avoid IDP outages by keeping our LDAP servers clustered and geographically diversified. Because LDAP is 99.999% of the time reads, not writes, this makes clustering pretty simple.
      - webvictim 7y ago
        Having been on the rough end of this during a huge LDAP outage, I can confirm that LDAP is great until such time as it isn't.
        sytringy05 7y ago
        +1 this is no fun.
  - webvictim 7y ago
    This is definitely the premise of what I was going for with the post. I'm a firm believer in the idea that short-lived certificates which expire by default are one of the best ways to provide access to infrastructure, and enforcing that access comes from a limited list of bastions gives you an easy choke point to withdraw access as desired when you need to.
    - sytringy05 7y ago
      Isn't there a netflix ssh CA that does this?
      - V-eHGsd_ 7y ago
        yes, it's called bless. https://github.com/Netflix/bless https://github.com/Netflix/bless there's also cashier https://github.com/nsheridan/cashier https://github.com/nsheridan/cashier
- jturpin 7y ago
  We do both and it seems to work well - normal users do LDAP, and machine users generate certificates out of Vault that are extremely short lived, 10-30 minutes for CI jobs.
- jldugger 7y ago
  How about asymmetric kerberos?
- vbezhenar 7y ago
  I think that you can use intermediate proxy server. So everyone logs into that proxy server and logs into other server using key from proxy server. Key is protected with passphrase and unlocked with ssh-agent by someone after reboot. If you need to revoke access, you can just revoke access in the proxy server.
- x3haloed 7y ago
  What is your opinion on using a secret management system? https://learn.hashicorp.com/vault/secrets-management/sm-ssh-otp https://learn.hashicorp.com/vault/secrets-management/sm-ssh-...
- inshadows 7y ago
  How is the communication between servers lookup up keys and LDAP protected? Does it run over TLS?
  - chupasaurus 7y ago
    I think any modern LDAP client has TLS support, same goes for servers.
davidcollantes 7y ago
Assuming RSA keys are being used for compatibility?
- Kovah 7y ago
  Wanted to ask the same. A lot of tutorials still use RSA but I see no reason to use it in 2020. ECDSA keys are supported since an OpenSSL version from 2013.
  - msoucy 7y ago
    Some places still have RHEL6, unfortunately, which predates that.
    - floatingatoll 7y ago
      RHEL6 EOL is either in November 2020 or, if you purchase extended support, June 2024. https://access.redhat.com/support/policy/updates/errata https://access.redhat.com/support/policy/updates/errata
  - GekkePrutser 7y ago
    I still use RSA mainly because Yubikey doesn't support them yet with OpenPGP keys.. I felt the increase in security usage was a worthy compromise to using RSA with a key length that's not yet broken. I think Fido2 mode does support it but I don't know whether I trust Fido2 fully for SSH.. Especially because it seems to still store a private key locally, which defeats the whole purpose of using a hardware key (I saw this here: https://developers.yubico.com/SSH/ https://developers.yubico.com/SSH/ ). The whole idea is to not have the private key anywhere but inside the token itself. It's also a bit early in development where RSA has proven itself. It has some known weaknesses (including high quantum computing vulnerability) but they're well understood.
    - tialaramex 7y ago
      No, the FIDO OpenSSH support doesn't really store a private key "locally" but I can see why it's confusing. I'll explain. FIDO was conceived for the Web, and so the SSH authentication model wasn't a core consideration in its design. In SSH authentication always goes like this: Client: "I can prove I'm Alice because I can do X" Server: "OK, do X" Client: (Proves they are Alice by doing X) OR Client: "I can prove I'm Alice because I can do X" Server: "Not good enough. What else?" (Repeat with a different method or give up) Whereas WebAuthn (and U2F and other models FIDO was proposed for) have authentication go like this: Server: Prove you are Alice, if you are you can do Y or Z Client: (Does Z) In FIDO there's a magic "cookie" used, in theory this could be just a "serial number" for a credential, but in reality for a typical FIDO dongle (such as Yubico's Yubikey) it's actually your private key for that site, encrypted using a symmetric key known only to that Yubikey. This way your Yubikey can unlock a genuinely unlimited number of sites, each with completely fresh credentials that can't be correlated, without infinite flash storage. A site says "Here's that cookie you gave me, now prove you're GekkePrutser" and your Yubikey consumes the cookie, decrypts it to get a private key, then uses that private key to prove you're GekkePrutser. But as we saw in SSH the authentication doesn't happen in that order. The Security Key needs the cookie first, but the remote server is waiting to hear how you'll prove who you are first, it's a stalemate. To break the stalemate the cookie is stored on your local system, so that can be fed to the Security Key, and then things can proceed. Down the road apparently the OpenSSH team will add support for FIDO2's "usernameless" behaviour which doesn't need a cookie but does consume limited resources on the Yubikey. If you just can't accept the idea of storing the opaque cookie value because you know it has your private key encrypted inside it, then that mode will suit. It's also useful for "road warriors" who are willing to trust other people's computers briefly but don't carry their own. But that isn't finished today and it seems they don't expect it to be the usual way this is used, since it requires a FIDO2 (not just FIDO) Security Key.
sneak 7y ago
Is active MITM that much of a threat model item that running a whole SSH host key CA is the answer? I can't see that it is. I thought this was going to be about Yubikeys or other HSMs.
- kasey_junk 7y ago
  The issue is you want cert based auth that is easily revoked. So you set up very short lived certs Yubikrys and the like is then only used for exceptional cases .
- munchbunny 7y ago
  It's not really MITM that's the issue. It's about maintaining access control for SSH access at scale. Once you have a big enough team, and especially if you store sensitive customer data, you generally want to start thinking about a system that can temporarily provision SSH access on demand so that nobody has standing access, at which point going through a CA and bastion servers is a canonical approach. Exactly how big your team needs to be is a judgement call, but I think a reasonable threshold where you really should start thinking about CAs for SSH access is around where it's no longer possible for the dev ops people to be aware of what everyone who needs production access is working on.
  - sneak 7y ago
    I hadn't realized that this blog post is from the same people who make Teleport, which is a bastion/logging/access ssh proxy for solving exactly that problem. It seems to me that the linked article is probably just content marketing spam (much like Digital Ocean's extensive "how to install $x on a vps" posts) to boost the domain's pagerank/authoritativeness. I still don't think people should be using SSH CAs for anything that isn't a short-lived, single-use bastion/proxy key. They're on the right track with Teleport; not so much with this standalone article.
    - dielectricboog 7y ago
      Those how to install X on a VPS articles are requested by active customers, occasionally written by customers, and consumed far more often than you think by customers. DigitalOcean responded to Linode Library (as it was then called) when building that out, and having been somewhat involved in the development of Linode Library, I can assure you that neither SEO nor PageRank was near the top of the rationale list. Maybe tenth on the list as a nice bonus. The people who worked on Linode Library in the beginning were engineers, not social media people nor content marketers. The person who wrote its first CMS is a former nuclear submariner and Perl guy with like nine followers on Twitter. I’m sure you imagine a huge meat grinder spam operation paying by the word, but basically one engineer thanklessly wrote more than half of it in the beginning and went on to a distinguished career at 10gen/Mongo, not sitting on Mechanical Turk. I’d bet DO’s operation is identical and exists for the same reasons, and they’re just as proud of it as I am my extremely limited contributions to an offering that objectively improves the operations discipline. It’s annoying to watch valuable work that expands the profession get dunked on and called spam by a snide comment for absolutely no reason. There’s no reason to be so cynical about things in a hot take, particularly when it was an irrelevant shot in context. Not everyone is blessed with innate knowledge nor a gift for studying man pages, and those docs are useful checklists to make sure I didn’t forget anything with common software like ntpd to this day in my own, principal level, career. People learn Unix operations by buying a Linode. For a $10/month VPS you also get a bunch of tested, free guides that work on them and teach you how to do things. They’re not paywalled to customers and show up in your Google results. I know. The abject horror is almost unbearable.
sandGorgon 7y ago
what's your opinion of using epass2003/pkcs11 for ssh? https://stuff.purdon.ca/?page_id=70 https://stuff.purdon.ca/?page_id=70 https://sigg-iten.ch/learningbits/2014/11/13/first-steps-with-the-feitian-epass2003-smart-token-in-os-x/ https://sigg-iten.ch/learningbits/2014/11/13/first-steps-wit...
- jlgaddis 7y ago
  Oh wow, I had forgotten all about the ePass2003. I've still got one here, but it's probably been 10 years since I've used it. I haven't forgotten how much of a PITA it was to get it working, though!
otterley 7y ago
On AWS you can use SSM Session Manager and gain secure shell access to instances without using SSH at all. As long as the instance has the SSM Agent on it and network connectivity to the SSM API, it just works: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html https://docs.aws.amazon.com/systems-manager/latest/userguide... (Disclaimer: I work for AWS, but opinions are my own and not necessarily those of my employer.)
- pritambaral 7y ago
  Does it work with the `ssh(1)` client? I'd be okay with using this as long as it replaced _only_ the network-connectivity and authentication parts. I do use many of the other features made possible by SSH: 1. Network forwarding (local-to-remote, remote-to-local, dynamic, and unix socket support). The link you supplied mentions only local-to-remote. 2. Agent-forwarding (don't worry, I have confirmation-on-every-use, see my other comment: https://news.ycombinator.com/item?id=22753590 https://news.ycombinator.com/item?id=22753590). This I use all the time to be able to authenticate to other SSH servers and git hosts. This is a must-have for me now for remote development and pair-programming. 3. sshfs, sftp, and rsync. Again, absolute beauties at the job of managing files remotely. This is just off the top of my head, because I use these features day-to-day, but there's many other nice things ssh does that have nothing to do with the underlying connection or authentication method.
  - tialaramex 7y ago
    Agent forwarding is powerful but dangerous. Prefer ProxyJump where applicable. With Agent forwarding any authorised machine you connect to (and thus anyone who controls that machine) gets to use your credentials for the duration, to my knowledge no general purpose clients give you feedback on this usage, so you won't know if this happened - if Agent forwarding is enabled it can be used even if you never use it. With ProxyJump the intermediate doesn't get any view of your credentials, not even whether those are the same credentials you used for that intermediate host, or different. It is only enabling you to connect to a host it can reach that you can't reach directly and nothing more.
dastx 7y ago
All of this is done for you if you set up Hashicorp Vault. In case someone doesn't know, it provides secrets as a service that allows you to store and retrieve static secrets as well as dynamic secrets. The great thing about it is that you can set up authentication from multiple sources including EC2/GCE instances, LDAP and much more. But it also allows you to generate dynamic secrets including, but not limited to, SSH certs, IAM users, AD users, sql users and much more. It's Hashicorps most mature tool and should be in everyone's toolchain.
- outworlder 7y ago
  Vault requires a non-trivial amount of setup and configuration, and then you need to configure SSHD to use it. You also have to maintain the vault server, HA, yada yada. And educate people on how to use it. Vault truly shines in its ability to provide ephemeral credentials. It can even "package" arbitrary secrets for you, and ensure that they are only ever retrieved once. It's a fantastic piece of software, one that more companies should be using, but it is not a push button deployment. It is borderline impossible to implement in many 'traditional' organizations.
  - eeZah7Ux 7y ago
    > truly shines in its ability to provide ephemeral credentials. It can even "package" arbitrary secrets for you Those credentials should not be generated, stored or provided centrally in the first place.
Ayesh 7y ago
- Use Ed25519 keys. They are now supported in pretty much every server out there (all recent OpenSSH versions, GitHub, GitLab, etc). RSA 2048 keys are unbreakable for the foreseeable future, and using 4096 bit keys are just being paranoid with no gain. You can fit 4x Ed25519 keys in a tweet. - Setup and use ssh-agent. They make the life so easy. - You can use Yubikey and pretty much every other solution out there with with OpenSSH now.
- mason55 7y ago
  > They are now supported in pretty much every server out there Sadly whatever library Spring Integration uses does not support them yet, as I found out the hard way. I just tried to search for it and couldn't find the reference but just dealt with it a month or two ago.
- ddevault 7y ago
  >Setup and use ssh-agent. They make the life so easy. My #1 ssh usability tip: put this into ~/.ssh/config: AddKeysToAgent yes It'll automatically add keys to your agent the first time you use them during a session, so you don't need a separate step for adding keys every time you log in.
  - res0nat0r 7y ago
    I've not noticed this option before, so will this replace keychain? I install this on every new laptop I use so I essentially only have to enter my ssh-agent password for the key once per reboot, and every bash shell hooks into it automatically. https://www.funtoo.org/Keychain https://www.funtoo.org/Keychain
  - Lammy 7y ago
    Be careful; there are plenty of ways ssh-agent can bite you too. Here are a few: http://rabexc.org/posts/pitfalls-of-ssh-agents http://rabexc.org/posts/pitfalls-of-ssh-agents
    - ddevault 7y ago
jontro 7y ago
How is ssh certificates the next level up? Feels like the added complexity of using this is might be worthwhile for larger orgs but maybe not smaller shops
- kempbellt 7y ago
  I second this perspective. It might be useful for a larger org, but for my use case and that of my team, I don't see much value-add, and the added complexity would likely make managing this system more difficult than it is worth. I've also never had an issue with key management, like the article suggests is common. Maybe I'm not the target audience though.
- yjftsjthsd-h 7y ago
  Is one person a small enough shop? ;) I found it useful for personal systems because it meant that I didn't need to verify keys (user keys and host keys) for every server/laptop combination.
- old-gregg 7y ago
  Disclaimer: I work at Gravitational, one of our engineers is the author of the article. The added complexity is indeed significant, that's why we built Teleport [1] which is much, much simpler open source alternative to OpenSSH for such use case (the blog post does not mention Teleport because it was written for OpenSSH users). But generally, certificates have a number of advantages: they expire quickly (you can have one-shot certs), they contain your full name, email address, or whatever you wish via metadata, this allows to implement RBAC on top of SSH. They address "trust on first use" issue, and they're easy to synchronize with other protocols, i.e. you can have the same CA issue certificates for both Kubernetes and SSH (with identical RBAC rules). When implemented properly, they're actually easier to use. The whole world uses certificates to do online banking and shopping, there's no reason not to have the same seamless experience for SSH. [1] https://gravitational.com/teleport https://gravitational.com/teleport
  - Gaelan 7y ago
    btw, huge props for not mentioning teleport in that blog post. So many companies have blog posts that masquerade as tutorials for "how to do X" but then change halfway through to "OOOORRRR you can not do all that complicated stuff and just buy our product instead," which feels pretty scummy. Thanks for taking the high road.
nelsonmarcos 7y ago
It's a long article written on April 1st. Please, don't say it's an april's fool joke.
jonluca 7y ago
The title of this doesn't properly match the contents. This is basically a how to set up an enterprise SSH environment using certificates. This doesn't tell you anything about "How to SSH Properly".
- Aachen 7y ago
  I almost thought so too and was itching to hit back and post an annoyed comment, but there are actually two more sections: using 2FA (straightforward TOTP branded by our favorite not-evil corp though I've not no idea what they've got to do with it) and setting up a jump host / bastion host. So maybe the title shouldn't be "I like ssh certs" but "ssh security for organisations", but your point still stands.
  - mynameisvlad 7y ago
    > though I've not no idea what they've got to do with it) It's the name of the TOTP PAM module they use, which was created by Google: https://github.com/google/google-authenticator-libpam https://github.com/google/google-authenticator-libpam
- webvictim 7y ago
  Author here - thanks for the feedback. As another reply points out, I did try to also cover the use of a bastion host along with one form of 2-factor authentication. I'm considering doing a future post on how to set up U2F for SSH with hardware devices (like a Yubikey) as well. I'm curious if you have anything else you'd like to see on this topic.
- jlgaddis 7y ago
  The company that authored this "How to SSH Properly" blog post sells a product that will enable you to use SSH "properly" (according to their definition). That's not a coincidence, of course. It's purely sales and marketing. To be clear, the ultimate goal of this blog post is simply to get you or your employer to give them some of your money -- that's it. (To be clear, giving an external entity full control over authentication and authorization of your critical servers is definitely not doing "SSH properly", IMO!)
egwor 7y ago
One thing I find useful is to config default usernames. You can do this by adding a line like below into ~/.ssh/config Host alias domain.com HostName domain.com User aUser This allows you to just write `ssh alias` and it will convert it to ssh aUser@domain.com.
- megavolcano 7y ago
  you can also specify which private key to present using IdentifyFile /path/to/key/file when I have different hosts that require different keys to access, this is very useful so I don't have to specify them every time and agent forwarding is useful as well, for example, I can push to github from a vm using my laptop ssh key and I don't have to generate an additional key for the vm ForwardAgent true with the new Windows terminal, I setup different tabs to be SSH to different machines based on my .ssh/config file { "guid": "{1c9b268e-7606-4a07-b097-d8bc62fb5207}", "hidden": false, "name": "ubuntu.localdomain", "commandline": "ssh.exe me@ubuntu", "colorScheme": "Elementary" }, { "guid": "{b7161acc-8235-4283-9a85-e93df3d09125}", "hidden": false, "name": "db.localdomain", "commandline": "ssh.exe me@db", "colorScheme": "Ubuntu" }, Windows Terminal with a very nicely define ssh config is a killer combo.
  - inetknght 7y ago
    If you have more than one private key in your agent, then SSH will try each one sequentially. That can lead to mistakenly getting banned by monitoring software for what can appear indistinguishable to be an authentication attack. Not to mention the wrongness of presenting "any and all keys" to any host you connect to. If you have multiple keys in your agent, you _really_ should manually configure what key to present using `-i` or `IdentityFile`. But you should also use `IdentitiesOnly` too.
    - megavolcano 7y ago
      thanks for the tips!
perlgeek 7y ago
Somewhat off-topic, but I've noticed that ssh-keygen is much faster at generating key pairs that GnuPG, due to the latter wanting more "proper" entropy. Does that mean that GnuPG is overly paranoid? Or ssh-keygen's keys potentially insecure? (I really hope not). There must be some good explanation for this huge speed difference.
- Aachen 7y ago
  The difference must be urandom vs random. You'll find a lot of words spent arguing the misinformed vs the enlightened if you look for those terms, on HN or elsewhere. I'm in the urandom camp and think that's the only thing that makes sense on the condition that it blocks/errors until seeded properly once, which means so much as that gpg is misinformedly paranoid in my opinion and that if ssh keygen had been insecure it would have been fixed long ago, but do read up and form your own opinion.
  - emmelaich 7y ago
    I don't think it's urandom v random. It really just waits longer.
- vbezhenar 7y ago
  GnuPG is overly paranoid. Not a bad thing for something that performed very rarely.
et1337 7y ago
There are a number of turnkey solutions for this. One is ScaleFT[1]. My company is another: StrongDM[2]. Our product also keeps audit logs for compliance. [1] https://www.scaleft.com/ https://www.scaleft.com/ [2] https://www.strongdm.com/ https://www.strongdm.com/
- old-gregg 7y ago
  https://gravitational.com/teleport https://gravitational.com/teleport Free and open source, made by the authors of the article.
  - jlgaddis 7y ago
    > ... made by the authors of the article. And, just so everyone is clear, your employer.
ansible 7y ago
With regards to using ssh-agent, this is what I have in my .bash_profile: export SSH_ENV=$HOME/.ssh/environment_${HOSTNAME} function start_agent { echo -n "Initialising new SSH agent... " touch ${SSH_ENV} chmod 600 ${SSH_ENV} /usr/bin/ssh-agent -s | sed 's/^echo/#echo/' > ${SSH_ENV} echo "succeeded." . ${SSH_ENV} > /dev/null } # Don't use Gnome keyring for SSH if [[ $SSH_AUTH_SOCK =~ /run/user ]]; then unset SSH_AUTH_SOCK unset SSH_AGENT_PID fi # Source SSH settings, if applicable ssh-add -l > /dev/null 2>&1 if [ $? -eq 0 -o $? -eq 1 ]; then echo "SSH Agent found." else echo -n "Looking for SSH Agent... " if [ -f "${SSH_ENV}" ]; then . ${SSH_ENV} > /dev/null ssh-add -l > /dev/null 2>&1 if [ $? -eq 0 -o $? -eq 1 ]; then echo "found pid: ${SSH_AGENT_PID}." else echo "not found." start_agent; fi else start_agent; fi ssh-add -l fi This starts up a new ssh-agent if it can't find one, but uses the same agent if it is already running on this machine (I remote into a bunch of systems). So even if I use a bunch of screen sessions, or remote in to a machine multiple times, it will always try to use the same agent.
- jpeeler 7y ago
  Have you considered using https://github.com/funtoo/keychain https://github.com/funtoo/keychain? Edit: I see it's been mentioned elsewhere here.
  - ansible 7y ago
    Well, that's interesting. And a whole bunch of code. From a quick skim, it is basically solving the same problem in a different style. I didn't want or need to wrap ssh-agent. I'm fine with using it and ssh-add by themselves. I just wanted to automatically find and re-use an existing agent rather than spawning a new one.
- Spivak 7y ago
  Why not use the GNOME keyring as your SSH agent? It's a full agent that you don't really need to think about.
prussian 7y ago
Host certificates are fine, but user certs are pointless landmines when features like AuthorizedKeysCommand exist. why worry about revocations, expiries and all that noise when you can simply have some script just pull the allowed keys? sure maybe login is a bit slower, but does that matter much?
- webvictim 7y ago
  Don't get me wrong, using AuthorizedKeysCommand is a lot better than having a static ~/.ssh/authorized_keys file on a server, but it isn't anything like as powerful as using user certificates. Certificates can do a lot more than authorized keys can, like enforcing the use of specific principals, commands and options and embedding that information into the file itself without needing to modify each server's SSH configuration. They're also self-contained and will still work in situations where some external service providing a list of keys goes down. I've been on the rough side of a huge LDAP outage which prevented necessary access to the infrastructure to fix it, and it was a horrible experience. There's none of that problem with certificates as long as you make sure you have one which is currently valid. I'm also generally of the opinion that it's safer to enforce the use of authentication which expires by default rather than relying on some external process to do that for you.
  - Spivak 7y ago
    But AuthorizedKeysCommand and certs are at least equally powerful because they're both ways of specifying the content of the same authorized_keys file.
    - webvictim 7y ago
      It's something of an implementation detail - you don't generally specify the usage of certs on a user-by-user level, you do it by trusting the entire CA in /etc/ssh/sshd_config and then using the signed content of the individual cert (expiry date, principals etc) to dictate whether someone should be allowed to get access or not. Look at it in terms of building in a decision at compile-time rather than at runtime. With AuthorizedKeysCommand, you're running something just-in-time on an SSH login to determine whether something should be allowed to proceed. With a CA and a process for issuing certificates, that decision is made at the time the cert is issued and then the cert is good for the duration it's issued for. It's entirely self-contained as sshd itself is making the decision about whether the cert is within its validity period or not. It's obviously a decision that people can make based on their own infrastructure, but my opinion is that the compile-time model is more reliable as it's a fully self-contained system and doesn't rely on an entire fleet of servers being able to connect back to an external service at runtime to determine whether you should be allowed to log in. That sort of thing invariably comes back to bite you when you really _need_ to be able to log in and you can't because the external service is down.
i9wsu 7y ago
Just update and use Use Ed25519 keys resource of how to: http://wolf-tm.com http://wolf-tm.com
293984j29384 7y ago
Slightly off topic but the author recommends expiry periods for certificates. Other than rotating a key because of compromise, this seems unnecessary. We've finally decided that passwords should no longer be mandatory rotated, why should certificates?
- stephenr 7y ago
  How do you revoke access if the certificate has no expiry?
  - hvidgaard 7y ago
    If it's for SSH, you should be able to blacklist that particular certificate. You want to do this to revoke access anyway. Only downside is an every growing black list file, but that is probably a problem you need centuries of use to actual need to solve.
  - 293984j29384 7y ago
    Forgive my ignorance but what's the correlation between expiry and revoking? I'd assume there are several reasons you'd want to revoke a certificate such as it's been compromised. How would it's expiry date help that?
    - webvictim 7y ago
      Author here. My take on this is that fail-closed is a vastly better security model than fail-open. I am genuinely surprised that OpenSSH actually issues certificates with no expiry date as a default. If you have a certificate which expires within a day by default then an unsuccessful revocation is no longer a huge cause of stress. In the worst case, you lock down access to your bastions and disallow the issue of any future certificates for that user. Within a day, any potential threat from that certificate has vanished. This seems preferable to having a mandatory requirement of an up-to-date revocation database which is synced everywhere.
      - 293984j29384 7y ago
kitotik 7y ago
It seems weird there wasn’t more emphasis on the implications of running your own CA. I can imagine lots of people coming across this article and doing a quick copy pasta not realizing what just happened.
m463 7y ago
I like that with macos, you can create keys with a passphrase, and then that passphrase can be provided by the keychain which is unlocked at login ssh-keygen -t ed25519 -b 521 -f .ssh/id_foo_ed25519 cat id_foo_ed25519.pub | ssh foo tee -a .ssh/authorized_keys ssh-add -K ~/.ssh/id_foo_ed25519 along with: Host foo HostName foo.bar User me StrictHostKeyChecking yes IdentitiesOnly yes IdentityFile ~/.ssh/id_foo_ed25519
- LilBytes 7y ago
  Yeah this is a huge time saver.
- terrywang 7y ago
  In addition to the traditional | way of adding public key to target hosts, ssh-copy-id [1] is a wrapper script (shipped by most distros, for macOS it is a separate formula) which can be used to batch add a public key target hosts (love to use it prep for Ansible). Put the target hosts in a file e.g. ssh_ducks while read -r i; do ssh-copy-id -i /path/to/ops_ed25519.pub <user>@$i -f; done <ssh_ducks storm (ssh like a boss), it works well when managing (~/.ssh/config) in CLI / scripts but it has an annoying bug [2] (not yet fixed) that will silently convert ssh_config Host keywords to lowercase (luckily ONLY arguments are case sensitive, NOT keywords, otherwise it'll be easier to detect), annoying. Otherwise, for small to medium scale SSH access, I prefer to use jumpserver (open source 4A bastion / remote access gateway) behind VPN (WireGuard of course, for now, self-hosted or Tailscale), a lot easier to deploy in comparison to Teleport, user-friendly UI for average users. [1]: https://github.com/openssh/openssh-portable/blob/master/contrib/ssh-copy-id https://github.com/openssh/openssh-portable/blob/master/cont... [2]: https://github.com/emre/storm/issues/157 https://github.com/emre/storm/issues/157
  - ryall 7y ago
    Ansible can prep a host for Ansible...
- deleted 7y ago
  [deleted]
anilakar 7y ago
I've found it a good practice to add a YYYYMMDD serial number: -z 20200402 That way, keys generated within a certain time interval can be revoked with a simple serial number check.
fractalf 7y ago
Is there a way to set up 2 ways auth with andOTP instead of the google auth?
ZoomZoomZoom 7y ago
More fitting to the title would be this page: https://infosec.mozilla.org/guidelines/openssh https://infosec.mozilla.org/guidelines/openssh It's a nice practical guideline with all the basics covered.
rauhl 7y ago
On a related topic, what is the current best practice for avoiding TOFU for cloud providers? Sure, SSH certificates provide that for normal users, but how does the SSH CA know — ideally in an automated fashion — what that public key is? An admin can SSH into the machine to check it, but then he has to trust the key he sees. He could access it via the console, but that is a manual process. All the other approaches I can think of involve generating at least one key outside the machine which uses it, which seems like a poor solution.
- wraithm112 7y ago
  At least on AWS, you can parse the System Console Output. On the first boot, the box dumps its pub keys there. We have a tool that parses the output: https://github.com/bitnomial/aws-ec2-knownhosts https://github.com/bitnomial/aws-ec2-knownhosts Right now, this is pretty specific to our use of it, specifically our use with Terraform EC2 instances. We'd happily suggest changes to make it more generic. But you can see the parsing logic there.
GRMrGecko 7y ago
This is perfect timing as I just wrote this tool to help sign ssh host keys and user keys. https://github.com/GRMrGecko/ssh-cert https://github.com/GRMrGecko/ssh-cert

584 points by old-gregg 7y ago

As an alternative to ssh certs, we lookup keys in an LDAP Cluster. Each approach has pros/cons. SSH certificates are decentralized and resilient against failure, but revocation of a breached certificate isn't instantaneous. Keys-in-LDAP offers less resiliency, but offers instant key revocation. It's also simpler and offers a familiar experience to most devs. The most important thing is to look at your attack surface realistically and plan accordingly. Just blinding putting in an SSH cert authority of an unsecured-non-redundant ldap cluster is worse than no key management at all.

closeparen 7y ago
1) Programmatically issue certificates with short lifetimes after authenticating against your IdP. This way if a user is deactivated in LDAP their access will expire on its own after a few hours. 2) Firewall off all but a few bastion hosts and ProxyCommand through those. Immediate revocation can be assured by updating the CRLs on those specific hosts.
- Spivak 7y ago
  Buy why go through the extra effort compared to just having your servers ask the IdP for the user's keys?
  - cbhl 7y ago
    If your IdP goes down, how will you SSH in to fix it?
    - exabrial 7y ago
      Currently we use the root "physical" console. Generally we avoid IDP outages by keeping our LDAP servers clustered and geographically diversified. Because LDAP is 99.999% of the time reads, not writes, this makes clustering pretty simple.
    - webvictim 7y ago
      Having been on the rough end of this during a huge LDAP outage, I can confirm that LDAP is great until such time as it isn't.
      - sytringy05 7y ago
        +1 this is no fun.
- webvictim 7y ago
  This is definitely the premise of what I was going for with the post. I'm a firm believer in the idea that short-lived certificates which expire by default are one of the best ways to provide access to infrastructure, and enforcing that access comes from a limited list of bastions gives you an easy choke point to withdraw access as desired when you need to.
  - sytringy05 7y ago
    Isn't there a netflix ssh CA that does this?
    - V-eHGsd_ 7y ago
      yes, it's called bless. https://github.com/Netflix/bless https://github.com/Netflix/bless there's also cashier https://github.com/nsheridan/cashier https://github.com/nsheridan/cashier
jturpin 7y ago
We do both and it seems to work well - normal users do LDAP, and machine users generate certificates out of Vault that are extremely short lived, 10-30 minutes for CI jobs.
jldugger 7y ago
How about asymmetric kerberos?
vbezhenar 7y ago
I think that you can use intermediate proxy server. So everyone logs into that proxy server and logs into other server using key from proxy server. Key is protected with passphrase and unlocked with ssh-agent by someone after reboot. If you need to revoke access, you can just revoke access in the proxy server.
x3haloed 7y ago
What is your opinion on using a secret management system? https://learn.hashicorp.com/vault/secrets-management/sm-ssh-otp https://learn.hashicorp.com/vault/secrets-management/sm-ssh-...
inshadows 7y ago
How is the communication between servers lookup up keys and LDAP protected? Does it run over TLS?
- chupasaurus 7y ago
  I think any modern LDAP client has TLS support, same goes for servers.

davidcollantes 7y ago

Assuming RSA keys are being used for compatibility?

Kovah 7y ago
Wanted to ask the same. A lot of tutorials still use RSA but I see no reason to use it in 2020. ECDSA keys are supported since an OpenSSL version from 2013.
- msoucy 7y ago
  Some places still have RHEL6, unfortunately, which predates that.
  - floatingatoll 7y ago
    RHEL6 EOL is either in November 2020 or, if you purchase extended support, June 2024. https://access.redhat.com/support/policy/updates/errata https://access.redhat.com/support/policy/updates/errata
- GekkePrutser 7y ago
  I still use RSA mainly because Yubikey doesn't support them yet with OpenPGP keys.. I felt the increase in security usage was a worthy compromise to using RSA with a key length that's not yet broken. I think Fido2 mode does support it but I don't know whether I trust Fido2 fully for SSH.. Especially because it seems to still store a private key locally, which defeats the whole purpose of using a hardware key (I saw this here: https://developers.yubico.com/SSH/ https://developers.yubico.com/SSH/ ). The whole idea is to not have the private key anywhere but inside the token itself. It's also a bit early in development where RSA has proven itself. It has some known weaknesses (including high quantum computing vulnerability) but they're well understood.
  - tialaramex 7y ago
    No, the FIDO OpenSSH support doesn't really store a private key "locally" but I can see why it's confusing. I'll explain. FIDO was conceived for the Web, and so the SSH authentication model wasn't a core consideration in its design. In SSH authentication always goes like this: Client: "I can prove I'm Alice because I can do X" Server: "OK, do X" Client: (Proves they are Alice by doing X) OR Client: "I can prove I'm Alice because I can do X" Server: "Not good enough. What else?" (Repeat with a different method or give up) Whereas WebAuthn (and U2F and other models FIDO was proposed for) have authentication go like this: Server: Prove you are Alice, if you are you can do Y or Z Client: (Does Z) In FIDO there's a magic "cookie" used, in theory this could be just a "serial number" for a credential, but in reality for a typical FIDO dongle (such as Yubico's Yubikey) it's actually your private key for that site, encrypted using a symmetric key known only to that Yubikey. This way your Yubikey can unlock a genuinely unlimited number of sites, each with completely fresh credentials that can't be correlated, without infinite flash storage. A site says "Here's that cookie you gave me, now prove you're GekkePrutser" and your Yubikey consumes the cookie, decrypts it to get a private key, then uses that private key to prove you're GekkePrutser. But as we saw in SSH the authentication doesn't happen in that order. The Security Key needs the cookie first, but the remote server is waiting to hear how you'll prove who you are first, it's a stalemate. To break the stalemate the cookie is stored on your local system, so that can be fed to the Security Key, and then things can proceed. Down the road apparently the OpenSSH team will add support for FIDO2's "usernameless" behaviour which doesn't need a cookie but does consume limited resources on the Yubikey. If you just can't accept the idea of storing the opaque cookie value because you know it has your private key encrypted inside it, then that mode will suit. It's also useful for "road warriors" who are willing to trust other people's computers briefly but don't carry their own. But that isn't finished today and it seems they don't expect it to be the usual way this is used, since it requires a FIDO2 (not just FIDO) Security Key.

sneak 7y ago

Is active MITM that much of a threat model item that running a whole SSH host key CA is the answer? I can't see that it is. I thought this was going to be about Yubikeys or other HSMs.

kasey_junk 7y ago
The issue is you want cert based auth that is easily revoked. So you set up very short lived certs Yubikrys and the like is then only used for exceptional cases .
munchbunny 7y ago
It's not really MITM that's the issue. It's about maintaining access control for SSH access at scale. Once you have a big enough team, and especially if you store sensitive customer data, you generally want to start thinking about a system that can temporarily provision SSH access on demand so that nobody has standing access, at which point going through a CA and bastion servers is a canonical approach. Exactly how big your team needs to be is a judgement call, but I think a reasonable threshold where you really should start thinking about CAs for SSH access is around where it's no longer possible for the dev ops people to be aware of what everyone who needs production access is working on.
- sneak 7y ago
  I hadn't realized that this blog post is from the same people who make Teleport, which is a bastion/logging/access ssh proxy for solving exactly that problem. It seems to me that the linked article is probably just content marketing spam (much like Digital Ocean's extensive "how to install $x on a vps" posts) to boost the domain's pagerank/authoritativeness. I still don't think people should be using SSH CAs for anything that isn't a short-lived, single-use bastion/proxy key. They're on the right track with Teleport; not so much with this standalone article.
  - dielectricboog 7y ago
    Those how to install X on a VPS articles are requested by active customers, occasionally written by customers, and consumed far more often than you think by customers. DigitalOcean responded to Linode Library (as it was then called) when building that out, and having been somewhat involved in the development of Linode Library, I can assure you that neither SEO nor PageRank was near the top of the rationale list. Maybe tenth on the list as a nice bonus. The people who worked on Linode Library in the beginning were engineers, not social media people nor content marketers. The person who wrote its first CMS is a former nuclear submariner and Perl guy with like nine followers on Twitter. I’m sure you imagine a huge meat grinder spam operation paying by the word, but basically one engineer thanklessly wrote more than half of it in the beginning and went on to a distinguished career at 10gen/Mongo, not sitting on Mechanical Turk. I’d bet DO’s operation is identical and exists for the same reasons, and they’re just as proud of it as I am my extremely limited contributions to an offering that objectively improves the operations discipline. It’s annoying to watch valuable work that expands the profession get dunked on and called spam by a snide comment for absolutely no reason. There’s no reason to be so cynical about things in a hot take, particularly when it was an irrelevant shot in context. Not everyone is blessed with innate knowledge nor a gift for studying man pages, and those docs are useful checklists to make sure I didn’t forget anything with common software like ntpd to this day in my own, principal level, career. People learn Unix operations by buying a Linode. For a $10/month VPS you also get a bunch of tested, free guides that work on them and teach you how to do things. They’re not paywalled to customers and show up in your Google results. I know. The abject horror is almost unbearable.

sandGorgon 7y ago

what's your opinion of using epass2003/pkcs11 for ssh? https://stuff.purdon.ca/?page_id=70 https://stuff.purdon.ca/?page_id=70 https://sigg-iten.ch/learningbits/2014/11/13/first-steps-with-the-feitian-epass2003-smart-token-in-os-x/ https://sigg-iten.ch/learningbits/2014/11/13/first-steps-wit...

jlgaddis 7y ago
Oh wow, I had forgotten all about the ePass2003. I've still got one here, but it's probably been 10 years since I've used it. I haven't forgotten how much of a PITA it was to get it working, though!

otterley 7y ago

On AWS you can use SSM Session Manager and gain secure shell access to instances without using SSH at all. As long as the instance has the SSM Agent on it and network connectivity to the SSM API, it just works: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html https://docs.aws.amazon.com/systems-manager/latest/userguide... (Disclaimer: I work for AWS, but opinions are my own and not necessarily those of my employer.)

pritambaral 7y ago
Does it work with the `ssh(1)` client? I'd be okay with using this as long as it replaced _only_ the network-connectivity and authentication parts. I do use many of the other features made possible by SSH: 1. Network forwarding (local-to-remote, remote-to-local, dynamic, and unix socket support). The link you supplied mentions only local-to-remote. 2. Agent-forwarding (don't worry, I have confirmation-on-every-use, see my other comment: https://news.ycombinator.com/item?id=22753590 https://news.ycombinator.com/item?id=22753590). This I use all the time to be able to authenticate to other SSH servers and git hosts. This is a must-have for me now for remote development and pair-programming. 3. sshfs, sftp, and rsync. Again, absolute beauties at the job of managing files remotely. This is just off the top of my head, because I use these features day-to-day, but there's many other nice things ssh does that have nothing to do with the underlying connection or authentication method.
- tialaramex 7y ago
  Agent forwarding is powerful but dangerous. Prefer ProxyJump where applicable. With Agent forwarding any authorised machine you connect to (and thus anyone who controls that machine) gets to use your credentials for the duration, to my knowledge no general purpose clients give you feedback on this usage, so you won't know if this happened - if Agent forwarding is enabled it can be used even if you never use it. With ProxyJump the intermediate doesn't get any view of your credentials, not even whether those are the same credentials you used for that intermediate host, or different. It is only enabling you to connect to a host it can reach that you can't reach directly and nothing more.

dastx 7y ago

All of this is done for you if you set up Hashicorp Vault. In case someone doesn't know, it provides secrets as a service that allows you to store and retrieve static secrets as well as dynamic secrets. The great thing about it is that you can set up authentication from multiple sources including EC2/GCE instances, LDAP and much more. But it also allows you to generate dynamic secrets including, but not limited to, SSH certs, IAM users, AD users, sql users and much more. It's Hashicorps most mature tool and should be in everyone's toolchain.

outworlder 7y ago
Vault requires a non-trivial amount of setup and configuration, and then you need to configure SSHD to use it. You also have to maintain the vault server, HA, yada yada. And educate people on how to use it. Vault truly shines in its ability to provide ephemeral credentials. It can even "package" arbitrary secrets for you, and ensure that they are only ever retrieved once. It's a fantastic piece of software, one that more companies should be using, but it is not a push button deployment. It is borderline impossible to implement in many 'traditional' organizations.
- eeZah7Ux 7y ago
  > truly shines in its ability to provide ephemeral credentials. It can even "package" arbitrary secrets for you Those credentials should not be generated, stored or provided centrally in the first place.

Ayesh 7y ago

- Use Ed25519 keys. They are now supported in pretty much every server out there (all recent OpenSSH versions, GitHub, GitLab, etc). RSA 2048 keys are unbreakable for the foreseeable future, and using 4096 bit keys are just being paranoid with no gain. You can fit 4x Ed25519 keys in a tweet. - Setup and use ssh-agent. They make the life so easy. - You can use Yubikey and pretty much every other solution out there with with OpenSSH now.

mason55 7y ago
> They are now supported in pretty much every server out there Sadly whatever library Spring Integration uses does not support them yet, as I found out the hard way. I just tried to search for it and couldn't find the reference but just dealt with it a month or two ago.
ddevault 7y ago
>Setup and use ssh-agent. They make the life so easy. My #1 ssh usability tip: put this into ~/.ssh/config: AddKeysToAgent yes It'll automatically add keys to your agent the first time you use them during a session, so you don't need a separate step for adding keys every time you log in.
- res0nat0r 7y ago
  I've not noticed this option before, so will this replace keychain? I install this on every new laptop I use so I essentially only have to enter my ssh-agent password for the key once per reboot, and every bash shell hooks into it automatically. https://www.funtoo.org/Keychain https://www.funtoo.org/Keychain
- Lammy 7y ago
  Be careful; there are plenty of ways ssh-agent can bite you too. Here are a few: http://rabexc.org/posts/pitfalls-of-ssh-agents http://rabexc.org/posts/pitfalls-of-ssh-agents
  - ddevault 7y ago

jontro 7y ago

How is ssh certificates the next level up? Feels like the added complexity of using this is might be worthwhile for larger orgs but maybe not smaller shops

kempbellt 7y ago
I second this perspective. It might be useful for a larger org, but for my use case and that of my team, I don't see much value-add, and the added complexity would likely make managing this system more difficult than it is worth. I've also never had an issue with key management, like the article suggests is common. Maybe I'm not the target audience though.
yjftsjthsd-h 7y ago
Is one person a small enough shop? ;) I found it useful for personal systems because it meant that I didn't need to verify keys (user keys and host keys) for every server/laptop combination.
old-gregg 7y ago
Disclaimer: I work at Gravitational, one of our engineers is the author of the article. The added complexity is indeed significant, that's why we built Teleport [1] which is much, much simpler open source alternative to OpenSSH for such use case (the blog post does not mention Teleport because it was written for OpenSSH users). But generally, certificates have a number of advantages: they expire quickly (you can have one-shot certs), they contain your full name, email address, or whatever you wish via metadata, this allows to implement RBAC on top of SSH. They address "trust on first use" issue, and they're easy to synchronize with other protocols, i.e. you can have the same CA issue certificates for both Kubernetes and SSH (with identical RBAC rules). When implemented properly, they're actually easier to use. The whole world uses certificates to do online banking and shopping, there's no reason not to have the same seamless experience for SSH. [1] https://gravitational.com/teleport https://gravitational.com/teleport
- Gaelan 7y ago
  btw, huge props for not mentioning teleport in that blog post. So many companies have blog posts that masquerade as tutorials for "how to do X" but then change halfway through to "OOOORRRR you can not do all that complicated stuff and just buy our product instead," which feels pretty scummy. Thanks for taking the high road.

nelsonmarcos 7y ago

It's a long article written on April 1st. Please, don't say it's an april's fool joke.

jonluca 7y ago

The title of this doesn't properly match the contents. This is basically a how to set up an enterprise SSH environment using certificates. This doesn't tell you anything about "How to SSH Properly".

Aachen 7y ago
I almost thought so too and was itching to hit back and post an annoyed comment, but there are actually two more sections: using 2FA (straightforward TOTP branded by our favorite not-evil corp though I've not no idea what they've got to do with it) and setting up a jump host / bastion host. So maybe the title shouldn't be "I like ssh certs" but "ssh security for organisations", but your point still stands.
- mynameisvlad 7y ago
  > though I've not no idea what they've got to do with it) It's the name of the TOTP PAM module they use, which was created by Google: https://github.com/google/google-authenticator-libpam https://github.com/google/google-authenticator-libpam
webvictim 7y ago
Author here - thanks for the feedback. As another reply points out, I did try to also cover the use of a bastion host along with one form of 2-factor authentication. I'm considering doing a future post on how to set up U2F for SSH with hardware devices (like a Yubikey) as well. I'm curious if you have anything else you'd like to see on this topic.
jlgaddis 7y ago
The company that authored this "How to SSH Properly" blog post sells a product that will enable you to use SSH "properly" (according to their definition). That's not a coincidence, of course. It's purely sales and marketing. To be clear, the ultimate goal of this blog post is simply to get you or your employer to give them some of your money -- that's it. (To be clear, giving an external entity full control over authentication and authorization of your critical servers is definitely not doing "SSH properly", IMO!)

egwor 7y ago

One thing I find useful is to config default usernames. You can do this by adding a line like below into ~/.ssh/config Host alias domain.com HostName domain.com User aUser This allows you to just write `ssh alias` and it will convert it to ssh aUser@domain.com.

megavolcano 7y ago
you can also specify which private key to present using IdentifyFile /path/to/key/file when I have different hosts that require different keys to access, this is very useful so I don't have to specify them every time and agent forwarding is useful as well, for example, I can push to github from a vm using my laptop ssh key and I don't have to generate an additional key for the vm ForwardAgent true with the new Windows terminal, I setup different tabs to be SSH to different machines based on my .ssh/config file { "guid": "{1c9b268e-7606-4a07-b097-d8bc62fb5207}", "hidden": false, "name": "ubuntu.localdomain", "commandline": "ssh.exe me@ubuntu", "colorScheme": "Elementary" }, { "guid": "{b7161acc-8235-4283-9a85-e93df3d09125}", "hidden": false, "name": "db.localdomain", "commandline": "ssh.exe me@db", "colorScheme": "Ubuntu" }, Windows Terminal with a very nicely define ssh config is a killer combo.
- inetknght 7y ago
  If you have more than one private key in your agent, then SSH will try each one sequentially. That can lead to mistakenly getting banned by monitoring software for what can appear indistinguishable to be an authentication attack. Not to mention the wrongness of presenting "any and all keys" to any host you connect to. If you have multiple keys in your agent, you _really_ should manually configure what key to present using `-i` or `IdentityFile`. But you should also use `IdentitiesOnly` too.
  - megavolcano 7y ago
    thanks for the tips!

perlgeek 7y ago

Somewhat off-topic, but I've noticed that ssh-keygen is much faster at generating key pairs that GnuPG, due to the latter wanting more "proper" entropy. Does that mean that GnuPG is overly paranoid? Or ssh-keygen's keys potentially insecure? (I really hope not). There must be some good explanation for this huge speed difference.

Aachen 7y ago
The difference must be urandom vs random. You'll find a lot of words spent arguing the misinformed vs the enlightened if you look for those terms, on HN or elsewhere. I'm in the urandom camp and think that's the only thing that makes sense on the condition that it blocks/errors until seeded properly once, which means so much as that gpg is misinformedly paranoid in my opinion and that if ssh keygen had been insecure it would have been fixed long ago, but do read up and form your own opinion.
- emmelaich 7y ago
  I don't think it's urandom v random. It really just waits longer.
vbezhenar 7y ago
GnuPG is overly paranoid. Not a bad thing for something that performed very rarely.

et1337 7y ago

There are a number of turnkey solutions for this. One is ScaleFT[1]. My company is another: StrongDM[2]. Our product also keeps audit logs for compliance. [1] https://www.scaleft.com/ https://www.scaleft.com/ [2] https://www.strongdm.com/ https://www.strongdm.com/

old-gregg 7y ago
https://gravitational.com/teleport https://gravitational.com/teleport Free and open source, made by the authors of the article.
- jlgaddis 7y ago
  > ... made by the authors of the article. And, just so everyone is clear, your employer.

ansible 7y ago

With regards to using ssh-agent, this is what I have in my .bash_profile: export SSH_ENV=$HOME/.ssh/environment_${HOSTNAME} function start_agent { echo -n "Initialising new SSH agent... " touch ${SSH_ENV} chmod 600 ${SSH_ENV} /usr/bin/ssh-agent -s | sed 's/^echo/#echo/' > ${SSH_ENV} echo "succeeded." . ${SSH_ENV} > /dev/null } # Don't use Gnome keyring for SSH if [[ $SSH_AUTH_SOCK =~ /run/user ]]; then unset SSH_AUTH_SOCK unset SSH_AGENT_PID fi # Source SSH settings, if applicable ssh-add -l > /dev/null 2>&1 if [ $? -eq 0 -o $? -eq 1 ]; then echo "SSH Agent found." else echo -n "Looking for SSH Agent... " if [ -f "${SSH_ENV}" ]; then . ${SSH_ENV} > /dev/null ssh-add -l > /dev/null 2>&1 if [ $? -eq 0 -o $? -eq 1 ]; then echo "found pid: ${SSH_AGENT_PID}." else echo "not found." start_agent; fi else start_agent; fi ssh-add -l fi This starts up a new ssh-agent if it can't find one, but uses the same agent if it is already running on this machine (I remote into a bunch of systems). So even if I use a bunch of screen sessions, or remote in to a machine multiple times, it will always try to use the same agent.

jpeeler 7y ago
Have you considered using https://github.com/funtoo/keychain https://github.com/funtoo/keychain? Edit: I see it's been mentioned elsewhere here.
- ansible 7y ago
  Well, that's interesting. And a whole bunch of code. From a quick skim, it is basically solving the same problem in a different style. I didn't want or need to wrap ssh-agent. I'm fine with using it and ssh-add by themselves. I just wanted to automatically find and re-use an existing agent rather than spawning a new one.
Spivak 7y ago
Why not use the GNOME keyring as your SSH agent? It's a full agent that you don't really need to think about.

prussian 7y ago

Host certificates are fine, but user certs are pointless landmines when features like AuthorizedKeysCommand exist. why worry about revocations, expiries and all that noise when you can simply have some script just pull the allowed keys? sure maybe login is a bit slower, but does that matter much?

webvictim 7y ago
Don't get me wrong, using AuthorizedKeysCommand is a lot better than having a static ~/.ssh/authorized_keys file on a server, but it isn't anything like as powerful as using user certificates. Certificates can do a lot more than authorized keys can, like enforcing the use of specific principals, commands and options and embedding that information into the file itself without needing to modify each server's SSH configuration. They're also self-contained and will still work in situations where some external service providing a list of keys goes down. I've been on the rough side of a huge LDAP outage which prevented necessary access to the infrastructure to fix it, and it was a horrible experience. There's none of that problem with certificates as long as you make sure you have one which is currently valid. I'm also generally of the opinion that it's safer to enforce the use of authentication which expires by default rather than relying on some external process to do that for you.
- Spivak 7y ago
  But AuthorizedKeysCommand and certs are at least equally powerful because they're both ways of specifying the content of the same authorized_keys file.
  - webvictim 7y ago
    It's something of an implementation detail - you don't generally specify the usage of certs on a user-by-user level, you do it by trusting the entire CA in /etc/ssh/sshd_config and then using the signed content of the individual cert (expiry date, principals etc) to dictate whether someone should be allowed to get access or not. Look at it in terms of building in a decision at compile-time rather than at runtime. With AuthorizedKeysCommand, you're running something just-in-time on an SSH login to determine whether something should be allowed to proceed. With a CA and a process for issuing certificates, that decision is made at the time the cert is issued and then the cert is good for the duration it's issued for. It's entirely self-contained as sshd itself is making the decision about whether the cert is within its validity period or not. It's obviously a decision that people can make based on their own infrastructure, but my opinion is that the compile-time model is more reliable as it's a fully self-contained system and doesn't rely on an entire fleet of servers being able to connect back to an external service at runtime to determine whether you should be allowed to log in. That sort of thing invariably comes back to bite you when you really _need_ to be able to log in and you can't because the external service is down.

i9wsu 7y ago

Just update and use Use Ed25519 keys resource of how to: http://wolf-tm.com http://wolf-tm.com

293984j29384 7y ago

Slightly off topic but the author recommends expiry periods for certificates. Other than rotating a key because of compromise, this seems unnecessary. We've finally decided that passwords should no longer be mandatory rotated, why should certificates?

stephenr 7y ago
How do you revoke access if the certificate has no expiry?
- hvidgaard 7y ago
  If it's for SSH, you should be able to blacklist that particular certificate. You want to do this to revoke access anyway. Only downside is an every growing black list file, but that is probably a problem you need centuries of use to actual need to solve.
- 293984j29384 7y ago
  Forgive my ignorance but what's the correlation between expiry and revoking? I'd assume there are several reasons you'd want to revoke a certificate such as it's been compromised. How would it's expiry date help that?
  - webvictim 7y ago
    Author here. My take on this is that fail-closed is a vastly better security model than fail-open. I am genuinely surprised that OpenSSH actually issues certificates with no expiry date as a default. If you have a certificate which expires within a day by default then an unsuccessful revocation is no longer a huge cause of stress. In the worst case, you lock down access to your bastions and disallow the issue of any future certificates for that user. Within a day, any potential threat from that certificate has vanished. This seems preferable to having a mandatory requirement of an up-to-date revocation database which is synced everywhere.
    - 293984j29384 7y ago

kitotik 7y ago

It seems weird there wasn’t more emphasis on the implications of running your own CA. I can imagine lots of people coming across this article and doing a quick copy pasta not realizing what just happened.

m463 7y ago

I like that with macos, you can create keys with a passphrase, and then that passphrase can be provided by the keychain which is unlocked at login ssh-keygen -t ed25519 -b 521 -f .ssh/id_foo_ed25519 cat id_foo_ed25519.pub | ssh foo tee -a .ssh/authorized_keys ssh-add -K ~/.ssh/id_foo_ed25519 along with: Host foo HostName foo.bar User me StrictHostKeyChecking yes IdentitiesOnly yes IdentityFile ~/.ssh/id_foo_ed25519

LilBytes 7y ago
Yeah this is a huge time saver.
terrywang 7y ago
In addition to the traditional | way of adding public key to target hosts, ssh-copy-id [1] is a wrapper script (shipped by most distros, for macOS it is a separate formula) which can be used to batch add a public key target hosts (love to use it prep for Ansible). Put the target hosts in a file e.g. ssh_ducks while read -r i; do ssh-copy-id -i /path/to/ops_ed25519.pub <user>@$i -f; done <ssh_ducks storm (ssh like a boss), it works well when managing (~/.ssh/config) in CLI / scripts but it has an annoying bug [2] (not yet fixed) that will silently convert ssh_config Host keywords to lowercase (luckily ONLY arguments are case sensitive, NOT keywords, otherwise it'll be easier to detect), annoying. Otherwise, for small to medium scale SSH access, I prefer to use jumpserver (open source 4A bastion / remote access gateway) behind VPN (WireGuard of course, for now, self-hosted or Tailscale), a lot easier to deploy in comparison to Teleport, user-friendly UI for average users. [1]: https://github.com/openssh/openssh-portable/blob/master/contrib/ssh-copy-id https://github.com/openssh/openssh-portable/blob/master/cont... [2]: https://github.com/emre/storm/issues/157 https://github.com/emre/storm/issues/157
- ryall 7y ago
  Ansible can prep a host for Ansible...
deleted 7y ago
[deleted]

anilakar 7y ago

I've found it a good practice to add a YYYYMMDD serial number: -z 20200402 That way, keys generated within a certain time interval can be revoked with a simple serial number check.

fractalf 7y ago

Is there a way to set up 2 ways auth with andOTP instead of the google auth?

ZoomZoomZoom 7y ago

More fitting to the title would be this page: https://infosec.mozilla.org/guidelines/openssh https://infosec.mozilla.org/guidelines/openssh It's a nice practical guideline with all the basics covered.

rauhl 7y ago

On a related topic, what is the current best practice for avoiding TOFU for cloud providers? Sure, SSH certificates provide that for normal users, but how does the SSH CA know — ideally in an automated fashion — what that public key is? An admin can SSH into the machine to check it, but then he has to trust the key he sees. He could access it via the console, but that is a manual process. All the other approaches I can think of involve generating at least one key outside the machine which uses it, which seems like a poor solution.

wraithm112 7y ago
At least on AWS, you can parse the System Console Output. On the first boot, the box dumps its pub keys there. We have a tool that parses the output: https://github.com/bitnomial/aws-ec2-knownhosts https://github.com/bitnomial/aws-ec2-knownhosts Right now, this is pretty specific to our use of it, specifically our use with Terraform EC2 instances. We'd happily suggest changes to make it more generic. But you can see the parsing logic there.

GRMrGecko 7y ago

This is perfect timing as I just wrote this tool to help sign ssh host keys and user keys. https://github.com/GRMrGecko/ssh-cert https://github.com/GRMrGecko/ssh-cert