8 ms·

> But the problem with static ssh keys is that if they are stolen, it’s undetectable. It's like, super detectable. You have endpoint logs for the file access,

> But the problem with static ssh keys is that if they are stolen, it’s undetectable. It's like, super detectable. You have endpoint logs for the file access, you have network logs, you have sshd logs (which contain the public key and the IP), etc. > if you have your ssh private key on several machines, you have to remember to copy it to all those places Your ssh key should never leave a host. That should be a policy and you should write rules to detect when that policy is being violated (check for processes accessing the file). If you need access from N computers you should be generating N keys. The reason rotation isn't recommended is because it leads to bad practices (people just add a '1' to their password), it's a hassle, and it can never be fast enough to meaningfully impact an attacker - once they have SSH keys it's likely they can gain persistence and C2 before your rotation takes place. Not because people reuse their passwords in multiple places. Setting up a CA for SSH is definitely a really good practice but I think that most companies would find it far simpler to just enforce 2FA for SSH access. Still, I'd really like to see an article about how you set that up, especially if it targets smaller enterprise customers. For others who might be interested, here's bless from Netflix: https://github.com/Netflix/bless https://github.com/Netflix/bless edit: Oh, and just to be clear, 2FA for your SSH is not a silver bullet - even a yubikey. But it's a cheap, scalable, near-zero overhead way to protect against an attacker who's got access to your key (but not one who has access to your system, assuming an active session!).

amenghra 6y ago
> Still, I'd really like to see an article about how you set that up, especially if it targets smaller enterprise customers. https://github.com/square/sharkey https://github.com/square/sharkey
- staticassertion 6y ago
  Oh I mean I know how to do it, I just mean I'd like to see companies talking about how they do it, and I'd really really like to see someone make it drop-dead simple. It's valuable to hear how companies are protecting their infrastructure if they're going to break down how they accomplished it, how they continue to maintain it, etc.
  - shandor 6y ago
    OpenSSH 8.4 added support for FIDO2. It's literally the same commands (with some different parameters) to take into use as a normal keypair. And FIDO2 tokens are becoming quite affordable. To me 2FA with SSH became a solved problem with that.
    - ldng 6y ago
      Could you point to a relevant article explaining that setup please?
      - shandor 6y ago
        This one had quite a lot of background, but also the needed commands: https://www.stavros.io/posts/u2f-fido2-with-ssh/ https://www.stavros.io/posts/u2f-fido2-with-ssh/ With a properly set-up token, basically you only need 2 commands: ssh-keygen -t ed25519-sk -O resident -f ~/.ssh/id_mykey_sk ssh-add -K I used a Yubikey and needed also to install 'ykman' to set a PIN for my token, otherwise ssh-add kept failing. Dunno if I omitted something for a proper setup for my token initially, but I don't think that was a problem with OpenSSH in particular. Apart from the small headache with the PIN, the whole thing was almost magical in its simplicity.
        loulouxiv 6y ago
        The token must be pin protected to be eligible to resident credentials. I think it is one of the significant differences between FIDO1 and FIDO2. I also think that you can use ed25519-sk with a non pin-protected, but then you won't be able to authenticate if you can't access to the generated key file (if it was deleted or on another machine)
        shandor 6y ago
        Thanks for the info. Makes complete sense to require a PIN with resident keys, I'm glad. Now I'll need to write the article myself, with this information included :)
sneak 6y ago
Forget 2FA - yubikeys can be actual full key generation/storage for a GPG key in smartcard mode, and then via gpg-agent those can be used for ssh. (gpg-agent replaces ssh-agent.) My ssh keys never leave the yubikey. I have a different dedicated yubikey in each computer, with its own unique key, and a stolen key is useless without its unlock PIN.
- devwastaken 6y ago
  My concern with physical keys is what happens if it stops working?
  - sneak 6y ago
    I have one key permanently in a usb port of each computer, and one on my keychain as a backup (which is also used for U2F and has lightning so works with my phone for 2FA), so five in total, each with their own RSA key for ssh.
  - danielheath 6y ago
    You need two; one stowed safely and only retrieved to enroll a new service, and one to use day-to-day.
    - sneak 6y ago
      enroll a new service? we are talking about gpg smartcard usage, for ssh - not u2f 2fa. you don't need the physical key to "enroll", you just keep a copy of its pubkey.
      - tmottabr 6y ago
        the idea is to generate the key in the device, so it never leave the device, therefore you cannot store it somewhere else like another device. So, the ideia is for you to have two devices, each with its own key, the first device you use daily and the second you use store in a safe location. if your first device in daily use stop working or is lost you use the second device you have stored to login to your systems to remove the keys from the lost device and add the keys for a new device that replace it and then store the second device back in a safe location.
chmod775 6y ago
No. You can't assume your logs are trustworthy if you are compromised. You have to assume an attacker will remove any traces of themselves from those logs or will stop them from being written in the first place. It's pretty standard practice to do this post-exploitation. Further this requires you to regularly check your log files for suspicious activity, which is way more work than just rotating your ssh keys - which can be easily automated. Running a script that will automatically rotate your ssh keys on all servers in your .ssh/known_hosts is trivial. Also, rotating your ssh keys is something that has a chance to prevent intrusion, whereas if you see something in your logs it's already too late.
- staticassertion 6y ago
  You can assume a lot of logs are not compromised. For example, logs from your network infrastructure. In this case you could trust sshd logs (so long as you ship them off quickly) and network logs. As for endpoint logs, yes, a privileged attacker could disrupt them. But honestly, even with regards to your endpoint, attackers often don't disable logging - though I do see it, for sure. For "blessed" logging like Windows Event Log you'll have an even harder time - they do take measures to protect the files on disk, even against privileged attackers, and supported methods of deleting the event log actually themselves generate a "Someone deleted the event log" event, which I would highly recommend you watch out for :) The simplest advice for dealing with this is to ship logs off of the device ASAP and to make sure that disrupting the service requires privileges.
  - chmod775 6y ago
    > For "blessed" logging like Windows Event Log you'll have an even harder time [...] even against privileged attackers Trying to prevent a determined privileged attacker from doing something is an exercise in futility, since it's impossible in every sense of the word. If they have total control over a system, they can do whatever they want, even if you put up a bunch of stopgaps. Most post-exploitation frameworks (prominent example: DanderSpritz) have modules to remove stuff from the windows event logs without leaving traces. It's pretty basic stuff.
mschuster91 6y ago
> You have endpoint logs for the file access No endpoint security system I'm aware of being used outside of core banking/telco/government system logs all file accesses. It would crash instantly for one single build of your average NodeJS application.
- goguy 6y ago
  I've worked at places that use the below or a flavour of to monitor specific files for modification etc. https://www.newnettechnologies.com/nnt-file-integrity-monitoring-fim.html https://www.newnettechnologies.com/nnt-file-integrity-monito...
- staticassertion 6y ago
  File monitoring is particularly slow, though not for any fundamental reason. But you can still monitor lots of files efficiently by just limiting which files you look for, which lots of companies do.
dspillett 6y ago
> Your ssh key should never leave a host. A lot of people think of the SSH keys like PGP keys (where the one private key is your identity) which is not how they are intended to be used (by my understanding which seems to agree with yours). authorised_keys can contain many so you should never need to duplicate a key because you need to access a given account from multiple hosts. > That should be a policy and you should write rules to detect when that policy is being violated (check for processes accessing the file). For static locations, one option I like for this is whitelisting the source address for each key (https://unix.stackexchange.com/questions/353044/ https://unix.stackexchange.com/questions/353044/). You can then monitor abuse of the policy by looking for keys with no source limit, though this isn't something I've ever done, and it means that a stolen private key is more difficult to use from another location. Of course this doesn't work for connecting directly from client hosts that move around (i.e. a user connecting from a laptop that could potentially connect from any address unless you enforce VPN access for sensitive resources).
Uptrenda 6y ago
Posts like this just make me realize how impossible it is to follow every security best practice and stay one step ahead of the latest techniques. You can maybe try tick the most common boxes, but expecting users to be able to police their machines, networks, and other attack surfaces (and still find time to get work done) is unrealistic, IMO. I'm starting to think the common factor here between secure systems and insecure ones is lucking out on having no interested / skilled attackers...
- mscarborough 6y ago
  It's definitely a balancing act, calculating the ROI for improving security vs implementing more immediate customer-focused work. This common problem is what led to rise of the MSSPs https://en.wikipedia.org/wiki/Managed_security_service https://en.wikipedia.org/wiki/Managed_security_service
thegeekbin 6y ago
> Your ssh key should never leave a host. If you need an ssh key for whatever reason from a host (for example, git pulls on a staging machine), you should generate one on that box and narrow it's scope in the machine that will receive it (eg, Gitlab Deployment Keys -- locked in read-only mode, single purpose). > The reason rotation isn't recommended is because it leads to bad practices (people just add a '1' to their password) To some degree. I personally rotate my keys whenever I change my personal/work device (perm. change), or, around every year or so. It's not a hard requirement, but just a personal preference. > it's a hassle ssh-keygen -b 4096 isn't a hassle... and if you do deployment properly it shouldn't be an issue to sync these keys (eg, an AD system holding public key, cloud directories like JumpCloud, etc can all sync the moment it's updated -- even Salt/Ansible/Chef/etc can do it easily, with modules pre-written to sync keys already). In any event, MFA is always a good idea. But, my biggest concern is that someone would leave ssh open to the public... the time it takes to setup an ACL or VPN to connect to the machine is hardly anything these days with the amount of automated tooling to do it... so why aren't people?
- staticassertion 6y ago
  > ssh-keygen -b 4096 isn't a hassle... I was referring to password rotation specifically with those points, not SSH key rotation, because the quote in my post was also in the context of password rotation.
420codebro 6y ago
SSH keys should not be stored on a locally accessible file system. Hardware tokens or you are gonna get them swiped someday.
dotancohen 6y ago
> check for processes accessing the file How? I could check at any point in time if the file is being accessed now, but how could I ensure that it hasn't been accessed in the past? The ext4 access time can be forged by the same process that accesses the file. Should I have a daemon running to check this? If so, then why isn't this already a feature in common Linux distros?
- jlgaddis 6y ago
  > If so, then why isn't this already a feature in common Linux distros? It is. It's called auditd, is quite possibly already installed (albeit probably not configured to do much), and can easily ship its logs off to another host (natively or via syslogd).
  - dotancohen 6y ago
    Thank you. I'm checking out auditd now.

amenghra 6y ago

> Still, I'd really like to see an article about how you set that up, especially if it targets smaller enterprise customers. https://github.com/square/sharkey https://github.com/square/sharkey

staticassertion 6y ago
Oh I mean I know how to do it, I just mean I'd like to see companies talking about how they do it, and I'd really really like to see someone make it drop-dead simple. It's valuable to hear how companies are protecting their infrastructure if they're going to break down how they accomplished it, how they continue to maintain it, etc.
- shandor 6y ago
  OpenSSH 8.4 added support for FIDO2. It's literally the same commands (with some different parameters) to take into use as a normal keypair. And FIDO2 tokens are becoming quite affordable. To me 2FA with SSH became a solved problem with that.
  - ldng 6y ago
    Could you point to a relevant article explaining that setup please?
    - shandor 6y ago
      This one had quite a lot of background, but also the needed commands: https://www.stavros.io/posts/u2f-fido2-with-ssh/ https://www.stavros.io/posts/u2f-fido2-with-ssh/ With a properly set-up token, basically you only need 2 commands: ssh-keygen -t ed25519-sk -O resident -f ~/.ssh/id_mykey_sk ssh-add -K I used a Yubikey and needed also to install 'ykman' to set a PIN for my token, otherwise ssh-add kept failing. Dunno if I omitted something for a proper setup for my token initially, but I don't think that was a problem with OpenSSH in particular. Apart from the small headache with the PIN, the whole thing was almost magical in its simplicity.
      - loulouxiv 6y ago
        The token must be pin protected to be eligible to resident credentials. I think it is one of the significant differences between FIDO1 and FIDO2. I also think that you can use ed25519-sk with a non pin-protected, but then you won't be able to authenticate if you can't access to the generated key file (if it was deleted or on another machine)
        shandor 6y ago
        Thanks for the info. Makes complete sense to require a PIN with resident keys, I'm glad. Now I'll need to write the article myself, with this information included :)

sneak 6y ago

Forget 2FA - yubikeys can be actual full key generation/storage for a GPG key in smartcard mode, and then via gpg-agent those can be used for ssh. (gpg-agent replaces ssh-agent.) My ssh keys never leave the yubikey. I have a different dedicated yubikey in each computer, with its own unique key, and a stolen key is useless without its unlock PIN.

devwastaken 6y ago
My concern with physical keys is what happens if it stops working?
- sneak 6y ago
  I have one key permanently in a usb port of each computer, and one on my keychain as a backup (which is also used for U2F and has lightning so works with my phone for 2FA), so five in total, each with their own RSA key for ssh.
- danielheath 6y ago
  You need two; one stowed safely and only retrieved to enroll a new service, and one to use day-to-day.
  - sneak 6y ago
    enroll a new service? we are talking about gpg smartcard usage, for ssh - not u2f 2fa. you don't need the physical key to "enroll", you just keep a copy of its pubkey.
    - tmottabr 6y ago
      the idea is to generate the key in the device, so it never leave the device, therefore you cannot store it somewhere else like another device. So, the ideia is for you to have two devices, each with its own key, the first device you use daily and the second you use store in a safe location. if your first device in daily use stop working or is lost you use the second device you have stored to login to your systems to remove the keys from the lost device and add the keys for a new device that replace it and then store the second device back in a safe location.

chmod775 6y ago

No. You can't assume your logs are trustworthy if you are compromised. You have to assume an attacker will remove any traces of themselves from those logs or will stop them from being written in the first place. It's pretty standard practice to do this post-exploitation. Further this requires you to regularly check your log files for suspicious activity, which is way more work than just rotating your ssh keys - which can be easily automated. Running a script that will automatically rotate your ssh keys on all servers in your .ssh/known_hosts is trivial. Also, rotating your ssh keys is something that has a chance to prevent intrusion, whereas if you see something in your logs it's already too late.

staticassertion 6y ago
You can assume a lot of logs are not compromised. For example, logs from your network infrastructure. In this case you could trust sshd logs (so long as you ship them off quickly) and network logs. As for endpoint logs, yes, a privileged attacker could disrupt them. But honestly, even with regards to your endpoint, attackers often don't disable logging - though I do see it, for sure. For "blessed" logging like Windows Event Log you'll have an even harder time - they do take measures to protect the files on disk, even against privileged attackers, and supported methods of deleting the event log actually themselves generate a "Someone deleted the event log" event, which I would highly recommend you watch out for :) The simplest advice for dealing with this is to ship logs off of the device ASAP and to make sure that disrupting the service requires privileges.
- chmod775 6y ago
  > For "blessed" logging like Windows Event Log you'll have an even harder time [...] even against privileged attackers Trying to prevent a determined privileged attacker from doing something is an exercise in futility, since it's impossible in every sense of the word. If they have total control over a system, they can do whatever they want, even if you put up a bunch of stopgaps. Most post-exploitation frameworks (prominent example: DanderSpritz) have modules to remove stuff from the windows event logs without leaving traces. It's pretty basic stuff.

mschuster91 6y ago

> You have endpoint logs for the file access No endpoint security system I'm aware of being used outside of core banking/telco/government system logs all file accesses. It would crash instantly for one single build of your average NodeJS application.

goguy 6y ago
I've worked at places that use the below or a flavour of to monitor specific files for modification etc. https://www.newnettechnologies.com/nnt-file-integrity-monitoring-fim.html https://www.newnettechnologies.com/nnt-file-integrity-monito...
staticassertion 6y ago
File monitoring is particularly slow, though not for any fundamental reason. But you can still monitor lots of files efficiently by just limiting which files you look for, which lots of companies do.

dspillett 6y ago

> Your ssh key should never leave a host. A lot of people think of the SSH keys like PGP keys (where the one private key is your identity) which is not how they are intended to be used (by my understanding which seems to agree with yours). authorised_keys can contain many so you should never need to duplicate a key because you need to access a given account from multiple hosts. > That should be a policy and you should write rules to detect when that policy is being violated (check for processes accessing the file). For static locations, one option I like for this is whitelisting the source address for each key (https://unix.stackexchange.com/questions/353044/ https://unix.stackexchange.com/questions/353044/). You can then monitor abuse of the policy by looking for keys with no source limit, though this isn't something I've ever done, and it means that a stolen private key is more difficult to use from another location. Of course this doesn't work for connecting directly from client hosts that move around (i.e. a user connecting from a laptop that could potentially connect from any address unless you enforce VPN access for sensitive resources).

Uptrenda 6y ago

Posts like this just make me realize how impossible it is to follow every security best practice and stay one step ahead of the latest techniques. You can maybe try tick the most common boxes, but expecting users to be able to police their machines, networks, and other attack surfaces (and still find time to get work done) is unrealistic, IMO. I'm starting to think the common factor here between secure systems and insecure ones is lucking out on having no interested / skilled attackers...

mscarborough 6y ago
It's definitely a balancing act, calculating the ROI for improving security vs implementing more immediate customer-focused work. This common problem is what led to rise of the MSSPs https://en.wikipedia.org/wiki/Managed_security_service https://en.wikipedia.org/wiki/Managed_security_service

thegeekbin 6y ago

> Your ssh key should never leave a host. If you need an ssh key for whatever reason from a host (for example, git pulls on a staging machine), you should generate one on that box and narrow it's scope in the machine that will receive it (eg, Gitlab Deployment Keys -- locked in read-only mode, single purpose). > The reason rotation isn't recommended is because it leads to bad practices (people just add a '1' to their password) To some degree. I personally rotate my keys whenever I change my personal/work device (perm. change), or, around every year or so. It's not a hard requirement, but just a personal preference. > it's a hassle ssh-keygen -b 4096 isn't a hassle... and if you do deployment properly it shouldn't be an issue to sync these keys (eg, an AD system holding public key, cloud directories like JumpCloud, etc can all sync the moment it's updated -- even Salt/Ansible/Chef/etc can do it easily, with modules pre-written to sync keys already). In any event, MFA is always a good idea. But, my biggest concern is that someone would leave ssh open to the public... the time it takes to setup an ACL or VPN to connect to the machine is hardly anything these days with the amount of automated tooling to do it... so why aren't people?

staticassertion 6y ago
> ssh-keygen -b 4096 isn't a hassle... I was referring to password rotation specifically with those points, not SSH key rotation, because the quote in my post was also in the context of password rotation.

420codebro 6y ago

SSH keys should not be stored on a locally accessible file system. Hardware tokens or you are gonna get them swiped someday.

dotancohen 6y ago

> check for processes accessing the file How? I could check at any point in time if the file is being accessed now, but how could I ensure that it hasn't been accessed in the past? The ext4 access time can be forged by the same process that accesses the file. Should I have a daemon running to check this? If so, then why isn't this already a feature in common Linux distros?

jlgaddis 6y ago
> If so, then why isn't this already a feature in common Linux distros? It is. It's called auditd, is quite possibly already installed (albeit probably not configured to do much), and can easily ship its logs off to another host (natively or via syslogd).
- dotancohen 6y ago
  Thank you. I'm checking out auditd now.