3 ms·

I have two dozen co-located servers to log into. So far I've resisted the idea of setting up a bastion host, because that seems like a way to lock myself out of

I have two dozen co-located servers to log into. So far I've resisted the idea of setting up a bastion host, because that seems like a way to lock myself out of SSH access if that machine dies. I'm also not sure of the rationale for having a bastion host, other than "big companies do it". How do other non-cloud people handle this access vs. security tradeoff? I'm also curious what people's preferred fallback method is for preserving access to machines if you lose access to your yubikey(s), assuming you keep your private SSH key stored on one.

mazon 8y ago
backup key on offline usb. Safely stored in a safe, never to be used except in emergency.
zegl 8y ago
If the yubikey dies, either another person can provision your replacement, or a configuration management system can do so. We’re doing this successfully where I’m working.
- idlewords 8y ago
  I'm not sure I understand what you mean. I run a one-person business and my servers are 110 miles away in Sacramento; what do I do if my yubikey breaks?
  - jetrink 8y ago
    I think the SOP is to have a second Yubikey registered and stored in a safe place and/or a set of one-time codes that you can use instead.
    - idlewords 8y ago
      I actually had no idea SSH had support for one-time passwords, but I see docs for OTPW here: https://www.digitalocean.com/community/tutorials/install-and-use-otpw https://www.digitalocean.com/community/tutorials/install-and... This looks like exactly the answer I needed; thank you!
      - tptacek 8y ago
        It sounds like you're trying to do something like Yubikey OTP or U2F with SSH. Both are doable, but I don't think that's the normal way people use hardware tokens with SSH; what I've seen much more is that people use Y4 keys (which are basically pocket HSMs), generate RSA keypairs with them, and configure SSH to accept the public key on the token. This doesn't require any support on the serverside; as far as the server's concerned, the Y4 is just another RSA key. It does require the more expensive Yubikey, but there are some security advantages to using it.
        idlewords 8y ago
        No, my thinking is along the lines of what you're describing (generate the key on the Y5 Yubikey so that it never leaves it). The discussion of one-time stuff is just as backup for scenarios where I lose the key, which I invariably will, or need to give other people "break glass in case of emergency" access to the servers.
        tialaramex 8y ago
        > something like Yubikey OTP or U2F with SSH. Both are doable I would be interested in how you think U2F with SSH is doable. Maybe with a custom SSH server and client? Have you actually see this done in anger?
        wolf550e 8y ago
        ssh server trusts internal CA, an application that requires U2F login can issue 5 minute certificates signed by the CA, the ssh session does not close when the login cert expires. See https://github.com/Netflix/bless https://github.com/Netflix/bless and https://github.com/gravitational/teleport https://github.com/gravitational/teleport
  - tanderson92 8y ago
    You should already have a full backup of your (private) RSA key anyway on an encrypted USB drive in a desk drawer somewhere, so you use that until you can provision a new Yubikey.
perlgeek 8y ago
> I'm also not sure of the rationale for having a bastion host It's a central place where you can do your logging, which many enterprises must do for compliance reasons.
- pm90 8y ago
  Really? Do any compliance standards specifically require this? Or is the specific control defined in the organization itself that requires this? I've found many compliance standards to be pretty open-ended and function-driven, rather than prescribing specific standards. Client contracts.... they may be a different beast, and often require specific promises by vendors.
  - dvtrn 8y ago
    PCI-DSS comes to mind and it’s segmentation guideline, often met by deploying bastions.
  - perlgeek 8y ago
    I'm pretty sure that at least some of the German financial regulations require that you can audit administrative actions on systems that are relevant for the financial transactions. Now, you could try to implement that on the systems themselves, but how do you establish a mechanism that logs actions of an administrative user, and at the same time cannot be disabled by the same administrative user? So instead you use a shell control box as a bastion, and regular administration of the target hosts don't have root access to the shell control box, so they cannot circumvent logging. I haven't yet any standards that explicitly demand bastions, but it seems to be one of the standard implementations that have proven to pass audits, and so it's a pretty low-risk implementation of the logging requirements.
zamadatix 8y ago
Run 2 bastion hosts?
tptacek 8y ago
Easy: on each host you log into, you enter into authorized_keys your Yubikey RSA pubkey and a standard software SSH pubkey. The private key for the software keypair, you stick on a thumb drive and keep in a drawer or a safe or whatever. Later Sorry, I see below you're thinking about using Yubikeys in addition to keypairs. This obviously doesn't answer that question.
- idlewords 8y ago
  No, your original answer was what I needed! I'm fine using just keypairs where the private half has never left a yubikey.
pm90 8y ago
VPN
toast0 8y ago
It's key to setup two or three bastion hosts with similar diversity to your prod fleet (are you in multiple locations? put a bastion in at least two of those; if not, do you have multiple racks? put a bastion in at least two of those, etc) -- you will have network incidents where some hosts are unreachable, you don't want that to hit all your bastions or you'll be very sad. Bastions are nice, because you can harden them, or put 2fa on them, and not worry as much about the ssh config on the other hosts -- just make sure they don't accept ssh connections from the outside world.
- sbrown12 8y ago
  For anyone interested in trying a bastion host approach, here's a step-by-step guide to spin up a bastion host and enable ssh logging: https://www.strongdm.com/bastion-hosts-with-audit-logging-part-one/ https://www.strongdm.com/bastion-hosts-with-audit-logging-pa...
awinder 8y ago
I’ve only ever seen this handled with VPN, and the answer there is that you’d need to launch another vpn server in the right vpc/subnet if one went down (or have failover setup). It looks like there’s some benefits with bastion hosts around locking down commands & access in a fiber-grained way from vpn, but man what a pain in the ass. I can see a certain perspective that would really care to do things this way but VPN from my cold dead hands.

mazon 8y ago

backup key on offline usb. Safely stored in a safe, never to be used except in emergency.

zegl 8y ago

If the yubikey dies, either another person can provision your replacement, or a configuration management system can do so. We’re doing this successfully where I’m working.

idlewords 8y ago
I'm not sure I understand what you mean. I run a one-person business and my servers are 110 miles away in Sacramento; what do I do if my yubikey breaks?
- jetrink 8y ago
  I think the SOP is to have a second Yubikey registered and stored in a safe place and/or a set of one-time codes that you can use instead.
  - idlewords 8y ago
    I actually had no idea SSH had support for one-time passwords, but I see docs for OTPW here: https://www.digitalocean.com/community/tutorials/install-and-use-otpw https://www.digitalocean.com/community/tutorials/install-and... This looks like exactly the answer I needed; thank you!
    - tptacek 8y ago
      It sounds like you're trying to do something like Yubikey OTP or U2F with SSH. Both are doable, but I don't think that's the normal way people use hardware tokens with SSH; what I've seen much more is that people use Y4 keys (which are basically pocket HSMs), generate RSA keypairs with them, and configure SSH to accept the public key on the token. This doesn't require any support on the serverside; as far as the server's concerned, the Y4 is just another RSA key. It does require the more expensive Yubikey, but there are some security advantages to using it.
      - idlewords 8y ago
        No, my thinking is along the lines of what you're describing (generate the key on the Y5 Yubikey so that it never leaves it). The discussion of one-time stuff is just as backup for scenarios where I lose the key, which I invariably will, or need to give other people "break glass in case of emergency" access to the servers.
      - tialaramex 8y ago
        > something like Yubikey OTP or U2F with SSH. Both are doable I would be interested in how you think U2F with SSH is doable. Maybe with a custom SSH server and client? Have you actually see this done in anger?
        wolf550e 8y ago
        ssh server trusts internal CA, an application that requires U2F login can issue 5 minute certificates signed by the CA, the ssh session does not close when the login cert expires. See https://github.com/Netflix/bless https://github.com/Netflix/bless and https://github.com/gravitational/teleport https://github.com/gravitational/teleport
- tanderson92 8y ago
  You should already have a full backup of your (private) RSA key anyway on an encrypted USB drive in a desk drawer somewhere, so you use that until you can provision a new Yubikey.

perlgeek 8y ago

> I'm also not sure of the rationale for having a bastion host It's a central place where you can do your logging, which many enterprises must do for compliance reasons.

pm90 8y ago
Really? Do any compliance standards specifically require this? Or is the specific control defined in the organization itself that requires this? I've found many compliance standards to be pretty open-ended and function-driven, rather than prescribing specific standards. Client contracts.... they may be a different beast, and often require specific promises by vendors.
- dvtrn 8y ago
  PCI-DSS comes to mind and it’s segmentation guideline, often met by deploying bastions.
- perlgeek 8y ago
  I'm pretty sure that at least some of the German financial regulations require that you can audit administrative actions on systems that are relevant for the financial transactions. Now, you could try to implement that on the systems themselves, but how do you establish a mechanism that logs actions of an administrative user, and at the same time cannot be disabled by the same administrative user? So instead you use a shell control box as a bastion, and regular administration of the target hosts don't have root access to the shell control box, so they cannot circumvent logging. I haven't yet any standards that explicitly demand bastions, but it seems to be one of the standard implementations that have proven to pass audits, and so it's a pretty low-risk implementation of the logging requirements.

zamadatix 8y ago

Run 2 bastion hosts?

tptacek 8y ago

Easy: on each host you log into, you enter into authorized_keys your Yubikey RSA pubkey and a standard software SSH pubkey. The private key for the software keypair, you stick on a thumb drive and keep in a drawer or a safe or whatever. Later Sorry, I see below you're thinking about using Yubikeys in addition to keypairs. This obviously doesn't answer that question.

idlewords 8y ago
No, your original answer was what I needed! I'm fine using just keypairs where the private half has never left a yubikey.

pm90 8y ago

VPN

toast0 8y ago

It's key to setup two or three bastion hosts with similar diversity to your prod fleet (are you in multiple locations? put a bastion in at least two of those; if not, do you have multiple racks? put a bastion in at least two of those, etc) -- you will have network incidents where some hosts are unreachable, you don't want that to hit all your bastions or you'll be very sad. Bastions are nice, because you can harden them, or put 2fa on them, and not worry as much about the ssh config on the other hosts -- just make sure they don't accept ssh connections from the outside world.

sbrown12 8y ago
For anyone interested in trying a bastion host approach, here's a step-by-step guide to spin up a bastion host and enable ssh logging: https://www.strongdm.com/bastion-hosts-with-audit-logging-part-one/ https://www.strongdm.com/bastion-hosts-with-audit-logging-pa...

awinder 8y ago

I’ve only ever seen this handled with VPN, and the answer there is that you’d need to launch another vpn server in the right vpc/subnet if one went down (or have failover setup). It looks like there’s some benefits with bastion hosts around locking down commands & access in a fiber-grained way from vpn, but man what a pain in the ass. I can see a certain perspective that would really care to do things this way but VPN from my cold dead hands.