8 ms·

> being able to pop a shell on a running app was table-stakes for the platform. Tangent: that's debatable IMO. In my company's current AWS infrastructure, ther

> being able to pop a shell on a running app was table-stakes for the platform. Tangent: that's debatable IMO. In my company's current AWS infrastructure, there's no shell access to either the production containers or the host machines. I did write a script to create an ephemeral container that lets me (and future staff) run a shell inside the production network. And the thing I usually do in that shell is run psql; I suppose that's not ideal for auditability. But still, I can't poke around in the live containers or the host machines; in theory they could be distroless, with no shell at all. I'm trying to take immutable infrastructure to the max here; this seems like a good thing for security. It does mean that for debugging production problems, I can only go by what I find in logs and the database. But that has been acceptable so far. Edit: Given tptacek's security background, I was surprised that he considered production shell access essential for a new app platform.

remram 5y ago
It's useful when developing. Not being able to shell into the production system is fine if you can shell into a staging/development system. If you are running Docker containers and you can shell into local containers, that is usually "close enough" that you can do useful troubleshooting. But fly.io (and CloudFlare workers, etc) are different enough from off-the-shelf containers that it is very important to be able to poke at containers when they break, even if they are not the actual production containers.
tptacek 5y ago
It's funny you bring this up, because I had the same thought when I was first implementing SSH (the initial implementation relied on native WireGuard, and just plugged a client certificate into your running SSH agent). I thought people might not want to enable SSH access, and so I made the provisioning of the root SSH certificate optional: you have to run `flyctl ssh establish` to tell us to set up a root cert for your organization. It's turning out to have been a misfeature that confuses people more than it helps anyone, and we may get to a place soon where we just automatically provision a root cert for new organizations.
- mwcampbell 5y ago
  So do you think I'm being too extreme on this? Or did you just implement SSH because customers want it?
  - tptacek 5y ago
    I think it's sensible to run an application fleet without SSH access, but it's tough for a hosting provider that has to support lots of different application fleets to not offer a way to get a shell. Our authorization systems are about to get sharply more interesting as we roll out Macaroon-style tokens this quarter, so I'm optimistic we'll get to a place where we make both styles of application owners happy. I was more on your side when I wrote the feature, and I'm less on your side now. Also, I SSH into instances to debug things all the time. :)
    - 10000truths 5y ago
      I think Linode’s approach to this is best: they offer a “virtual console” that basically consists of an SSH gateway that pipes to your VPS’s virtual serial port.
      - deleted 5y ago
        [deleted]
    - ignoramous 5y ago
      Might be noob question (ahead of time, even), why/when would one prefer authz/acl with Macaroons over Zanzibar? All I could gather from summaries is that, for Cloud, Macaroons are better suited since 'decentralised'; whereas, Zanzibar appears to favour consistency with centralised architecture capable of expressing relationship graphs and permission inheritance.
FujiApple 5y ago
> In my company's current AWS infrastructure, there's no shell access to either the production containers or the host machines. I did write a script to create an ephemeral container that lets me (and future staff) run a shell inside the production network. I'm a fan of this approach, it is hard improve upon the security of a server that doesn't exist. I recently setup an AWS serverless (mainly ECS Fargate) stack for a project and took this approach of spinning up an ephemeral EC2 server as part of a "breakglass" runbook for the rare cases where such access is needed. This, combined with Tailscale userspace networking [0] (so the "breakglass" servers can run on a private subnet and are never exposed to the internet), Pulumi [1] (for managing the lifecycle of the "breakglass" instances) and Yubikey based MFA for short lived credentials [2] (required to spin up the server via Pulumi), I found to work well. This approach is also useful for ensuring that whenever a "breakglass" server is started it is using the latest AMI version (Pulumi's `aws.ec2.get_ami_output()` is useful for this) and runs the usual security updates on startup. The ssh keypair for the server can also be created (and later destroyed) on the fly so there is no need to manage any long term ssh credentials. [0] https://tailscale.com/kb/1113/aws-lambda/ https://tailscale.com/kb/1113/aws-lambda/ [1] https://www.pulumi.com/ https://www.pulumi.com/ [2] https://aws.amazon.com/blogs/security/enhance-programmatic-access-for-iam-users-using-yubikey-for-multi-factor-authentication/ https://aws.amazon.com/blogs/security/enhance-programmatic-a...

remram 5y ago

It's useful when developing. Not being able to shell into the production system is fine if you can shell into a staging/development system. If you are running Docker containers and you can shell into local containers, that is usually "close enough" that you can do useful troubleshooting. But fly.io (and CloudFlare workers, etc) are different enough from off-the-shelf containers that it is very important to be able to poke at containers when they break, even if they are not the actual production containers.

tptacek 5y ago

It's funny you bring this up, because I had the same thought when I was first implementing SSH (the initial implementation relied on native WireGuard, and just plugged a client certificate into your running SSH agent). I thought people might not want to enable SSH access, and so I made the provisioning of the root SSH certificate optional: you have to run `flyctl ssh establish` to tell us to set up a root cert for your organization. It's turning out to have been a misfeature that confuses people more than it helps anyone, and we may get to a place soon where we just automatically provision a root cert for new organizations.

mwcampbell 5y ago
So do you think I'm being too extreme on this? Or did you just implement SSH because customers want it?
- tptacek 5y ago
  I think it's sensible to run an application fleet without SSH access, but it's tough for a hosting provider that has to support lots of different application fleets to not offer a way to get a shell. Our authorization systems are about to get sharply more interesting as we roll out Macaroon-style tokens this quarter, so I'm optimistic we'll get to a place where we make both styles of application owners happy. I was more on your side when I wrote the feature, and I'm less on your side now. Also, I SSH into instances to debug things all the time. :)
  - 10000truths 5y ago
    I think Linode’s approach to this is best: they offer a “virtual console” that basically consists of an SSH gateway that pipes to your VPS’s virtual serial port.
    - deleted 5y ago
      [deleted]
  - ignoramous 5y ago
    Might be noob question (ahead of time, even), why/when would one prefer authz/acl with Macaroons over Zanzibar? All I could gather from summaries is that, for Cloud, Macaroons are better suited since 'decentralised'; whereas, Zanzibar appears to favour consistency with centralised architecture capable of expressing relationship graphs and permission inheritance.

FujiApple 5y ago

> In my company's current AWS infrastructure, there's no shell access to either the production containers or the host machines. I did write a script to create an ephemeral container that lets me (and future staff) run a shell inside the production network. I'm a fan of this approach, it is hard improve upon the security of a server that doesn't exist. I recently setup an AWS serverless (mainly ECS Fargate) stack for a project and took this approach of spinning up an ephemeral EC2 server as part of a "breakglass" runbook for the rare cases where such access is needed. This, combined with Tailscale userspace networking [0] (so the "breakglass" servers can run on a private subnet and are never exposed to the internet), Pulumi [1] (for managing the lifecycle of the "breakglass" instances) and Yubikey based MFA for short lived credentials [2] (required to spin up the server via Pulumi), I found to work well. This approach is also useful for ensuring that whenever a "breakglass" server is started it is using the latest AMI version (Pulumi's `aws.ec2.get_ami_output()` is useful for this) and runs the usual security updates on startup. The ssh keypair for the server can also be created (and later destroyed) on the fly so there is no need to manage any long term ssh credentials. [0] https://tailscale.com/kb/1113/aws-lambda/ https://tailscale.com/kb/1113/aws-lambda/ [1] https://www.pulumi.com/ https://www.pulumi.com/ [2] https://aws.amazon.com/blogs/security/enhance-programmatic-access-for-iam-users-using-yubikey-for-multi-factor-authentication/ https://aws.amazon.com/blogs/security/enhance-programmatic-a...