4 ms·

I've spent the last 7 years using SSH to get onto a prod box a few dozen times. Less than once a month. When I do have to get on the box, we use signed keys tha

I've spent the last 7 years using SSH to get onto a prod box a few dozen times. Less than once a month. When I do have to get on the box, we use signed keys that expire in 24 hours (a manager or SRE is required to sign the key if you need to get on a box). The article is talking about the scaling issues of SSH access via jump boxes or bastions. I'd argue that a better solution to scaling SSH access is to invest in tooling that makes SSH access unnecessary for most of your team. Centralized logs, error reporting, distributed traces, etc. are fairly well solved for most installations. Interactive access to prod (e.g. running DB migrations) requires a little more investment, but tools like ECS Exec make that fairly accessible without requiring SSH access.

vasco 4y ago
DB migrations should definitely not be done interactively and definitely not in an SSH session. Write them beforehand, have them reviewed with the rest of the code, and have your deployment process run them.
- PedroBatista 4y ago
  Agree, but in the real world that works 100% most of the time.
  - ranting-moth 4y ago
    100% most of the time. One of those rare cases when I laugh out loud at a HN comment. I'm definitely stealing this phrase!
- phamilton 4y ago
  Just to clarify our process: DB migrations are part of code review, they run as a script, but we don't have that script run automatically as part of CD. We've been bit by enough migration surprises that we require someone watching and able to interrupt and cancel the migration if needed. But that's the extent of action required. Run this command, Ctrl-C if necessary. Definitely not YOLO'ing in a psql in prod.
  - vasco 4y ago
    That's a really good process that scales forever even if you have a good person for it. Once you want to have more people do it, I find it easier to have hard-coded timeouts that cancel and rollback if the migration will be locked. I've also done auto rollback in these cases.
debarshri 4y ago
I agree with you. But you are viewing it from a web application centric organisation. There are lot of other types of organisations out there, for eg. a company that uses vendor products and now has to give access to servers, databases etc. to the vendor. The scenarios you need access to the downstream resources is mainly for doing ops on them. Other things such as viewing logs can is predictable and can be exposed via some tool to developers.
zie 4y ago
It's still all access control, I don't really see a difference. Yes SSH access could perhaps give you access to more things, with less fine-grained permissions, but it just depends on the replacement(s). The overall point should be, figure out what your needed access control permissions are and then find and use tools to meet those needs. SSH might be the tool, it might not be. Shifting from tail -f <logs> to any web UI doesn't remove the access control for logging, it just shifts it into the fancy web UI. That may or may not be better, it's just different. One could certainly argue the fancy web UI is a better UX, but that's a totally different reason for selecting that tool or not. Access Control happens either way.
xorcist 4y ago
None of takes away the need for tcpdump, strace, core dumps, etc. Surprisingly much troubleshooting needs to be done in production. I have yet to see views of the contrary survive contact with the real world. The idea of expiring ssh keys seems completely reasonable, but it's also a good idea to have enough centralized access control to be able to actually give someone temporary access allocations. Key validity is evaluated locally, against system time.
- phamilton 4y ago
  tcpdump and co do happen occasionally, but less than I had expected. This isn't a small installation. We support around 30M active users and routinely push 20k qps in web traffic. We run on a few thousand vCPUs in EC2. It's a pretty big architectural philosophy shift. All our production workload runs on spot instances, unhealthy hosts come and go quickly enough and our traffic rebalances fast enough that we just don't spend that much time debugging low-level issues. Core dumps get shipped to S3. We do continuous profiling with Google Cloud Profiler. There's a lot of tooling required but once that investment is made things run very well.

vasco 4y ago

DB migrations should definitely not be done interactively and definitely not in an SSH session. Write them beforehand, have them reviewed with the rest of the code, and have your deployment process run them.

PedroBatista 4y ago
Agree, but in the real world that works 100% most of the time.
- ranting-moth 4y ago
  100% most of the time. One of those rare cases when I laugh out loud at a HN comment. I'm definitely stealing this phrase!
phamilton 4y ago
Just to clarify our process: DB migrations are part of code review, they run as a script, but we don't have that script run automatically as part of CD. We've been bit by enough migration surprises that we require someone watching and able to interrupt and cancel the migration if needed. But that's the extent of action required. Run this command, Ctrl-C if necessary. Definitely not YOLO'ing in a psql in prod.
- vasco 4y ago
  That's a really good process that scales forever even if you have a good person for it. Once you want to have more people do it, I find it easier to have hard-coded timeouts that cancel and rollback if the migration will be locked. I've also done auto rollback in these cases.

debarshri 4y ago

I agree with you. But you are viewing it from a web application centric organisation. There are lot of other types of organisations out there, for eg. a company that uses vendor products and now has to give access to servers, databases etc. to the vendor. The scenarios you need access to the downstream resources is mainly for doing ops on them. Other things such as viewing logs can is predictable and can be exposed via some tool to developers.

zie 4y ago

It's still all access control, I don't really see a difference. Yes SSH access could perhaps give you access to more things, with less fine-grained permissions, but it just depends on the replacement(s). The overall point should be, figure out what your needed access control permissions are and then find and use tools to meet those needs. SSH might be the tool, it might not be. Shifting from tail -f <logs> to any web UI doesn't remove the access control for logging, it just shifts it into the fancy web UI. That may or may not be better, it's just different. One could certainly argue the fancy web UI is a better UX, but that's a totally different reason for selecting that tool or not. Access Control happens either way.

xorcist 4y ago

None of takes away the need for tcpdump, strace, core dumps, etc. Surprisingly much troubleshooting needs to be done in production. I have yet to see views of the contrary survive contact with the real world. The idea of expiring ssh keys seems completely reasonable, but it's also a good idea to have enough centralized access control to be able to actually give someone temporary access allocations. Key validity is evaluated locally, against system time.

phamilton 4y ago
tcpdump and co do happen occasionally, but less than I had expected. This isn't a small installation. We support around 30M active users and routinely push 20k qps in web traffic. We run on a few thousand vCPUs in EC2. It's a pretty big architectural philosophy shift. All our production workload runs on spot instances, unhealthy hosts come and go quickly enough and our traffic rebalances fast enough that we just don't spend that much time debugging low-level issues. Core dumps get shipped to S3. We do continuous profiling with Google Cloud Profiler. There's a lot of tooling required but once that investment is made things run very well.