4 ms·
It is a major issue. Things like autoscaling have got a lot of bad habits ingrained in people though. You should definitely use SELinux or AppArmor. For any pr
by cthalupa 6y ago
It is a major issue. Things like autoscaling have got a lot of bad habits ingrained in people though.
You should definitely use SELinux or AppArmor. For any production workload I do.
But for any production workload I don't allow SSH over the public internet. I require a VPN connection to a subnet that only has jumphost(s), the jumphost access logs are shipped to an ELK stack, SSH requires both a key + MFA to login to the jumphost, and SSH between production servers is blocked and alarmed on.
- tekknik 6y agoSSH onto the prod servers should also be denied. You say autoscaling so do you have an image? If yes then why do you need prod SSH access anyway? If a box is acting up kill it and let the ASG create a new one.
- cthalupa 6y agoSometimes you have trouble reproducing an issue outside of prod, even with things like tcpreplay or blkreplay. You could just kill off a problematic instance, but then you have trouble knowing why there was a problem to begin with. Grey failures might not be obvious in logs or metrics. The idea that you never ever have to SSH into a production server is a nice ideal, but I've never seen it survive reality unless you just shrug about issues occurring and don't mind not being able to root cause them.