3 ms·
None of takes away the need for tcpdump, strace, core dumps, etc. Surprisingly much troubleshooting needs to be done in production. I have yet to see views of t
by xorcist 4y ago
None of takes away the need for tcpdump, strace, core dumps, etc. Surprisingly much troubleshooting needs to be done in production. I have yet to see views of the contrary survive contact with the real world.
The idea of expiring ssh keys seems completely reasonable, but it's also a good idea to have enough centralized access control to be able to actually give someone temporary access allocations. Key validity is evaluated locally, against system time.
- phamilton 4y agotcpdump and co do happen occasionally, but less than I had expected. This isn't a small installation. We support around 30M active users and routinely push 20k qps in web traffic. We run on a few thousand vCPUs in EC2. It's a pretty big architectural philosophy shift. All our production workload runs on spot instances, unhealthy hosts come and go quickly enough and our traffic rebalances fast enough that we just don't spend that much time debugging low-level issues. Core dumps get shipped to S3. We do continuous profiling with Google Cloud Profiler. There's a lot of tooling required but once that investment is made things run very well.