tcpdump and co do happen occasionally, but less than I had expected. This isn't a small installation. We support around 30M active users and routinely push 20k

tcpdump and co do happen occasionally, but less than I had expected. This isn't a small installation. We support around 30M active users and routinely push 20k qps in web traffic. We run on a few thousand vCPUs in EC2. It's a pretty big architectural philosophy shift. All our production workload runs on spot instances, unhealthy hosts come and go quickly enough and our traffic rebalances fast enough that we just don't spend that much time debugging low-level issues. Core dumps get shipped to S3. We do continuous profiling with Google Cloud Profiler. There's a lot of tooling required but once that investment is made things run very well.