"Extremely strict RPC settings. I’m talking zero retries (or MAYBE one), and a timeout like 3x the p99. We are striving for predictability here, and sprinkling

"Extremely strict RPC settings. I’m talking zero retries (or MAYBE one), and a timeout like 3x the p99. We are striving for predictability here, and sprinkling retries or long timeouts as a quick fix for a flaky downstream service will turn into a week-long investigation and a migraine a year from now. Fix the flaky service!" You want the 80%, right? Might be the network is flaky, might be a service, might be a load balancer. Use timeouts and retries to ignore the flakiness until it eats into your error budgets. "Never give up on local testing. It keeps dev cycle time much shorter than needing to rely on (and fiddle with) CI or remote workspaces. Containerizing the local test environment can make it easier to keep dependencies straight and consistent across machines." Until debugging the snowflakey local testing, or the difference between local dev and CI, ends up eating more time than just using a local<->remote sync tool. There's like a thousand tools now to combine local dev and remote envs, and they will save your organization a lot of time and hassle. "Your git repository is your point-in-time-recoverable source of truth." I call this "suggestion-of-truth". The truth is you never know what state things are actually in, but you hope it's like what's in Git. And there's never enough information kept in Git, like all the versions of all your services used at each deploy, or the state of the entire system, or filesystem snapshots, so recovery often fails. "For infra changes, make plans extremely obvious. This could mean “post the Terraform plan as a comment on the pull request”," Sadly this isn't enough as they might not be using a plan file, and when they apply something unexpected happens. Even if you have a plan file, there's a million other things you should be "making obvious" at deploy time, but people never do. Hard problem to solve. "Use Kubernetes. Assuming you have more than one service and more than one instance, you either need or will need stuff like service discovery, autoscaling, and deployment versioning, and you do not want to manage all that by yourself. Kubernetes gives infra teams scalability superpowers." Yeah, if you pay for it, in money, in expertise, in person-hours. Those things don't just come out of the box. Autoscaling: not at all. Service discovery: you have to set up a DNS controller with an external nameserver, define a naming + namespace convention and stick to it, and use external service definitions for stuff not hosted in K8s. Deployment versioning: you have to define a deployment pattern that stores the versions of everything at deploy time and then link that to your K8s deploys. There's still weeks/months of setting shit up, and then dealing with the complexity of it all and the constant upgrading. Do literally everything you can to abstract away K8s setup + maintenance. Pay anyone to do it all for you. If you're unwilling to pay for it, don't use it. If you do it yourself, even with a managed control plane, it's like building a car from scratch. People on HN sometimes say "oh setting up / maintaining K8s was easy" and I think of the guy who built his own 2-story wooden deck saying it was easy. I wouldn't stand near it.