This is an old, dead horse larger shops solved: - Systems get messy unless you have configuration management that converges on top of ephemeral instances - Bu

This is an old, dead horse larger shops solved: - Systems get messy unless you have configuration management that converges on top of ephemeral instances - Build packages that work the same in all environments if possible - Dev-prod parity: if that means running Docker locally or on some throw-away dev servers, do it - 12factor principles - Every business department has an API and shares the same or similar messaging and storage tech - Change control: have high-available (HA) to where you're never upgrading individual machines and always backup/restore/replacing with fresh nodes - Repeatable, precise, cached, incremental, distributed builds. Even if it means throwing out timestamps in the binaries - Cache build artifacts forever (almost) - Cryptographically sign commits and build artifacts (don't sign hashes because those are weaker) - Canary, sharded deployments - Rarely/never allow changes directly to boxes - Require PRs made through configuration management - Require PRs have to have another developer code review them - Automate the heck out of linting and testing before it's generally allowed to drop in prod - Store secrets either in conf mgmt or in a separate system like Vault - Have SREs who know how things will end up in prod to troubleshoot the complexity and provide CI/CD and {I,P,Db,S}aaS - Reduce the number of duplicated systems to a minimum but not to where it is too awkward or difficult to maintain - Remember that prod, corp, and endpoints (phones and laptops) aren't the same but try to share bits as much as possible while isolating differences - Corp tends to run more heterogeneous than prod. It's okay but don't let it get out-of-hand without having recommended (as opposed to mandated) standards except for security and third-party component review - Eliminate technical debt: don't allow layers of crap or be precious about gross code just because it works - Consider the risks and impact before making changes - Have a backup/DR/BCP plan