3 ms·
> If the “DevOps” team ships a Postgres RDS instance it will run fine forever, that is until an application starts using it. All of a sudden a cascade of N+1s h
by mjr00 4y ago
> If the “DevOps” team ships a Postgres RDS instance it will run fine forever, that is until an application starts using it. All of a sudden a cascade of N+1s hit, the CPU spikes, and queries grind to a halt. Who is woken up? And why does this always happen at 2 AM? In this scenario, there is nothing for operations personnel to do, yet here they are.
This is definitely a symptom of a broken model and not what I would call devops. IMO the most important tenant of devops is "if you build it, you run it," meaning the appdev team that decided to use Postgres RDS is the one getting woken up at 2am.
It's also, in my experience, one of the best ways to reduce masturbatory engineering decisions and get people to focus on picking boring technology that works. Coding up a serverless application in Rust that's using a CockroachDB backend at a Python/MySQL shop would get a lot of engineers excited, but those people would be less excited knowing they're going to be the ones paged at 2am when this new and exciting architecture falls over in an unfamiliar way (as opposed to Python/MySQL, where a wealth of operational knowledge at the org has already been built up).
Similarly, it naturally reduces architecturally complexity. Younger senior engineers love drawing boxes of queues, multiple microservices, event buses, etc to show off their skill in creating the ultimate engineering fantasy, but once you throw enough late night operational incidents at a senior engineer, suddenly the preferred architecture becomes "an executable running on a box that I can SSH into when things go wrong."
- Spivak 4y ago> This is definitely a symptom of a broken model and not what I would call devops. I'm I the crazy one here? This kind of work is my bread and butter as a "devops" person. PagerDuty fires, I find the bad query, match it up with the most recent PRs to find where the n+1 got introduced and either patch it right there at 2AM with the on-call manager's approval or roll it back. Then we have a postmorterm in the morning with the team. I'm the person positioned the best to do this work because I'm god in my little ops domain, have the most visibility and the biggest toolbox of potential fixes.
- mjr00 4y agoThis sounds like what I mean with building it and running it being devops, yeah -- the fact that you have access to make pull requests or code patches, or even know where the code repository exists in the first place, shows that you're at least somewhat involved in building the application. Contrast this to what was traditionally ops and is now "SRE"; in those roles, applications are usually black boxes where an ops person doesn't know/care how they're developed, because they're responsible for the overall health of the system, which could be managing 100 applications made by 50 development teams.