3 ms·
I am very opinionated about what SRE and DevOps own vs what devs own; and, I didn't really have anything negative to say about my (admitted) skim of the article
by t-writescode 4y ago
I am very opinionated about what SRE and DevOps own vs what devs own; and, I didn't really have anything negative to say about my (admitted) skim of the article.
As an SWE, I want to and need to know how to provide metrics on my system to be able to understand its health, and I should have good safeguards in place, or at least have communicated with the SREs what I need to provide to them to help them have good safeguards in place, to make sure the application keeps running. If the application goes down, it's my responsibility to make sure it's not my fault (bug in application code) that caused the system to fail.
What I, an SWE, want out of an SRE, though, is infrastructure management. I want to be able to ask them for some queues, and for a redis instance with high availability. I want them to set up the Kafka cluster, the database. I want us to have a conversation about where the secrets are to be stored. I want to be able to ask them what I need to do in code to get a secret and use it. I want them to be able to give me a good template for k8s deployments - or maybe to pair with them, given the docker containers and sidecars I need for a deployment and the projected scaling I'll need and come out with a best-practices set of k8s deployments.
I would be grateful if they monitor the database for some horrible queries; and, use their knowledge of which deployments made that bad query, to file a ticket to the right team so they fix their code or add an index or whatever is necessary.
Infrastructure, be it k8s or nomad, configuring redis, making rabbitmq highly available, configuring and organizing (especially organizing) k8s deployments into something sane and logical, and so many other things related to infrastructure are as specialized of skills as writing high-performance or unusually architected, large systems. I've seen the systems that come up when SWE-on-assignment create infrastructure; and, I've seen the literal years of work SREs have in their backlog to fix it with best practices.
It's similar to front-end developers: it's an entirely different skill set; and, while each person in each tear can stumble around in the other tiers, it's way better if we are all there, working together toward a common goal, and especially focusing in the areas we have each specialized our craft.
addendum: of course there are exceptions; but I think those exceptions are 1 in 100 or 1 in 1000.
- solatic 4y ago> I would be grateful if they monitor the database for some horrible queries; and, use their knowledge of which deployments made that bad query, to file a ticket to the right team so they fix their code or add an index or whatever is necessary. What you're asking for is: a) SRE wrote an alert for slow queries, because slow queries affect all shared infrastructure users b) SRE gets woken up at 2 AM by his alert c) SRE sees Dev wrote a bad query d) SRE files a ticket for Dev e) SRE goes back to sleep f) SRE gets woken up at 2:30 AM by his alert, because the query fired again g) SRE has a restless night h) SRE goes into work, asks Dev to prioritize the ticket to fix the query because sleep i) Dev tells SRE that maybe it'll get into the next sprint, in the meantime the alerts are SRE's problem No thanks.
- t-writescode 4y agoThere are ticketing paradigms that don't require alerting in the middle of the night. All you need to repo and fix a bad query is the query itself and/or an explain of the query itself, and you can work on it during business hours. I'm sorry you've had bad experiences with alert-all-the-things ticketing paradigms. addendum: in fact, you can automate all of this so it just shows up in a team's "known issues" query that they may manage in standup or sprint planning.
- amne 4y agoI still have the proverbial nightmare from hearing at 2 am: "Hello, this is Nagios" in a creepy robotic voice (we had it setup to go through Asterisk with a TTS spelling out what the alert was about. but it always started with that prompt. Free (as in beer) TTS voice packs in 2010 were .. interesting)