5 ms·
As someone who is on a regular rotation - being oncall sucks. But there are definite advantages. Having the team feel direct pain is a great motivator for buil
by sarahj 12y ago
As someone who is on a regular rotation - being oncall sucks. But there are definite advantages.
Having the team feel direct pain is a great motivator for building robust applications - if you know that that quick hack could lead to you getting paged at 3 in the morning you are far more likely to seek out additional solutions(anecdotally speaking) - it means you also have weight within the team to make the right engineering calls.
The team can bond over operations - it sucks being oncall, everyone knows that so everyone tries to make it less sucky, by clearing ops queues before handing off or making ticket messages a bit nicer.
When you start having ticket-free weeks or months it is an awesome feeling, the service works and is robust and your team can spend that time writing new stuff.
Additionally, when you have a small team building/maintaining a service it makes far more sense to rotate the oncall responsibility between them rather than an external engineer.
- Someone1234 12y ago> Having the team feel direct pain is a great motivator for building robust applications But the OP is a dedicated reliability engineer, they don't build the actual applications and couldn't given the size of the company. Essentially you're being punished for other team's generated issues. You getting punished when your own stuff breaks is more a small business issue, not something corporations have.
- sarahj 12y agoThere are definitely corporations where engineers who build the applications take responsibility for maintaining and running the applications - I happen to work for one.
- blackaspen 12y agoThis should be all corporations.
- ryanburk 12y ago> You getting punished when your own stuff breaks is more a small business issue, not something corporations have. why do you think that should be the case? I'd argue quite the opposite having done this in a large corporation with pretty decent success. to be successful, it requires that your monitoring isn't noisy; most issues are repaired in an automated way; then when you get a "page" it is something that someone deeply familiar with the code / service is the best person to react to it. it will drive down the MTTR. and it typically gets the root cause fixed sooner in the code. these folks need to be engineers, but they don't all need to be just the devs, since internet scale services just are noisy and you need to have some randomization buffer. but those engineers need to be dedicated to the service and not a central org, where they aren't going to know how to look at logs, have depth on the intra-service interactions, the latest changes, etc. to me, this is also the best definition of "devops".
- nostrademons 12y agoSRE has veto power over many of the design choices that go into the actual application.