4 ms·
> Ah, so you worked on a team where the SRE needs were prioritized over the feature requests? Yes, it was an SRE team. All we do is write tools to make operat
by jedberg 5y ago
> Ah, so you worked on a team where the SRE needs were prioritized over the feature requests?
Yes, it was an SRE team. All we do is write tools to make operations better, but more importantly we write tools to make it easier for the dev teams to operate their own systems better. But yes, we had products teams that would push back on our requests because they had product to deliver, and that was fine. We'd either figure out how to do the work for them, or figure out a workaround.
> We do that too. If you could write code to Solve All The Problems then you'd never need to page a human in the first place ;)
Well yes, that's the idea. You can't get to 5 9s of reliability unless it's all automated. :)
> I'll give you a simple example of where you can't write code to solve this sort of thing.
I could easily write code to solve the thing. Step one, double the limit to alleviate immediate customer pain. Step two, page someone to wake up and look at the graphs and figure out what the better medium term solution is to get us through until the morning, including links to said relevant graphs.
You're not gonna have a cost overrun doubling the limit for one night. And if there is a big problem, the person will get paged again a few hours later and have more information to make a better decision.
> But the SRE model asks for operations automation to stick within operations teams, not development teams.
Yes, but I'm not sure I see why that's bad. I don't see any purpose for a dedicated operations team, especially a follow the sun team. If you're Google and you already have offices all around the world, sure, it will be better. But it makes no sense to hire an around the world team just for operations if the rest of your company is in one time zone.
- aij 5y ago> Step one, double the limit to alleviate immediate customer pain. I've been oncall for systems where that would not work. Doubling the memory means you need twice as many machines. Depending on the service, that could require significantly increased network bandwidth. Now the network is saturated and every node needs to queue more data. Now latency and throughput are even worse, and even more requests are being dropped, so you automatically double the limit again...
- jedberg 5y agoWhile that all may be true (but are indications of a poorly architected system), my code would still work. It would double the limit and then page someone. If they logged in and saw all those failures, then they could address those issues. The whole point is that having an around the world follow the sun team would not alleviate those issues or make anything better.
- solatic 5y ago> Yes, it was an SRE team. All we do is write tools to make operations better Go back to my original comment. If you're an SRE team, then basically, you're the operations team for the developers. I'm talking about where developers are responsible for their own operations and there is no team that gets paged instead of them - "most orgs ask a single team to handle on-call around the clock for their service." > Step one, double the limit to alleviate immediate customer pain. Step two, page someone to wake up See, what I read from this is: a) violate my system efficiency KPIs while b) paging someone in the middle of the night anyway. So, lose-lose. > But it makes no sense to hire an around the world team just for operations if the rest of your company is in one time zone. Why does it make any more sense to hire developers remotely who are in your time zone ± three hours? Because that's what most companies are doing these days. If you're already hiring people remotely then you can hire Operations/SRE staff a little further afield and see that as a benefit (follow the sun) rather than a problem. > the rest of your company is in one time zone. For what it's worth, we also hire salespeople around the globe :) Fact of the matter is, it would be so, so nice for Slack to turn off the ability to @channel in the #random channel so that people who are asleep don't get pinged ...
- jedberg 5y agoWe were an SRE team building tools for the development teams who got paged in the middle of the night. The devs writing the services were operating their own services and were getting paged. We would sometimes also get paged for a serious incident so we could coordinate if multiple development teams were involved. Each team managed their own rotation schedules, we just made sure they had one. > See, what I read from this is: a) violate my system efficiency KPI If you're being graded on your system efficiency and not customer satisfaction, well then sure, your way might make sense (but I'd still say it doesn't). But your business will suffer if you optimize for efficiency over customer satisfaction. > Why does it make any more sense to hire developers remotely who are in your time zone ± three hours? Because it's a lot easier to run a team where everyone on the team can meet at the same time. If you have an around the world team, there is no time of day where you can have a meeting and everyone gets to attend during their workday. Realistically you can maybe get away with a nine hour time difference. Any more than that and you have people excluded. Especially if the bulk of your devs are in one or two time zones, your operators will be even more disconnected from them since they will never be able to interact with the devs, and the devs will have no empathy for the operators who they also never interact with. > For what it's worth, we also hire salespeople around the globe Sure, but they aren't writing code that your operators have to run. :) I think we both agree that it's better for devs to get paged for their services instead of operators, and if that's the case, its far better for all the devs to work together and know each other and be in the same or nearly same time zone. A follow the sun model breaks that completely.