4 ms·

I also feel like the "on call" thing is oversold a little. I work in operations and I'm on call. Some months I hear nothing, some weeks everything goes wrong,

by sqldba 10y ago

I also feel like the "on call" thing is oversold a little. I work in operations and I'm on call. Some months I hear nothing, some weeks everything goes wrong, but on average I get about one call a month. That's not so bad. The thing is I spend 40 hours a week working out how to do proactive maintenance, monitoring, and alerting on those 400 systems I look after, so that I don't get called outside of hours. I mean that's what my operations job is meant to be, right? Developers can't do that. They're focused on one system or a few systems at most and they're developing 40 hours a week. When the SAN dies and every server in the DC crashes there's no point calling them and asking why they're not on top of it and to try to develop some solution to improve the speed of recovery next time. Me? I built a Jenkins server so next time it happens I press a button and get an immediate dashboard of where to focus my attention. Developers aren't operations. Things may change a little when things go into the deep cloud though and you have services instead of servers running services, seeing as operations may not know what to do with those. On the other hand maybe there is meant to be a handover of some kind so that one person can look after everything instead of 10 developers each having a small stake in their little piece of world.

gtirloni 10y ago
This article is myopic in that it fails to mention the premises only (mostly) work for web-only startup companies relying heavily on cloud computing for everything. And even then, only the ones that don't see operational/infrastructure efficiency as a differentiator. If you take those constraints into account, it makes a lot of sense: just ask your developers to learn the very basics of those 3rd-party services you signed up to and things will go fine. A developer will just be making sure he/she uses the logging provider API, that their 3rd-party monitoring and alerting system is updated, the automatic third-party load balancer has a little checkbox saying backend monitoring is configured, etc. Behind those magic services there will be your traditional Ops people (now better equipped with automation tools) making sure everything is running smoothly, properly sized, monitored and troubleshooted when it fails (it WILL fail, in unexpected ways, more often than you'd like). People parroting "Ops-less" and "Server-less" buzzwords seem to be so detached from the daily activities required to maintaining real-world computing systems that they forget it exists. It's really frustrating when the SV zeitgeist living in a distorted reality (or a very narrow one) proposes this is the way things are or will certainly be in the near future. You just need to skim over all the GitHub issues in these magical tools (or countless articles with workarounds for all sorts of issues) to learn things aren't that simple as they pretend to be (which is fine, things take a lot of time to mature). The work is not changing (except for increased automation everywhere), it's just shifting to third-parties or far away teams in your own company. If only the computing world was that dreamy. I'll think about that next time I spent a day analyzing packets with tcpdump, coming up with a patch for a basic flaw in some tool, profiling some code that is destroying the storage system, etc. Yeah, that won't be distracting at all for developers. Maybe they don't need to work on their team's core competencies. Everything will "just" work. </sarcasm, sorry>
perlgeek 10y ago
It's not just about the number of calls you receive. Typically if you're on call rotation, you're supposed to be able to get near a keyboard in $X minutes (where in our case, $X is about 15, I think). I simply don't want to have to ask myself the question "can I do $activity with my family this weekend?", with the answer potentially being "no, because on-call duty". I totally agree with your point about a proactive maintenance. Developers can try to make their software easier to operator, and generally do if provided with the right incentives, but that doesn't remove the need for capacity planning, for example.
- sqldba 10y ago
  I'm on a roster 1 week on 2 weeks off and so it becomes very bearable given proper resources (i.e. a laptop and 3G/4G dongle). You throw it in your car boot and forget about it. You might get a call but unless you're out cliff climbing every weekend or something it's unlikely to be during anything important (about the only thing I avoid scheduling is dinner/movies with friends because skipping out on that would suck).

by sqldba 10y ago

gtirloni 10y ago

This article is myopic in that it fails to mention the premises only (mostly) work for web-only startup companies relying heavily on cloud computing for everything. And even then, only the ones that don't see operational/infrastructure efficiency as a differentiator. If you take those constraints into account, it makes a lot of sense: just ask your developers to learn the very basics of those 3rd-party services you signed up to and things will go fine. A developer will just be making sure he/she uses the logging provider API, that their 3rd-party monitoring and alerting system is updated, the automatic third-party load balancer has a little checkbox saying backend monitoring is configured, etc. Behind those magic services there will be your traditional Ops people (now better equipped with automation tools) making sure everything is running smoothly, properly sized, monitored and troubleshooted when it fails (it WILL fail, in unexpected ways, more often than you'd like). People parroting "Ops-less" and "Server-less" buzzwords seem to be so detached from the daily activities required to maintaining real-world computing systems that they forget it exists. It's really frustrating when the SV zeitgeist living in a distorted reality (or a very narrow one) proposes this is the way things are or will certainly be in the near future. You just need to skim over all the GitHub issues in these magical tools (or countless articles with workarounds for all sorts of issues) to learn things aren't that simple as they pretend to be (which is fine, things take a lot of time to mature). The work is not changing (except for increased automation everywhere), it's just shifting to third-parties or far away teams in your own company. If only the computing world was that dreamy. I'll think about that next time I spent a day analyzing packets with tcpdump, coming up with a patch for a basic flaw in some tool, profiling some code that is destroying the storage system, etc. Yeah, that won't be distracting at all for developers. Maybe they don't need to work on their team's core competencies. Everything will "just" work. </sarcasm, sorry>

perlgeek 10y ago

It's not just about the number of calls you receive. Typically if you're on call rotation, you're supposed to be able to get near a keyboard in $X minutes (where in our case, $X is about 15, I think). I simply don't want to have to ask myself the question "can I do $activity with my family this weekend?", with the answer potentially being "no, because on-call duty". I totally agree with your point about a proactive maintenance. Developers can try to make their software easier to operator, and generally do if provided with the right incentives, but that doesn't remove the need for capacity planning, for example.

sqldba 10y ago
I'm on a roster 1 week on 2 weeks off and so it becomes very bearable given proper resources (i.e. a laptop and 3G/4G dongle). You throw it in your car boot and forget about it. You might get a call but unless you're out cliff climbing every weekend or something it's unlikely to be during anything important (about the only thing I avoid scheduling is dinner/movies with friends because skipping out on that would suck).