4 ms·

This article seems to assume that everything runs in the cloud. If not, Ops still have a big part in operating the self-service infrastructure that Susan talks

This article seems to assume that everything runs in the cloud. If not, Ops still have a big part in operating the self-service infrastructure that Susan talks about. Another point I'd like to make is that if developers need to take on more operational responsibility, it creates another barrier on entry. As somebody with a family, I don't want to be regularly on-call. And on the technical side, developers already need to know about the problem domain, the programming language, algorithm and data structures, design patterns, testing, version control and so on. Piling more operational knowledge onto this heap seems hardly promising to on-board more developers, and finding good developers outside the big tech hubs is already a pain.

sqldba 10y ago
I also feel like the "on call" thing is oversold a little. I work in operations and I'm on call. Some months I hear nothing, some weeks everything goes wrong, but on average I get about one call a month. That's not so bad. The thing is I spend 40 hours a week working out how to do proactive maintenance, monitoring, and alerting on those 400 systems I look after, so that I don't get called outside of hours. I mean that's what my operations job is meant to be, right? Developers can't do that. They're focused on one system or a few systems at most and they're developing 40 hours a week. When the SAN dies and every server in the DC crashes there's no point calling them and asking why they're not on top of it and to try to develop some solution to improve the speed of recovery next time. Me? I built a Jenkins server so next time it happens I press a button and get an immediate dashboard of where to focus my attention. Developers aren't operations. Things may change a little when things go into the deep cloud though and you have services instead of servers running services, seeing as operations may not know what to do with those. On the other hand maybe there is meant to be a handover of some kind so that one person can look after everything instead of 10 developers each having a small stake in their little piece of world.
- gtirloni 10y ago
  This article is myopic in that it fails to mention the premises only (mostly) work for web-only startup companies relying heavily on cloud computing for everything. And even then, only the ones that don't see operational/infrastructure efficiency as a differentiator. If you take those constraints into account, it makes a lot of sense: just ask your developers to learn the very basics of those 3rd-party services you signed up to and things will go fine. A developer will just be making sure he/she uses the logging provider API, that their 3rd-party monitoring and alerting system is updated, the automatic third-party load balancer has a little checkbox saying backend monitoring is configured, etc. Behind those magic services there will be your traditional Ops people (now better equipped with automation tools) making sure everything is running smoothly, properly sized, monitored and troubleshooted when it fails (it WILL fail, in unexpected ways, more often than you'd like). People parroting "Ops-less" and "Server-less" buzzwords seem to be so detached from the daily activities required to maintaining real-world computing systems that they forget it exists. It's really frustrating when the SV zeitgeist living in a distorted reality (or a very narrow one) proposes this is the way things are or will certainly be in the near future. You just need to skim over all the GitHub issues in these magical tools (or countless articles with workarounds for all sorts of issues) to learn things aren't that simple as they pretend to be (which is fine, things take a lot of time to mature). The work is not changing (except for increased automation everywhere), it's just shifting to third-parties or far away teams in your own company. If only the computing world was that dreamy. I'll think about that next time I spent a day analyzing packets with tcpdump, coming up with a patch for a basic flaw in some tool, profiling some code that is destroying the storage system, etc. Yeah, that won't be distracting at all for developers. Maybe they don't need to work on their team's core competencies. Everything will "just" work. </sarcasm, sorry>
- perlgeek 10y ago
  It's not just about the number of calls you receive. Typically if you're on call rotation, you're supposed to be able to get near a keyboard in $X minutes (where in our case, $X is about 15, I think). I simply don't want to have to ask myself the question "can I do $activity with my family this weekend?", with the answer potentially being "no, because on-call duty". I totally agree with your point about a proactive maintenance. Developers can try to make their software easier to operator, and generally do if provided with the right incentives, but that doesn't remove the need for capacity planning, for example.
  - sqldba 10y ago
    I'm on a roster 1 week on 2 weeks off and so it becomes very bearable given proper resources (i.e. a laptop and 3G/4G dongle). You throw it in your car boot and forget about it. You might get a call but unless you're out cliff climbing every weekend or something it's unlikely to be during anything important (about the only thing I avoid scheduling is dinner/movies with friends because skipping out on that would suck).
donavanm 10y ago
WRT to "the cloud" are you conflating infrastructure, systems tooling, and operations? Theyre not the same thing. For example I cant imagine why a monitoring system would necessitate more "operations" costs than an email or business logic system. It's possible to invest in either, driving down total "ops." Or to skip investments and pay those recurring costs in adhoc people time. Very very few subjects actually necessitate a humans continual interaction.
- perlgeek 10y ago
  Reformulated: If my services run on-premise, a developer isn't the best person to investigate and deal with a hardware failure. That's a role for a traditional ops person, even in the future.
  - donavanm 10y ago
    I think we're driving towards the same point with different language. You traditional ops person is my hourly employee working manual tasks from a queue. Your hardware failure case is actually remarkably similar for both on-premise and "cloud". In either case the development team can either invest in negating single component failures, or pay it in an adhoc fashion when it occurs. If that single server dying in the night wakes anyone up your business has chosen the latter for you. The difference that Ive seen is scale. With 100 servers it makes sense to pay the adhoc failures a few times per year. With 1000 servers its a few times per month. And at 10,000 its time to acknowledge the continual cost, get over The Really Big Server design, and hire an hourly tech to take touch that constant queue of broken hardware. Feel free to substitute "instance" or "droplet" or "router" in the proceeding paragraph.
    - perlgeek 10y ago
      Even if a single server failure doesn't wake anybody in the middle of the night, it has to be dealt with eventually. Otherwise dead hardware piles up in your racks. So, work for operators. But that's just the simplest case. Network congestion needs to be debugged, operating systems updated, security breaches investigated, and so on. To think that these types of activity can be automated in the near future is unrealistic. And burdening application developers with such tasks also seems like a weird choice. And since you made a point about scale: The larger the scale, the harder it becomes to investigate such issues.
rodgerd 10y ago
> As somebody with a family, I don't want to be regularly on-call. Neither do the ops team who pick up the sacks of shit developers keep chucking over the wall. What makes your family more important than theirs?
sciurus 10y ago
Susan works for Uber, which runs its own datacenters. See, for example, http://www.datacenterknowledge.com/archives/2016/09/19/ex-ebay-infrastructure-chief-takes-uber-data-center-strategy/ http://www.datacenterknowledge.com/archives/2016/09/19/ex-eb...

sqldba 10y ago

I also feel like the "on call" thing is oversold a little. I work in operations and I'm on call. Some months I hear nothing, some weeks everything goes wrong, but on average I get about one call a month. That's not so bad. The thing is I spend 40 hours a week working out how to do proactive maintenance, monitoring, and alerting on those 400 systems I look after, so that I don't get called outside of hours. I mean that's what my operations job is meant to be, right? Developers can't do that. They're focused on one system or a few systems at most and they're developing 40 hours a week. When the SAN dies and every server in the DC crashes there's no point calling them and asking why they're not on top of it and to try to develop some solution to improve the speed of recovery next time. Me? I built a Jenkins server so next time it happens I press a button and get an immediate dashboard of where to focus my attention. Developers aren't operations. Things may change a little when things go into the deep cloud though and you have services instead of servers running services, seeing as operations may not know what to do with those. On the other hand maybe there is meant to be a handover of some kind so that one person can look after everything instead of 10 developers each having a small stake in their little piece of world.

gtirloni 10y ago
This article is myopic in that it fails to mention the premises only (mostly) work for web-only startup companies relying heavily on cloud computing for everything. And even then, only the ones that don't see operational/infrastructure efficiency as a differentiator. If you take those constraints into account, it makes a lot of sense: just ask your developers to learn the very basics of those 3rd-party services you signed up to and things will go fine. A developer will just be making sure he/she uses the logging provider API, that their 3rd-party monitoring and alerting system is updated, the automatic third-party load balancer has a little checkbox saying backend monitoring is configured, etc. Behind those magic services there will be your traditional Ops people (now better equipped with automation tools) making sure everything is running smoothly, properly sized, monitored and troubleshooted when it fails (it WILL fail, in unexpected ways, more often than you'd like). People parroting "Ops-less" and "Server-less" buzzwords seem to be so detached from the daily activities required to maintaining real-world computing systems that they forget it exists. It's really frustrating when the SV zeitgeist living in a distorted reality (or a very narrow one) proposes this is the way things are or will certainly be in the near future. You just need to skim over all the GitHub issues in these magical tools (or countless articles with workarounds for all sorts of issues) to learn things aren't that simple as they pretend to be (which is fine, things take a lot of time to mature). The work is not changing (except for increased automation everywhere), it's just shifting to third-parties or far away teams in your own company. If only the computing world was that dreamy. I'll think about that next time I spent a day analyzing packets with tcpdump, coming up with a patch for a basic flaw in some tool, profiling some code that is destroying the storage system, etc. Yeah, that won't be distracting at all for developers. Maybe they don't need to work on their team's core competencies. Everything will "just" work. </sarcasm, sorry>
perlgeek 10y ago
It's not just about the number of calls you receive. Typically if you're on call rotation, you're supposed to be able to get near a keyboard in $X minutes (where in our case, $X is about 15, I think). I simply don't want to have to ask myself the question "can I do $activity with my family this weekend?", with the answer potentially being "no, because on-call duty". I totally agree with your point about a proactive maintenance. Developers can try to make their software easier to operator, and generally do if provided with the right incentives, but that doesn't remove the need for capacity planning, for example.
- sqldba 10y ago
  I'm on a roster 1 week on 2 weeks off and so it becomes very bearable given proper resources (i.e. a laptop and 3G/4G dongle). You throw it in your car boot and forget about it. You might get a call but unless you're out cliff climbing every weekend or something it's unlikely to be during anything important (about the only thing I avoid scheduling is dinner/movies with friends because skipping out on that would suck).

donavanm 10y ago

WRT to "the cloud" are you conflating infrastructure, systems tooling, and operations? Theyre not the same thing. For example I cant imagine why a monitoring system would necessitate more "operations" costs than an email or business logic system. It's possible to invest in either, driving down total "ops." Or to skip investments and pay those recurring costs in adhoc people time. Very very few subjects actually necessitate a humans continual interaction.

perlgeek 10y ago
Reformulated: If my services run on-premise, a developer isn't the best person to investigate and deal with a hardware failure. That's a role for a traditional ops person, even in the future.
- donavanm 10y ago
  I think we're driving towards the same point with different language. You traditional ops person is my hourly employee working manual tasks from a queue. Your hardware failure case is actually remarkably similar for both on-premise and "cloud". In either case the development team can either invest in negating single component failures, or pay it in an adhoc fashion when it occurs. If that single server dying in the night wakes anyone up your business has chosen the latter for you. The difference that Ive seen is scale. With 100 servers it makes sense to pay the adhoc failures a few times per year. With 1000 servers its a few times per month. And at 10,000 its time to acknowledge the continual cost, get over The Really Big Server design, and hire an hourly tech to take touch that constant queue of broken hardware. Feel free to substitute "instance" or "droplet" or "router" in the proceeding paragraph.
  - perlgeek 10y ago
    Even if a single server failure doesn't wake anybody in the middle of the night, it has to be dealt with eventually. Otherwise dead hardware piles up in your racks. So, work for operators. But that's just the simplest case. Network congestion needs to be debugged, operating systems updated, security breaches investigated, and so on. To think that these types of activity can be automated in the near future is unrealistic. And burdening application developers with such tasks also seems like a weird choice. And since you made a point about scale: The larger the scale, the harder it becomes to investigate such issues.

rodgerd 10y ago

> As somebody with a family, I don't want to be regularly on-call. Neither do the ops team who pick up the sacks of shit developers keep chucking over the wall. What makes your family more important than theirs?

sciurus 10y ago

Susan works for Uber, which runs its own datacenters. See, for example, http://www.datacenterknowledge.com/archives/2016/09/19/ex-ebay-infrastructure-chief-takes-uber-data-center-strategy/ http://www.datacenterknowledge.com/archives/2016/09/19/ex-eb...