5 ms·

And suddenly I understand why the worst tech job ive ever had as an Ops engineer was so bad. We really only existed to improve alerting, pipeline and wake up ot

And suddenly I understand why the worst tech job ive ever had as an Ops engineer was so bad. We really only existed to improve alerting, pipeline and wake up other engineers at 3am.

dilyevsky 4y ago
Majority of companies i talk to are really poorly run wrt to software operations. Case in point - misusing devops term to mean sysadmins/operators
lmarcos 4y ago
A sincere thanks. As a software engineer I couldn't care less about what happens to my company's services/products outside the 9-5 time range. Don't get me wrong, I give myself 100% at my job, keep myself educated regularly and I'm rather on the "boring and stable stuff" side of things (instead of the "shiny/trendy and unstable" side). I have commitments outside work and no amount of money is going to make me give more than the (already exhausting) 40h/week my contract states. The "you build it, you run it" may work for people on their 20s (they usually are excited to earn "easy money" by being oncall). For people on their 30s and above the extra oncall money is not worth at all.
- wpietri 4y ago
  I certainly believe that's true for you. But in the case where engineers choose not to ever run what they build, how do you reconnect the feedback loop? Put differently, I think one of the ways somebody goes from the "shiny/trendy and unstable" side to the "boring and stable stuff" side is by experiencing the operational pain of their choices. If the pain falls on others, will they still learn? Of course, the way you talk about your job makes me wonder if you are already experiencing so many systemic/managerial issues that there the feedback loops are already pretty broken, so this one may not make a ton of practical difference.
  - trombone5000 4y ago
    Engineers can run what they build during normal working hours. Oncall is a scourge not because of the experience of technical problems, but because people already working full time have to arrange their lives outside of work around a second "oncall job". A job which occurs after hours, one out of every X weeks. A dedicated, pure "Ops" night shift (perhaps in another time zone) would be more humane.
    - dilyevsky 4y ago
      Then build it in a way you almost never have to plug in outside of business hours.
      - trombone5000 4y ago
        Even if it were built perfectly, if engineers are still on-call, they would have to arrange their after-hours time around the possibility of an incident.
        dilyevsky 4y ago
        That's true but it's just a reality of being employed by a saas company these days. Customer support, sales, etc have those too (and usually less formalized and unpaid) so why are engineers immune to this? You can still probably find some shops that ship an offline distribution but that's becoming more rare.
    - kqr 4y ago
      > Engineers can run what they build during normal working hours. In my experience, this leads to design that pushes problems to outside of working hours. "We don't need to fix that edge case, just have the off-hours ops team do a manual workaround every now and then." Or "What does it matter that the deployment is error-prone? We can just schedule it with the off-hours ops team."
  - lmarcos 4y ago
    > If the pain falls on others, will they still learn? I think that depends on the seniority of the individual/team. In my experience, of course one can still learn. To give you a real example: years ago one of our systems went down on a Sunday morning and our team had no oncall people. The infrastructure team was the one who fixed the issue (don't remember the exact underlaying issue, but it did make clear one aspect of our service we didn't properly: signal handling). Next morning the team wrote down a Jira issue to improve the way we handle signals. Ticket got prioritized very high and was fixed the very same day. Now, what would have happened if the issue that Sunday morning was due to a bug in the software our team wrote? The same thing. The difference is that infra team would have no clue on how to fix the thing and would have to revert the service to a previous stable version. Would the business be fine with it? In our case, yeah. As a matter of fact, they didn't want to spend the extra money hiring ops people for each team to be on call. You see, if the business really cared, they would immediately have hired a software engineer willing to be on call... They just didn't care that much (and they couldn't force the current team to be oncall because our contracts didn't specify so and the average age in our team was around 35, and nobody wanted to be on call).
- arwhatever 4y ago
  “How about the whole team makes engineering decisions as though you’re unable to contact us after hours, or as though doing so were particularly costly.”
  - dvtrn 4y ago
    What, and break down all the monitoring and alerting silos we built by hiring a Devops engineer to come in and break down the development and infrastructure silos that were built when the company went ham adopting "Capital A" Agile?
- danielheath 4y ago
  “You build it, you run it” works just fine if you’re building something that doesn’t fail all the damn time. Work has had three out of hours pages in the last two years, all self resolved within a few minutes.

And suddenly I understand why the worst tech job ive ever had as an Ops engineer was so bad. We really only existed to improve alerting, pipeline and wake up other engineers at 3am.

dilyevsky 4y ago

Majority of companies i talk to are really poorly run wrt to software operations. Case in point - misusing devops term to mean sysadmins/operators

lmarcos 4y ago

A sincere thanks. As a software engineer I couldn't care less about what happens to my company's services/products outside the 9-5 time range. Don't get me wrong, I give myself 100% at my job, keep myself educated regularly and I'm rather on the "boring and stable stuff" side of things (instead of the "shiny/trendy and unstable" side). I have commitments outside work and no amount of money is going to make me give more than the (already exhausting) 40h/week my contract states. The "you build it, you run it" may work for people on their 20s (they usually are excited to earn "easy money" by being oncall). For people on their 30s and above the extra oncall money is not worth at all.

wpietri 4y ago
I certainly believe that's true for you. But in the case where engineers choose not to ever run what they build, how do you reconnect the feedback loop? Put differently, I think one of the ways somebody goes from the "shiny/trendy and unstable" side to the "boring and stable stuff" side is by experiencing the operational pain of their choices. If the pain falls on others, will they still learn? Of course, the way you talk about your job makes me wonder if you are already experiencing so many systemic/managerial issues that there the feedback loops are already pretty broken, so this one may not make a ton of practical difference.
- trombone5000 4y ago
  Engineers can run what they build during normal working hours. Oncall is a scourge not because of the experience of technical problems, but because people already working full time have to arrange their lives outside of work around a second "oncall job". A job which occurs after hours, one out of every X weeks. A dedicated, pure "Ops" night shift (perhaps in another time zone) would be more humane.
  - dilyevsky 4y ago
    Then build it in a way you almost never have to plug in outside of business hours.
    - trombone5000 4y ago
      Even if it were built perfectly, if engineers are still on-call, they would have to arrange their after-hours time around the possibility of an incident.
      - dilyevsky 4y ago
        That's true but it's just a reality of being employed by a saas company these days. Customer support, sales, etc have those too (and usually less formalized and unpaid) so why are engineers immune to this? You can still probably find some shops that ship an offline distribution but that's becoming more rare.
  - kqr 4y ago
    > Engineers can run what they build during normal working hours. In my experience, this leads to design that pushes problems to outside of working hours. "We don't need to fix that edge case, just have the off-hours ops team do a manual workaround every now and then." Or "What does it matter that the deployment is error-prone? We can just schedule it with the off-hours ops team."
- lmarcos 4y ago
  > If the pain falls on others, will they still learn? I think that depends on the seniority of the individual/team. In my experience, of course one can still learn. To give you a real example: years ago one of our systems went down on a Sunday morning and our team had no oncall people. The infrastructure team was the one who fixed the issue (don't remember the exact underlaying issue, but it did make clear one aspect of our service we didn't properly: signal handling). Next morning the team wrote down a Jira issue to improve the way we handle signals. Ticket got prioritized very high and was fixed the very same day. Now, what would have happened if the issue that Sunday morning was due to a bug in the software our team wrote? The same thing. The difference is that infra team would have no clue on how to fix the thing and would have to revert the service to a previous stable version. Would the business be fine with it? In our case, yeah. As a matter of fact, they didn't want to spend the extra money hiring ops people for each team to be on call. You see, if the business really cared, they would immediately have hired a software engineer willing to be on call... They just didn't care that much (and they couldn't force the current team to be oncall because our contracts didn't specify so and the average age in our team was around 35, and nobody wanted to be on call).
arwhatever 4y ago
“How about the whole team makes engineering decisions as though you’re unable to contact us after hours, or as though doing so were particularly costly.”
- dvtrn 4y ago
  What, and break down all the monitoring and alerting silos we built by hiring a Devops engineer to come in and break down the development and infrastructure silos that were built when the company went ham adopting "Capital A" Agile?
danielheath 4y ago
“You build it, you run it” works just fine if you’re building something that doesn’t fail all the damn time. Work has had three out of hours pages in the last two years, all self resolved within a few minutes.