6 ms·

DevOps Students Learn the Value of Uptime With 3 a.m. Calls

28 points by lclark 10y ago

lostcolony 10y ago
I expect it'd be more helpful to force it on managers in a management course. That way when ops says "This thing you paid an outside consultant to develop is fragile as hell and can fall over at any time" they'll be inclined to listen. And on developers, so they realize that reliability is as much a requirement as any of the functional requirements they were given.
- mancerayder 10y ago
  Agreed. While it's normal and expected that the 'Ops' or 'DevOps' or the artist previously known as 'Sys Admin' is expected to be available off-hours for stuff they didn't build (read: emergencies-that-are-not-within-their-purview-to-prevent)... on the flipside, the tendency of many companies to reduce the burden of responsibility of developers and product managers who are often directly responsible for instability in the environment, is prevalent in a way that makes my blood boil. Going back to this article, it's almost a celebration that DevOps folks have to shoulder that burden. While it's normal to be on-call, being woken up all the time is a sign of a badly run infrastructure OR release/change-management practices that are rushed and/or feeble. Everywhere I've worked I've stood up against this tendency, analyzing each issue that causes a page and seeing how it could be prevented. A quarter of the time it's technical: creating redundancy, deep diving into an ongoing issue, doing load tests and capacity management, etc. The rest of the time it's political: oh, the new code caused the memory to be sucked dry from the system, this started precisely after the last release (proof, here's a graph from my Check_MK setup); oh, the devs ran a crap query again on the Hadoop cluster even though we warned them not to do X. etc. I think if they want to prepare future DevOps students, rather than using PagerDuty as per the article, maybe they should give them shots of liquor and strong beer, their livers could use the preparation. It's an incredibly political role.
  - x0x0 10y ago
    I've had success granting myself comp time when I've been up late or worked on the weekend to fix ops issues. Eg if I'm up at 2 or 3, I come in at noon or 1pm, and if it takes more than 2 hours, I just take a full day. The one boss who complained shut up when I told him he had 3 options: (1) deal, (2) remove ops responsibilities, (3) I quit on the spot. It's incredibly necessary to push the costs of sloppy code to be really apparent to management.
    - lostcolony 10y ago
      Well said. I've done the same thing, and fought to make it standard department wide. I'm a dev, I -want- to do the right thing, but sometimes management insists on cutting corners. Fine, but the interest payments on that technical debt should be upon their head.
    - mancerayder 10y ago
      Amen, brother.
deleted 10y ago
[deleted]
aayala 10y ago
people will burn out soon or later.
hartator 10y ago
After that, we keep seeing posts about how coders are earning too much money.
hexadec0079 10y ago
Wait, this seems to ignore the fact that with good change controls and sound code, products do not just fail at 3am. If they call everyone at 3am without a failure the student could have prevented, that does not teach anything other than how to answer a phone. Instead, teach them to properly engineer and document their solution such that they aren't called at 3am. This seems like a waste of a good night's sleep to me.
- pbnjay 10y ago
  My hope would be that they somehow engineered a failure on all the systems that they would have to debug at 3am
  - hexadec0079 10y ago
    So the students engineered themselves to fail or the instructors gave them a terrible system that is bound to fail? Shit going sideways is a fact of life, but you can at least let people have the chance to prevent the problems before they exist. While there is always someone else's mess you have to support, that does not seem like a great teaching moment to me.
- cheald 10y ago
  > Wait, this seems to ignore the fact that with good change controls and sound code, products do not just fail at 3am. Because network outages never happen, disks never fail or fill up, memory is never an issue, programs always deal with only the data they were expected to, products never do more traffic than expected, and all infrastructure software ships completely bug-free. If you aren't occasionally up at 3 AM fixing unexpected outages, then either you haven't deployed a project that requires uptime or you're paying someone else to do it for you.
  - hexadec0079 10y ago
    The issue is not with problems, but with the arbitrary nature of this as a learning tool. Getting a page to fix an issue that you forsaw and prevented is dumb. Teach them to make resilient systems rather than get out of bed. I have never gotten up at 3am to fix an outage because outages do not impact systems that widely. If it is a network problem, let the network team do their job. All of your other problems are solved with multiple HA/ load balanced servers, monitoring, and proper testing.
FireBeyond 10y ago
Bleh. As a DevOps guy who is a paramedic, I just cringed at the image of someone sprinting to the computer (I know it's exaggerated - or at least I hope so). We don't run to cardiac arrests. Walk fast, with a purpose. I fail to see what production issue necessitates me running down the hall at 3am. And DevOps students should be learning how to prevent (sorry, minimize) this, not conducting fire drills.
- bogomipz 10y ago
  Yes exactly. Waking someone up at 3:00AM as a training exercise is ridiculous. That prepares you for nothing. Is this really part of the curriculum for a school that charges 17% of your first 3 year's salary after completing the course? Terrible.
  - humbleMouse 10y ago
    3 years??? Holy cats. I doubt that would hold up in non-compete court.
    - bogomipz 10y ago
      Yeah what a joke right? "That is why there is no upfront cost to join Holberton school. We only charge 17% of your internship earnings and 17% of your salary over 3 years once you find a job. If the company you join agrees to pay us a placement fee, this percentage will be reduced." https://www.holbertonschool.com/education https://www.holbertonschool.com/education
- toomuchtodo 10y ago
  As a DevOps guy at a startup, I'm expected to respond to alerts within 1-3 minutes at my current employer, 24/7 when I'm on my rotation. We rotate devs in to allow them to try building something "devopsy", but they're never included in the on-call rotation. Its unsettling.
  - serge2k 10y ago
groby_b 10y ago
Hey, DevOps has discovered hazing rituals. And keeping students on-call 24/7 is abuse, nothing less.
downrightmike 10y ago
Cost: 17% for 3 years isn't bad at all
- serge2k 10y ago
  Would have been easily more than my students loans for university
  - Declanomous 10y ago
    On the flip side, with income-based repayment you are paying 10% of your discretionary income for 20 years or so. Depending on what your salary ends up being this might not be so bad. For instance, 17% for three years would be way less than my student loans. I do work for a non-profit organization though.
serge2k 10y ago
> DevOps is a set of practices, a philosophy aiming for agile operations, to expand the collaboration between developers and operation folks to make them work toward the same goal: contribute to the entire product life cycle, from design, development and shipping, up to the production stage. This is a radical shift from the industry norm of separate engineering and operations departments which often operate in opposition to each other. Lots of words for "You can save money be shoving ops bullshit onto devs instead". Oh right, it's all about collaboration and ownership and it's way more efficient this way. Just like open offices. I am willing to acknowledge there are some pros to this system, but it's still screwing over devs by shoving extra (inconvenient) work onto their plates with no extra remuneration.
- hedwall 10y ago
  We deal with dev bullshit in production all day, every day. Maybe we should all care for it instead...
cturner 10y ago
Devops started as a message that developers were responsible for the solution they were building, not just for throwing builds over to deployment. The phrase turned into HR code for firefighters who hack scripts on production. When it's built right, operations doesn't exist. On this theme - we're hiring. Central London. We need a platform/infrastructure engineer: solid unix, automation-centric, someone who will drive evolution of the infrastructure and release platform. https://clearmatics.workable.com/jobs/257440 https://clearmatics.workable.com/jobs/257440

28 points by lclark 10y ago

I expect it'd be more helpful to force it on managers in a management course. That way when ops says "This thing you paid an outside consultant to develop is fragile as hell and can fall over at any time" they'll be inclined to listen. And on developers, so they realize that reliability is as much a requirement as any of the functional requirements they were given.

mancerayder 10y ago
Agreed. While it's normal and expected that the 'Ops' or 'DevOps' or the artist previously known as 'Sys Admin' is expected to be available off-hours for stuff they didn't build (read: emergencies-that-are-not-within-their-purview-to-prevent)... on the flipside, the tendency of many companies to reduce the burden of responsibility of developers and product managers who are often directly responsible for instability in the environment, is prevalent in a way that makes my blood boil. Going back to this article, it's almost a celebration that DevOps folks have to shoulder that burden. While it's normal to be on-call, being woken up all the time is a sign of a badly run infrastructure OR release/change-management practices that are rushed and/or feeble. Everywhere I've worked I've stood up against this tendency, analyzing each issue that causes a page and seeing how it could be prevented. A quarter of the time it's technical: creating redundancy, deep diving into an ongoing issue, doing load tests and capacity management, etc. The rest of the time it's political: oh, the new code caused the memory to be sucked dry from the system, this started precisely after the last release (proof, here's a graph from my Check_MK setup); oh, the devs ran a crap query again on the Hadoop cluster even though we warned them not to do X. etc. I think if they want to prepare future DevOps students, rather than using PagerDuty as per the article, maybe they should give them shots of liquor and strong beer, their livers could use the preparation. It's an incredibly political role.
- x0x0 10y ago
  I've had success granting myself comp time when I've been up late or worked on the weekend to fix ops issues. Eg if I'm up at 2 or 3, I come in at noon or 1pm, and if it takes more than 2 hours, I just take a full day. The one boss who complained shut up when I told him he had 3 options: (1) deal, (2) remove ops responsibilities, (3) I quit on the spot. It's incredibly necessary to push the costs of sloppy code to be really apparent to management.
  - lostcolony 10y ago
    Well said. I've done the same thing, and fought to make it standard department wide. I'm a dev, I -want- to do the right thing, but sometimes management insists on cutting corners. Fine, but the interest payments on that technical debt should be upon their head.
  - mancerayder 10y ago
    Amen, brother.

deleted 10y ago

[deleted]

aayala 10y ago

people will burn out soon or later.

hartator 10y ago

After that, we keep seeing posts about how coders are earning too much money.

hexadec0079 10y ago

Wait, this seems to ignore the fact that with good change controls and sound code, products do not just fail at 3am. If they call everyone at 3am without a failure the student could have prevented, that does not teach anything other than how to answer a phone. Instead, teach them to properly engineer and document their solution such that they aren't called at 3am. This seems like a waste of a good night's sleep to me.

pbnjay 10y ago
My hope would be that they somehow engineered a failure on all the systems that they would have to debug at 3am
- hexadec0079 10y ago
  So the students engineered themselves to fail or the instructors gave them a terrible system that is bound to fail? Shit going sideways is a fact of life, but you can at least let people have the chance to prevent the problems before they exist. While there is always someone else's mess you have to support, that does not seem like a great teaching moment to me.
cheald 10y ago
> Wait, this seems to ignore the fact that with good change controls and sound code, products do not just fail at 3am. Because network outages never happen, disks never fail or fill up, memory is never an issue, programs always deal with only the data they were expected to, products never do more traffic than expected, and all infrastructure software ships completely bug-free. If you aren't occasionally up at 3 AM fixing unexpected outages, then either you haven't deployed a project that requires uptime or you're paying someone else to do it for you.
- hexadec0079 10y ago
  The issue is not with problems, but with the arbitrary nature of this as a learning tool. Getting a page to fix an issue that you forsaw and prevented is dumb. Teach them to make resilient systems rather than get out of bed. I have never gotten up at 3am to fix an outage because outages do not impact systems that widely. If it is a network problem, let the network team do their job. All of your other problems are solved with multiple HA/ load balanced servers, monitoring, and proper testing.

FireBeyond 10y ago

Bleh. As a DevOps guy who is a paramedic, I just cringed at the image of someone sprinting to the computer (I know it's exaggerated - or at least I hope so). We don't run to cardiac arrests. Walk fast, with a purpose. I fail to see what production issue necessitates me running down the hall at 3am. And DevOps students should be learning how to prevent (sorry, minimize) this, not conducting fire drills.

bogomipz 10y ago
Yes exactly. Waking someone up at 3:00AM as a training exercise is ridiculous. That prepares you for nothing. Is this really part of the curriculum for a school that charges 17% of your first 3 year's salary after completing the course? Terrible.
- humbleMouse 10y ago
  3 years??? Holy cats. I doubt that would hold up in non-compete court.
  - bogomipz 10y ago
    Yeah what a joke right? "That is why there is no upfront cost to join Holberton school. We only charge 17% of your internship earnings and 17% of your salary over 3 years once you find a job. If the company you join agrees to pay us a placement fee, this percentage will be reduced." https://www.holbertonschool.com/education https://www.holbertonschool.com/education
toomuchtodo 10y ago
As a DevOps guy at a startup, I'm expected to respond to alerts within 1-3 minutes at my current employer, 24/7 when I'm on my rotation. We rotate devs in to allow them to try building something "devopsy", but they're never included in the on-call rotation. Its unsettling.
- serge2k 10y ago

groby_b 10y ago

Hey, DevOps has discovered hazing rituals. And keeping students on-call 24/7 is abuse, nothing less.

downrightmike 10y ago

Cost: 17% for 3 years isn't bad at all

serge2k 10y ago
Would have been easily more than my students loans for university
- Declanomous 10y ago
  On the flip side, with income-based repayment you are paying 10% of your discretionary income for 20 years or so. Depending on what your salary ends up being this might not be so bad. For instance, 17% for three years would be way less than my student loans. I do work for a non-profit organization though.

serge2k 10y ago

> DevOps is a set of practices, a philosophy aiming for agile operations, to expand the collaboration between developers and operation folks to make them work toward the same goal: contribute to the entire product life cycle, from design, development and shipping, up to the production stage. This is a radical shift from the industry norm of separate engineering and operations departments which often operate in opposition to each other. Lots of words for "You can save money be shoving ops bullshit onto devs instead". Oh right, it's all about collaboration and ownership and it's way more efficient this way. Just like open offices. I am willing to acknowledge there are some pros to this system, but it's still screwing over devs by shoving extra (inconvenient) work onto their plates with no extra remuneration.

hedwall 10y ago
We deal with dev bullshit in production all day, every day. Maybe we should all care for it instead...

cturner 10y ago

Devops started as a message that developers were responsible for the solution they were building, not just for throwing builds over to deployment. The phrase turned into HR code for firefighters who hack scripts on production. When it's built right, operations doesn't exist. On this theme - we're hiring. Central London. We need a platform/infrastructure engineer: solid unix, automation-centric, someone who will drive evolution of the infrastructure and release platform. https://clearmatics.workable.com/jobs/257440 https://clearmatics.workable.com/jobs/257440