4 ms·

This is great in principle, BUT.... it only works when the dev team are given the time and automony to prioritise the work. While many organisations pay lip se

This is great in principle, BUT.... it only works when the dev team are given the time and automony to prioritise the work. While many organisations pay lip service to uptime and reliability, the reality is that those are not sexy or exciting and Bob in sales just snared this million dollar client that will sign as soon as we implement frobnication with Othr. So dev team can either work on fixing the reliabilty issue that means they don't get paged at 3am or implement the thing that get the company another $1m. Which do you think they get to do?

dvtrn 5y ago
While many organisations pay lip service to uptime and reliability I had a very terse conversation about this, and directly included the words “lip service” to my superiors today about the very topic of on-call and overnight work this morning. I’m fed up, and made it known. Because IMO you’re 100% right, too many orgs pay lip service to availability and SLAs, yet deliberately choose to underinvest in tooling and resources to help mature the capabilities of incident responders. Does it bring it revenue? Probably not, but bailing water doesn’t move a ship across open seas either, you still do it because if you don’t surprise, no more fucking ship. Instead of a well defined monitoring system I get $40/mo for the lowest tier status board that’s generated more false-positives than actual on cable alerts, no matter how I tune it or try to define parameters. Instead of a mature incident response platform I’m demanded to conduct Spreadsheet postmortems. Instead of centralized logging and telemetry I get to troubleshoot exceptions over a bunch of shell windows. Incident response is still a mess of people slapping keyboards, there’s no triage, there’s no sense of structure or command of the process. And here I am, an SRE being prioritized to work on backing up service accounts because we might not get a contract with this trucking company if we don’t have that capacity. Sigh.
- cgrealy 5y ago
  Unfortunately, this kind of thing is almost impossible to deal with from a grass roots angle. Unless you have a culture that values these things from the top down, you're going to give yourself a concussion banging your head on a brick wall.
  - dvtrn 5y ago
    Mate I’m sure I already have. Been battling a migraine since 10am. (Joking, it’s really just a plain old sinus headache)

dvtrn 5y ago

While many organisations pay lip service to uptime and reliability I had a very terse conversation about this, and directly included the words “lip service” to my superiors today about the very topic of on-call and overnight work this morning. I’m fed up, and made it known. Because IMO you’re 100% right, too many orgs pay lip service to availability and SLAs, yet deliberately choose to underinvest in tooling and resources to help mature the capabilities of incident responders. Does it bring it revenue? Probably not, but bailing water doesn’t move a ship across open seas either, you still do it because if you don’t surprise, no more fucking ship. Instead of a well defined monitoring system I get $40/mo for the lowest tier status board that’s generated more false-positives than actual on cable alerts, no matter how I tune it or try to define parameters. Instead of a mature incident response platform I’m demanded to conduct Spreadsheet postmortems. Instead of centralized logging and telemetry I get to troubleshoot exceptions over a bunch of shell windows. Incident response is still a mess of people slapping keyboards, there’s no triage, there’s no sense of structure or command of the process. And here I am, an SRE being prioritized to work on backing up service accounts because we might not get a contract with this trucking company if we don’t have that capacity. Sigh.

cgrealy 5y ago
Unfortunately, this kind of thing is almost impossible to deal with from a grass roots angle. Unless you have a culture that values these things from the top down, you're going to give yourself a concussion banging your head on a brick wall.
- dvtrn 5y ago
  Mate I’m sure I already have. Been battling a migraine since 10am. (Joking, it’s really just a plain old sinus headache)