5 ms·

Any tips for the vast majority of SRE groups where people are paid a fraction of google employees and never given any time to fix things?

by rb808 7y ago

superflit 7y ago
Double the caffeine intake and "RUN FOREST" :) I feel your pain.
BrandonY 7y ago
Tech debt is most easily measurable in repetitive operational work. Measure it. Bring a story to your leaders: "We are spending 60 hours per week doing repetitive task X. With 200 hours of work, we could eliminate this work. It would pay for itself in a month." If your leadership declines to take you up on this, escalate. If that fails, you must choose between continuing to do the repetitive operational work as instructed or leaving.
Aeolun 7y ago
Fix shit anyway. You are never going to get time to fix your operational procedures, so just be working on project A or B, but really consider fixing your procedures part of those projects.
johngalt 7y ago
You would be surprised how far you can get with consistent pressure and time. Small iterative changes. It works for practices/culture as well as technology. There is never some big meeting where everyone decides that 'operations are going to change! From now on we are going to X!" Well sometimes there are such meetings, but often means merely adding another demand. Like a new years resolution, it is a strongly voiced command for better results, with no real changes behind it. Real improvement is about adding practices and habits. Starting with the most needed/highest payoff items and gradually building on them.
lonelappde 7y ago
High pay is great, but doesn't make better software. Do Googlers get time to fix things?
- saalweachter 7y ago
  I would say that Googlers can make time to fix things. In the parts of Google I've seen, engineers are largely given projects with a timeline of weeks or months and left to structure their time themselves. You're usually given more projects than can feasibly be accomplished in the time allotted, and learning which are high priority and which to let slide a quarter is a bit of an art, but as long as your projects are moving forward you can usually structure some time for paying down tech debt or a side project, either an official 20% or just experimenting with your area of responsibility outside any roadmaps. I tend to structure my weeks with Tuesday as the designated day to fix anything that's broken or, if everything is running smoothly, to pay down tech debt more for the opposite reason, so that I don't get caught in a rabbit-hole of fixing things that aren't broken and neglect to make progress on the new work.
- jacques_chester 7y ago
  According to the first of the SRE books, engineers working under the SRE banner are expected to devote approximately 50% to operational activities and 50% to engineering intended to make the other 50% easier. Given that the usual ration is 100% : -100%, 50:50 is going to be helpful in escaping capability traps.
  - smueller1234 7y ago
    It's actually "at most 50% on toil". But of course it's also hard to measure exactly. It's more of a barrier that if we exceed it, the team probably needs serious help! Ultimately, Google engineers can move between teams pretty freely, so if we allow a team to descend into an operational or a deadline-induced death march, and we don't address that quickly, chances are the engineers will move to another team. It's sometimes frustrating not to have more control as a manager, but it's a very nicely self correcting mechanism and fixes various incentives for us managers. (I'm a manager in Google SRE. Not speaking for Google.)
    - jacques_chester 7y ago
      I like the idea of at least targeting a high fraction of total bandwidth for "making stuff better". My reference to "capability traps" wasn't accidental, it's a serious risk to any business where maintenance and improvement has low observability vs production outputs[0]. In that situation economists rightly predict that effort is skewed towards what can be observed more easily ("equal compensation principle")[1]. Under those conditions it's easier to fix a time-spent target and observe time allocated, even if only approximately. [0] https://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_.pdf https://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01... [1] https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem#Contract_design https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...

Any tips for the vast majority of SRE groups where people are paid a fraction of google employees and never given any time to fix things?

by rb808 7y ago

Any tips for the vast majority of SRE groups where people are paid a fraction of google employees and never given any time to fix things?

superflit 7y ago

Double the caffeine intake and "RUN FOREST" :) I feel your pain.

BrandonY 7y ago

Tech debt is most easily measurable in repetitive operational work. Measure it. Bring a story to your leaders: "We are spending 60 hours per week doing repetitive task X. With 200 hours of work, we could eliminate this work. It would pay for itself in a month." If your leadership declines to take you up on this, escalate. If that fails, you must choose between continuing to do the repetitive operational work as instructed or leaving.

Aeolun 7y ago

Fix shit anyway. You are never going to get time to fix your operational procedures, so just be working on project A or B, but really consider fixing your procedures part of those projects.

johngalt 7y ago

You would be surprised how far you can get with consistent pressure and time. Small iterative changes. It works for practices/culture as well as technology. There is never some big meeting where everyone decides that 'operations are going to change! From now on we are going to X!" Well sometimes there are such meetings, but often means merely adding another demand. Like a new years resolution, it is a strongly voiced command for better results, with no real changes behind it. Real improvement is about adding practices and habits. Starting with the most needed/highest payoff items and gradually building on them.

lonelappde 7y ago

High pay is great, but doesn't make better software. Do Googlers get time to fix things?

saalweachter 7y ago
I would say that Googlers can make time to fix things. In the parts of Google I've seen, engineers are largely given projects with a timeline of weeks or months and left to structure their time themselves. You're usually given more projects than can feasibly be accomplished in the time allotted, and learning which are high priority and which to let slide a quarter is a bit of an art, but as long as your projects are moving forward you can usually structure some time for paying down tech debt or a side project, either an official 20% or just experimenting with your area of responsibility outside any roadmaps. I tend to structure my weeks with Tuesday as the designated day to fix anything that's broken or, if everything is running smoothly, to pay down tech debt more for the opposite reason, so that I don't get caught in a rabbit-hole of fixing things that aren't broken and neglect to make progress on the new work.
jacques_chester 7y ago
According to the first of the SRE books, engineers working under the SRE banner are expected to devote approximately 50% to operational activities and 50% to engineering intended to make the other 50% easier. Given that the usual ration is 100% : -100%, 50:50 is going to be helpful in escaping capability traps.
- smueller1234 7y ago
  It's actually "at most 50% on toil". But of course it's also hard to measure exactly. It's more of a barrier that if we exceed it, the team probably needs serious help! Ultimately, Google engineers can move between teams pretty freely, so if we allow a team to descend into an operational or a deadline-induced death march, and we don't address that quickly, chances are the engineers will move to another team. It's sometimes frustrating not to have more control as a manager, but it's a very nicely self correcting mechanism and fixes various incentives for us managers. (I'm a manager in Google SRE. Not speaking for Google.)
  - jacques_chester 7y ago
    I like the idea of at least targeting a high fraction of total bandwidth for "making stuff better". My reference to "capability traps" wasn't accidental, it's a serious risk to any business where maintenance and improvement has low observability vs production outputs[0]. In that situation economists rightly predict that effort is skewed towards what can be observed more easily ("equal compensation principle")[1]. Under those conditions it's easier to fix a time-spent target and observe time allocated, even if only approximately. [0] https://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_.pdf https://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01... [1] https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem#Contract_design https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...