2 ms·

Operating well means you could do things that doesn't scale/manually at first, but then you understand the process enough to automate it away so you can tackle

by a_c 4y ago

Operating well means you could do things that doesn't scale/manually at first, but then you understand the process enough to automate it away so you can tackle the next problem and you don't need to throwing people in linearly/exponentially. It applies to both software and organization level. There is a similar term in DevOps that I forgot. Hopefully someone can chime in.

PaulRobinson 4y ago
I think you're talking about eliminating toil. First time you have to step in and do something, cool, get in there, get it done. Second time, sounds like that wasn't a one-off, you have toil, and you need a runbook on the company wiki or internal doc site - that way anyone in the team (or organisation or company), can quickly step in and do that thing. Third time, have somebody else run through the runbook it, validate it, figure out if it can be improved. Perhaps have some better alarms and operational oversight. Now, start automating parts of the runbook. Some parts will be easy, some will be hard. Do the easy ones first and update the runbook. Automation will need alarms with the right thresholds. Think about those carefully and remember whenever an alarm fires there are only three things you should be doing: fix the problem; change the threshold; delete the alarm. If you do anything else - especially ignore it and just look at things and wait to see if anything else happens - you have misunderstood and are heading towards alarm fatigue. Don't do this. Ever. Remember the three possible actions. Once all the parts of the runbook are automated and alarms are working correctly to trigger automatic resolution, delete the runbook. You have now eliminated toil. This can apply to any manual operational task that needs to happen anywhere in a business, but is most easily used when improving resiliency and durability in systems that have easy automatic resolution (such as horizontal scaling by adding more instances when loads are high on existing infra). It's exactly what an AWS EC2 ASG is there to do, for example.

by a_c 4y ago

PaulRobinson 4y ago

I think you're talking about eliminating toil. First time you have to step in and do something, cool, get in there, get it done. Second time, sounds like that wasn't a one-off, you have toil, and you need a runbook on the company wiki or internal doc site - that way anyone in the team (or organisation or company), can quickly step in and do that thing. Third time, have somebody else run through the runbook it, validate it, figure out if it can be improved. Perhaps have some better alarms and operational oversight. Now, start automating parts of the runbook. Some parts will be easy, some will be hard. Do the easy ones first and update the runbook. Automation will need alarms with the right thresholds. Think about those carefully and remember whenever an alarm fires there are only three things you should be doing: fix the problem; change the threshold; delete the alarm. If you do anything else - especially ignore it and just look at things and wait to see if anything else happens - you have misunderstood and are heading towards alarm fatigue. Don't do this. Ever. Remember the three possible actions. Once all the parts of the runbook are automated and alarms are working correctly to trigger automatic resolution, delete the runbook. You have now eliminated toil. This can apply to any manual operational task that needs to happen anywhere in a business, but is most easily used when improving resiliency and durability in systems that have easy automatic resolution (such as horizontal scaling by adding more instances when loads are high on existing infra). It's exactly what an AWS EC2 ASG is there to do, for example.