4 ms·

Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. M

Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. Most attempts I've seen to automate tasks flounder for this reason. Also, there's no point automating something that happens once a year - the cost will exceed the benefit. Hence the comments about decided which tasks to automate with a backlog and metrics. Most anything _could_ be automated, eg 'Phone the customer and ask them to engage their network team', 'look for similar-looking strings', 'tcpdump the port, and see if the output is irregular', but is it really worth the effort to? That's where the backlog came in. BTW I wrote a whole framework for automation ianmiell.github.io/shutit/ and it's taught me a lot about the hidden costs of automation...

jedberg 10y ago
> Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. Most attempts I've seen to automate tasks flounder for this reason. If everything in your infrastructure is deployed with code, then automation is simply the act of making sure the infrastructure matches the deployment described in the code. It's true that automation is just as difficult if deployments are manual. Then remediation becomes changing the deployment code to fix a previously unknown problem, instead of manually fixing a problem. This then gains you the advantage of the problem being fixed in perpetuity. So they aren't in contradiction if your remediation and deployment are the same process, because then it by definition is always up to date. I guess I should add the caveat that deployment should be quick enough to solve problems via redeployment.
- geofft 10y ago
  Most of the environments I've worked at have been so far from being deployable with code that it would take you significant time and expense to get there, and in the meantime, you need to keep everything working. You need a reliability engineering team that knows how to triage problems and keep the existing system working while they automate what is feasible to automate. I'd probably agree that this is a separate mindset and skillset from keeping an infrastructure-as-code deployment running, and it's unfortunate that we don't have good terms to differentiate them. A lot of the SRE mindset applies, but not all of it. For instance, if you've got machines at customer sites where you manage the software, commercial filers or appliances where you don't really have a shell and you certainly can't spin up a VM, etc., your machines aren't fungible. You can't just redeploy a machine, and just about every problem on the machine is going to be a new one. You want the runbook so that it enumerates how to stop and start things around a human intervention (and yeah, as much of that as possible should be automated), but you can't automate the whole thing without changing your architecture. If you have the option of starting with an architecture where you don't have these problems, by all means take that option! Maybe do cloud-hosted SaaS instead of maintaining on-prem software for customers, or use some fancy cloud storage API instead of a physical old-school filer. But people who have the old-school architectures need to keep things running smoothly, too.
- eropple 10y ago
  FWIW, I agree with your overall notion of where one should go, but it's worth noting that you're describing an extremely optimistic world for probably 90% to 95% of shops. Which is not to say that it's a bad idea at all to get there, but incremental steps are important and they're what I feel are being described in this article (setting aside that that kind of pervasive automation wasn't really a thing outside of bleeding-edge shops at the time he was describing). Shameless plug for people who are feeling inferior while reading this thread: as it happens, my consulting gigs revolve pretty heavily around doing exactly what 'jedberg has described, and it's super worth getting to that point. ;) Email's in the profile if you're looking to get there.
munro 10y ago
> Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. OP said automating runbooks, and any fixes to production--so the code becomes the documentation of how to deploy, and what exactly was done to remedy an error. OP did not say automating failure responses. Which brings up OP's proposed question... >> I don't understand how that is possible? Could you give an example of a task that can't be automated? Could you give a concrete example? There's a vague reference in the article about automating responses for encoding errors which sounds interesting. It sounds like the system generates lots of server errors, and it's easier/cheaper to communicate them to customers directly instead of making code fixes.

jedberg 10y ago

> Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. Most attempts I've seen to automate tasks flounder for this reason. If everything in your infrastructure is deployed with code, then automation is simply the act of making sure the infrastructure matches the deployment described in the code. It's true that automation is just as difficult if deployments are manual. Then remediation becomes changing the deployment code to fix a previously unknown problem, instead of manually fixing a problem. This then gains you the advantage of the problem being fixed in perpetuity. So they aren't in contradiction if your remediation and deployment are the same process, because then it by definition is always up to date. I guess I should add the caveat that deployment should be quick enough to solve problems via redeployment.

geofft 10y ago
Most of the environments I've worked at have been so far from being deployable with code that it would take you significant time and expense to get there, and in the meantime, you need to keep everything working. You need a reliability engineering team that knows how to triage problems and keep the existing system working while they automate what is feasible to automate. I'd probably agree that this is a separate mindset and skillset from keeping an infrastructure-as-code deployment running, and it's unfortunate that we don't have good terms to differentiate them. A lot of the SRE mindset applies, but not all of it. For instance, if you've got machines at customer sites where you manage the software, commercial filers or appliances where you don't really have a shell and you certainly can't spin up a VM, etc., your machines aren't fungible. You can't just redeploy a machine, and just about every problem on the machine is going to be a new one. You want the runbook so that it enumerates how to stop and start things around a human intervention (and yeah, as much of that as possible should be automated), but you can't automate the whole thing without changing your architecture. If you have the option of starting with an architecture where you don't have these problems, by all means take that option! Maybe do cloud-hosted SaaS instead of maintaining on-prem software for customers, or use some fancy cloud storage API instead of a physical old-school filer. But people who have the old-school architectures need to keep things running smoothly, too.
eropple 10y ago
FWIW, I agree with your overall notion of where one should go, but it's worth noting that you're describing an extremely optimistic world for probably 90% to 95% of shops. Which is not to say that it's a bad idea at all to get there, but incremental steps are important and they're what I feel are being described in this article (setting aside that that kind of pervasive automation wasn't really a thing outside of bleeding-edge shops at the time he was describing). Shameless plug for people who are feeling inferior while reading this thread: as it happens, my consulting gigs revolve pretty heavily around doing exactly what 'jedberg has described, and it's super worth getting to that point. ;) Email's in the profile if you're looking to get there.

munro 10y ago

> Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. OP said automating runbooks, and any fixes to production--so the code becomes the documentation of how to deploy, and what exactly was done to remedy an error. OP did not say automating failure responses. Which brings up OP's proposed question... >> I don't understand how that is possible? Could you give an example of a task that can't be automated? Could you give a concrete example? There's a vague reference in the article about automating responses for encoding errors which sounds interesting. It sounds like the system generates lots of server errors, and it's easier/cheaper to communicate them to customers directly instead of making code fixes.