5 ms·

'Documentation is generally a waste of time unless you have a very static infrastructure' I definitely agree with that, and it's partly a corollary of 'documen

'Documentation is generally a waste of time unless you have a very static infrastructure' I definitely agree with that, and it's partly a corollary of 'documentation is expensive and requires costly maintenance'. Run books/checklists are mostly implemented really really badly. Automation is the ideal, but is costly, and itself requires maintenance. Most of the steps we had to perform did not lend themselves to automation, also.

jedberg 10y ago
> Automation is the ideal, but is costly, and itself requires maintenance. I would contend that the cost of automation is about the same as the cost of documentation plus the cost of having to manually do the work over an over. It's just a cost borne up front instead of over time. But to your point in the article, you have to have a culture that supports bearing that up front cost. > Most of the steps we had to perform did not lend themselves to automation, also. I don't understand how that is possible? Could you give an example of a task that can't be automated?
- zwischenzug 10y ago
  Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. Most attempts I've seen to automate tasks flounder for this reason. Also, there's no point automating something that happens once a year - the cost will exceed the benefit. Hence the comments about decided which tasks to automate with a backlog and metrics. Most anything _could_ be automated, eg 'Phone the customer and ask them to engage their network team', 'look for similar-looking strings', 'tcpdump the port, and see if the output is irregular', but is it really worth the effort to? That's where the backlog came in. BTW I wrote a whole framework for automation ianmiell.github.io/shutit/ and it's taught me a lot about the hidden costs of automation...
  - jedberg 10y ago
    > Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. Most attempts I've seen to automate tasks flounder for this reason. If everything in your infrastructure is deployed with code, then automation is simply the act of making sure the infrastructure matches the deployment described in the code. It's true that automation is just as difficult if deployments are manual. Then remediation becomes changing the deployment code to fix a previously unknown problem, instead of manually fixing a problem. This then gains you the advantage of the problem being fixed in perpetuity. So they aren't in contradiction if your remediation and deployment are the same process, because then it by definition is always up to date. I guess I should add the caveat that deployment should be quick enough to solve problems via redeployment.
    - geofft 10y ago
      Most of the environments I've worked at have been so far from being deployable with code that it would take you significant time and expense to get there, and in the meantime, you need to keep everything working. You need a reliability engineering team that knows how to triage problems and keep the existing system working while they automate what is feasible to automate. I'd probably agree that this is a separate mindset and skillset from keeping an infrastructure-as-code deployment running, and it's unfortunate that we don't have good terms to differentiate them. A lot of the SRE mindset applies, but not all of it. For instance, if you've got machines at customer sites where you manage the software, commercial filers or appliances where you don't really have a shell and you certainly can't spin up a VM, etc., your machines aren't fungible. You can't just redeploy a machine, and just about every problem on the machine is going to be a new one. You want the runbook so that it enumerates how to stop and start things around a human intervention (and yeah, as much of that as possible should be automated), but you can't automate the whole thing without changing your architecture. If you have the option of starting with an architecture where you don't have these problems, by all means take that option! Maybe do cloud-hosted SaaS instead of maintaining on-prem software for customers, or use some fancy cloud storage API instead of a physical old-school filer. But people who have the old-school architectures need to keep things running smoothly, too.
    - 10y ago
  - munro 10y ago
    > Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. OP said automating runbooks, and any fixes to production--so the code becomes the documentation of how to deploy, and what exactly was done to remedy an error. OP did not say automating failure responses. Which brings up OP's proposed question... >> I don't understand how that is possible? Could you give an example of a task that can't be automated? Could you give a concrete example? There's a vague reference in the article about automating responses for encoding errors which sounds interesting. It sounds like the system generates lots of server errors, and it's easier/cheaper to communicate them to customers directly instead of making code fixes.
- ekimekim 10y ago
  > I don't understand how that is possible? Could you give an example of a task that can't be automated? "Check the logs for service X (they're here <link>) and look for anything related to the issue" "If the user impact is high, write an update to the status page detailing the impact and an estimated time to recovery" The value of a runbook is that it can make use of human intelligence in its steps. No-one is arguing that you shouldn't be automating things like "if the CPU usage is > 90%, spin up another instance and configure it".
  - jedberg 10y ago
    > "Check the logs for service X (they're here <link>) and look for anything related to the issue" I have a long missive about how logs are useless and shouldn't be kept, but that's for another time. I'll summarize by saying that if you have to look at logs, then your monitoring has failed you. > If the user impact is high, write an update to the status page detailing the impact and an estimated time to recovery" I guess technically that would be a step in a runbook, that's fair. Although in my case that was left to PR to do based on updates to the trouble tickets. :) > The value of a runbook is that it can make use of human intelligence in its steps I'd rather human intelligence be spent on triage by reading the results of automated diagnosis and coding up remediation software than on repeating steps in a checklist. Sure, there are uses for checklists of things to check, but even that should be automated through the ticket system at the very least, which I no longer consider a runbook, but I guess some might still consider that a runbook.
    - icebraining 10y ago
      I have a long missive about how logs are useless and shouldn't be kept, but that's for another time. I'll summarize by saying that if you have to look at logs, then your monitoring has failed you. How does that work?
      - jedberg 10y ago
        First we have to make the distinction between logs and metrics. Logs are unstructured or loosely structured text, whereas metrics are discrete items that can be put into a time series database. If you emit metrics as necessary to a time series database, then you should be able to build alerting based on the time series metrics. Your monitoring systems should be good at building alerts based on a stream of metrics and visualizing the time series data. Sometimes you might have to look at the visualizations to find something, but ideally you then set up an alert on the thing you looked at so you have the alert for the next time it happens. A great monitoring system lets you turn graphs into alerts right in the interface, so if you're looking at a useful graph you can make an alert out of it. Sometimes logs can be useful, but only after your monitoring system has told you which system is not behaving, and then you can turn on logs for that system until you've solved the problem, but you shouldn't need access to old logs, because if the problem was only in the past, then it's not really a problem anymore, right? If you have an ongoing problem, then maybe have the logs on for that service while you're investing that problem, but then turn them off again. But having a ton of logs always generating and being stored tend to be fairly useless in practice with a good time series database at hand.

jedberg 10y ago

> Automation is the ideal, but is costly, and itself requires maintenance. I would contend that the cost of automation is about the same as the cost of documentation plus the cost of having to manually do the work over an over. It's just a cost borne up front instead of over time. But to your point in the article, you have to have a culture that supports bearing that up front cost. > Most of the steps we had to perform did not lend themselves to automation, also. I don't understand how that is possible? Could you give an example of a task that can't be automated?

zwischenzug 10y ago
Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. Most attempts I've seen to automate tasks flounder for this reason. Also, there's no point automating something that happens once a year - the cost will exceed the benefit. Hence the comments about decided which tasks to automate with a backlog and metrics. Most anything _could_ be automated, eg 'Phone the customer and ask them to engage their network team', 'look for similar-looking strings', 'tcpdump the port, and see if the output is irregular', but is it really worth the effort to? That's where the backlog came in. BTW I wrote a whole framework for automation ianmiell.github.io/shutit/ and it's taught me a lot about the hidden costs of automation...
- jedberg 10y ago
  > Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. Most attempts I've seen to automate tasks flounder for this reason. If everything in your infrastructure is deployed with code, then automation is simply the act of making sure the infrastructure matches the deployment described in the code. It's true that automation is just as difficult if deployments are manual. Then remediation becomes changing the deployment code to fix a previously unknown problem, instead of manually fixing a problem. This then gains you the advantage of the problem being fixed in perpetuity. So they aren't in contradiction if your remediation and deployment are the same process, because then it by definition is always up to date. I guess I should add the caveat that deployment should be quick enough to solve problems via redeployment.
  - geofft 10y ago
    Most of the environments I've worked at have been so far from being deployable with code that it would take you significant time and expense to get there, and in the meantime, you need to keep everything working. You need a reliability engineering team that knows how to triage problems and keep the existing system working while they automate what is feasible to automate. I'd probably agree that this is a separate mindset and skillset from keeping an infrastructure-as-code deployment running, and it's unfortunate that we don't have good terms to differentiate them. A lot of the SRE mindset applies, but not all of it. For instance, if you've got machines at customer sites where you manage the software, commercial filers or appliances where you don't really have a shell and you certainly can't spin up a VM, etc., your machines aren't fungible. You can't just redeploy a machine, and just about every problem on the machine is going to be a new one. You want the runbook so that it enumerates how to stop and start things around a human intervention (and yeah, as much of that as possible should be automated), but you can't automate the whole thing without changing your architecture. If you have the option of starting with an architecture where you don't have these problems, by all means take that option! Maybe do cloud-hosted SaaS instead of maintaining on-prem software for customers, or use some fancy cloud storage API instead of a physical old-school filer. But people who have the old-school architectures need to keep things running smoothly, too.
  - 10y ago
- munro 10y ago
  > Your arguments contradict themselves - if the infrastructure is changing then there's little point spending the time automating your response to its failure. OP said automating runbooks, and any fixes to production--so the code becomes the documentation of how to deploy, and what exactly was done to remedy an error. OP did not say automating failure responses. Which brings up OP's proposed question... >> I don't understand how that is possible? Could you give an example of a task that can't be automated? Could you give a concrete example? There's a vague reference in the article about automating responses for encoding errors which sounds interesting. It sounds like the system generates lots of server errors, and it's easier/cheaper to communicate them to customers directly instead of making code fixes.
ekimekim 10y ago
> I don't understand how that is possible? Could you give an example of a task that can't be automated? "Check the logs for service X (they're here <link>) and look for anything related to the issue" "If the user impact is high, write an update to the status page detailing the impact and an estimated time to recovery" The value of a runbook is that it can make use of human intelligence in its steps. No-one is arguing that you shouldn't be automating things like "if the CPU usage is > 90%, spin up another instance and configure it".
- jedberg 10y ago
  > "Check the logs for service X (they're here <link>) and look for anything related to the issue" I have a long missive about how logs are useless and shouldn't be kept, but that's for another time. I'll summarize by saying that if you have to look at logs, then your monitoring has failed you. > If the user impact is high, write an update to the status page detailing the impact and an estimated time to recovery" I guess technically that would be a step in a runbook, that's fair. Although in my case that was left to PR to do based on updates to the trouble tickets. :) > The value of a runbook is that it can make use of human intelligence in its steps I'd rather human intelligence be spent on triage by reading the results of automated diagnosis and coding up remediation software than on repeating steps in a checklist. Sure, there are uses for checklists of things to check, but even that should be automated through the ticket system at the very least, which I no longer consider a runbook, but I guess some might still consider that a runbook.
  - icebraining 10y ago
    I have a long missive about how logs are useless and shouldn't be kept, but that's for another time. I'll summarize by saying that if you have to look at logs, then your monitoring has failed you. How does that work?
    - jedberg 10y ago
      First we have to make the distinction between logs and metrics. Logs are unstructured or loosely structured text, whereas metrics are discrete items that can be put into a time series database. If you emit metrics as necessary to a time series database, then you should be able to build alerting based on the time series metrics. Your monitoring systems should be good at building alerts based on a stream of metrics and visualizing the time series data. Sometimes you might have to look at the visualizations to find something, but ideally you then set up an alert on the thing you looked at so you have the alert for the next time it happens. A great monitoring system lets you turn graphs into alerts right in the interface, so if you're looking at a useful graph you can make an alert out of it. Sometimes logs can be useful, but only after your monitoring system has told you which system is not behaving, and then you can turn on logs for that system until you've solved the problem, but you shouldn't need access to old logs, because if the problem was only in the past, then it's not really a problem anymore, right? If you have an ongoing problem, then maybe have the logs on for that service while you're investing that problem, but then turn them off again. But having a ton of logs always generating and being stored tend to be fairly useless in practice with a good time series database at hand.