6 ms·

> I don't understand how that is possible? Could you give an example of a task that can't be automated? "Check the logs for service X (they're here <link>) and

> I don't understand how that is possible? Could you give an example of a task that can't be automated? "Check the logs for service X (they're here <link>) and look for anything related to the issue" "If the user impact is high, write an update to the status page detailing the impact and an estimated time to recovery" The value of a runbook is that it can make use of human intelligence in its steps. No-one is arguing that you shouldn't be automating things like "if the CPU usage is > 90%, spin up another instance and configure it".

jedberg 10y ago
> "Check the logs for service X (they're here <link>) and look for anything related to the issue" I have a long missive about how logs are useless and shouldn't be kept, but that's for another time. I'll summarize by saying that if you have to look at logs, then your monitoring has failed you. > If the user impact is high, write an update to the status page detailing the impact and an estimated time to recovery" I guess technically that would be a step in a runbook, that's fair. Although in my case that was left to PR to do based on updates to the trouble tickets. :) > The value of a runbook is that it can make use of human intelligence in its steps I'd rather human intelligence be spent on triage by reading the results of automated diagnosis and coding up remediation software than on repeating steps in a checklist. Sure, there are uses for checklists of things to check, but even that should be automated through the ticket system at the very least, which I no longer consider a runbook, but I guess some might still consider that a runbook.
- icebraining 10y ago
  I have a long missive about how logs are useless and shouldn't be kept, but that's for another time. I'll summarize by saying that if you have to look at logs, then your monitoring has failed you. How does that work?
  - jedberg 10y ago
    First we have to make the distinction between logs and metrics. Logs are unstructured or loosely structured text, whereas metrics are discrete items that can be put into a time series database. If you emit metrics as necessary to a time series database, then you should be able to build alerting based on the time series metrics. Your monitoring systems should be good at building alerts based on a stream of metrics and visualizing the time series data. Sometimes you might have to look at the visualizations to find something, but ideally you then set up an alert on the thing you looked at so you have the alert for the next time it happens. A great monitoring system lets you turn graphs into alerts right in the interface, so if you're looking at a useful graph you can make an alert out of it. Sometimes logs can be useful, but only after your monitoring system has told you which system is not behaving, and then you can turn on logs for that system until you've solved the problem, but you shouldn't need access to old logs, because if the problem was only in the past, then it's not really a problem anymore, right? If you have an ongoing problem, then maybe have the logs on for that service while you're investing that problem, but then turn them off again. But having a ton of logs always generating and being stored tend to be fairly useless in practice with a good time series database at hand.
    - spyrosg 10y ago
      Given the low cost of long-term storage, if you have any auditing requirement, it's worth it to store lots of logs.
      - jedberg 10y ago
        What about the cost of moving the logs to storage, and the infrastructure required to move them around and put them in storage? Especially if you have a micro services architecture. Also the cost of the infrastructure to search the logs and view the logs.
        Zalastax 10y ago
        Remember that there are different sizes of deployments. For sites like Reddit or Netflix, logs becomes a burden since there's so much data. For smaller deployments it's possible that all logs can be aggregated on a single machine quite easily, and aggregating the logs is far more enjoyable than SSH:ing to each separate machine. > Sometimes logs can be useful, but only after your monitoring system has told you which system is not behaving, and then you can turn on logs for that system until you've solved the problem, but you shouldn't need access to old logs, because if the problem was only in the past, then it's not really a problem anymore, right? Some things happen rarely, but can still have large impact. E.g. Imagine a once a day job of moving files which fails twice a month, rendering those files inaccessible.
- deskamess 10y ago
  Automated diagnosis sounds like a time saver... What is your source material for automated diagnosis? i.e., how was it trained? Are there cases where an automated diagnosis could not be made for an incident and if so was manual recourse possible? How would you retrain the 'diagnosing app' to handle the new case?
  - jedberg 10y ago
    > What is your source material for automated diagnosis? i.e., how was it trained? Incident reviews. If something happened that wasn't covered, then it is added as an outcome of the incident review. > Are there cases where an automated diagnosis could not be made for an incident and if so was manual recourse possible? For sure. Manual recourse was to dig in and figure it out either with the command line or the monitoring system or whatever else. > How would you retrain the 'diagnosing app' to handle the new case? In most cases the "diagnosing app" was a dashboard on the monitoring system, with a set of relevant graphs, so you would add a new graph. There was also a tool that correlated graphs, so you could add a new graph and correlation.
- always_good 10y ago
  > I have a long missive about how logs are useless > and shouldn't be kept, but that's for another time. > I'll summarize by saying that if you have to look > at logs, then your monitoring has failed you. Eh, I'm extremely wary of people who think they have a silver bullet to everything and can speak in absolutes like this. I have to say, you have a tendency on HN to chime in from the peanut gallery and be a bit unrelenting and even combative because jedberg does things differently.
  - jedberg 10y ago
    > I have to say, you have a tendency on HN to chime in from the peanut gallery and be a bit unrelenting and even combative because jedberg does things differently. That's a fair critique, and thank you for pointing it out. I try to always back up what I say with the reasons for what I say, but sometimes I get lazy or don't have time to write it all out. I too worry about folks who speak in absolutes, although in this case I happen to actually believe it. The medium isn't always the best way to have a deep technical discussion unfortunately.

jedberg 10y ago

> "Check the logs for service X (they're here <link>) and look for anything related to the issue" I have a long missive about how logs are useless and shouldn't be kept, but that's for another time. I'll summarize by saying that if you have to look at logs, then your monitoring has failed you. > If the user impact is high, write an update to the status page detailing the impact and an estimated time to recovery" I guess technically that would be a step in a runbook, that's fair. Although in my case that was left to PR to do based on updates to the trouble tickets. :) > The value of a runbook is that it can make use of human intelligence in its steps I'd rather human intelligence be spent on triage by reading the results of automated diagnosis and coding up remediation software than on repeating steps in a checklist. Sure, there are uses for checklists of things to check, but even that should be automated through the ticket system at the very least, which I no longer consider a runbook, but I guess some might still consider that a runbook.

icebraining 10y ago
I have a long missive about how logs are useless and shouldn't be kept, but that's for another time. I'll summarize by saying that if you have to look at logs, then your monitoring has failed you. How does that work?
- jedberg 10y ago
  First we have to make the distinction between logs and metrics. Logs are unstructured or loosely structured text, whereas metrics are discrete items that can be put into a time series database. If you emit metrics as necessary to a time series database, then you should be able to build alerting based on the time series metrics. Your monitoring systems should be good at building alerts based on a stream of metrics and visualizing the time series data. Sometimes you might have to look at the visualizations to find something, but ideally you then set up an alert on the thing you looked at so you have the alert for the next time it happens. A great monitoring system lets you turn graphs into alerts right in the interface, so if you're looking at a useful graph you can make an alert out of it. Sometimes logs can be useful, but only after your monitoring system has told you which system is not behaving, and then you can turn on logs for that system until you've solved the problem, but you shouldn't need access to old logs, because if the problem was only in the past, then it's not really a problem anymore, right? If you have an ongoing problem, then maybe have the logs on for that service while you're investing that problem, but then turn them off again. But having a ton of logs always generating and being stored tend to be fairly useless in practice with a good time series database at hand.
  - spyrosg 10y ago
    Given the low cost of long-term storage, if you have any auditing requirement, it's worth it to store lots of logs.
    - jedberg 10y ago
      What about the cost of moving the logs to storage, and the infrastructure required to move them around and put them in storage? Especially if you have a micro services architecture. Also the cost of the infrastructure to search the logs and view the logs.
      - Zalastax 10y ago
        Remember that there are different sizes of deployments. For sites like Reddit or Netflix, logs becomes a burden since there's so much data. For smaller deployments it's possible that all logs can be aggregated on a single machine quite easily, and aggregating the logs is far more enjoyable than SSH:ing to each separate machine. > Sometimes logs can be useful, but only after your monitoring system has told you which system is not behaving, and then you can turn on logs for that system until you've solved the problem, but you shouldn't need access to old logs, because if the problem was only in the past, then it's not really a problem anymore, right? Some things happen rarely, but can still have large impact. E.g. Imagine a once a day job of moving files which fails twice a month, rendering those files inaccessible.
deskamess 10y ago
Automated diagnosis sounds like a time saver... What is your source material for automated diagnosis? i.e., how was it trained? Are there cases where an automated diagnosis could not be made for an incident and if so was manual recourse possible? How would you retrain the 'diagnosing app' to handle the new case?
- jedberg 10y ago
  > What is your source material for automated diagnosis? i.e., how was it trained? Incident reviews. If something happened that wasn't covered, then it is added as an outcome of the incident review. > Are there cases where an automated diagnosis could not be made for an incident and if so was manual recourse possible? For sure. Manual recourse was to dig in and figure it out either with the command line or the monitoring system or whatever else. > How would you retrain the 'diagnosing app' to handle the new case? In most cases the "diagnosing app" was a dashboard on the monitoring system, with a set of relevant graphs, so you would add a new graph. There was also a tool that correlated graphs, so you could add a new graph and correlation.
always_good 10y ago
> I have a long missive about how logs are useless > and shouldn't be kept, but that's for another time. > I'll summarize by saying that if you have to look > at logs, then your monitoring has failed you. Eh, I'm extremely wary of people who think they have a silver bullet to everything and can speak in absolutes like this. I have to say, you have a tendency on HN to chime in from the peanut gallery and be a bit unrelenting and even combative because jedberg does things differently.
- jedberg 10y ago
  > I have to say, you have a tendency on HN to chime in from the peanut gallery and be a bit unrelenting and even combative because jedberg does things differently. That's a fair critique, and thank you for pointing it out. I try to always back up what I say with the reasons for what I say, but sometimes I get lazy or don't have time to write it all out. I too worry about folks who speak in absolutes, although in this case I happen to actually believe it. The medium isn't always the best way to have a deep technical discussion unfortunately.