5 ms·

Logs have a much, much, much lower barrier to entry than a fully-complete time-series monitoring system that covers everything. Likewise, turning logs on only

by vacri 10y ago

Logs have a much, much, much lower barrier to entry than a fully-complete time-series monitoring system that covers everything. Likewise, turning logs on only after you've seen a problem means you miss out on troubleshooting the root cause of it - if there was a spike of badness this morning but you don't have logs for it, you're missing out on diagnostic information that may have protected you from repeats of that spike in future. I've also had business guys want to analyse things like access logs in ways that they didn't know previously. Logs provide a datastore of historical activity, which in smaller shops is a cheap data lake. Perhaps the 'no logs' thing works for your setup, but I think it's bad general advice. And your position is not that logs are useless ("turn on logs for that system until you've solved the problem"), but that retaining logs are useless - quite a significant difference between the two.

jedberg 10y ago
> And your position is not that logs are useless ("turn on logs for that system until you've solved the problem"), but that retaining logs are useless - quite a significant difference between the two. That's an important distinction, one that I agree with, and I should make clearer. Logs do have a purpose, but I'm not sure that retaining them does. Sure, for a very small shop, throw them on a disk, use awk, sed, grep, and perl to look through them, and call it a day. But once you get to the point of "spinning up a cluster of log servers" or something like it, I'd say you're probably better off investing in monitoring instead.
- jsmthrowaway 10y ago
  Having a corpus of live "test" requests for rebuilding an API is a compelling enough use case to justify infinite retention, in my opinion. Perhaps relevantly to your prior gig: "did we handle that weird Sony Bravia API call from five years ago in this redesign? Yes we did." I'm implying a little more structure to logging, though, because it sounds like you're thinking about random strings emitted from binaries (which I think are also approachable, to an extent). This overlaps with PII and such, though, and there is no one-size-fits-all answer. More than once I've run the entire corpus of requests to a system, ever, through a dummy rebuild as a pretty great integration test. It's a powerful SRE tool. Spelunking through all historical data is just icing on that cake, honestly. As the author says, SRE is basically just an information factory; I'd be haaaaaard pressed to agree with you on throwing away a lot of information -- you don't know what you don't know until you want to know it -- and betting all-in on monitoring. Retaining logs is not the hardest problem SRE deals with, either, but SREs turn around and force unrealistic latency requirements on the query side (I see a lot of ELK deploys running into this). You have to look at it as an Oracle. Oh, great Oracle of a pile of meaningless logs, cook off this map/reduce and tell me an interesting number that I can put in a Keynote for executives. Definitely not dashboarding from logs data. That's an impedance mismatch that Google gets away with because of the nature of their logging.
  - user5994461 10y ago
    Splunking is cool. Nonetheless, splunk is the most expensive software license on the planet. More than Oracle, yes.
    - jsmthrowaway 10y ago
      I meant spelunking, where they got their name. But yes, you're not wrong. (I've heard of $millions.)
- zwischenzug 10y ago
  Some envs require logs to be kept for regulatory purposes. Interestingly, we never used a cluster of log servers. I was always skeptical of their utility. It was grep, plus some hand-rolled utility scripts to interrogate. One was a thing of beauty I spent years on, which saved us a ton of time.
user5994461 10y ago
> Logs have a much, much, much lower barrier to entry than a fully-complete time-series monitoring system that covers everything. A monitoring system has a lower barrier to entry. http://datadoghq.com/ http://datadoghq.com/ => will do ALL of that and much more. You can deploy it in a few hours to thousands of hosts, no problem. Direct competitor: http://signalfx.com/ http://signalfx.com/ Have no money to pay for high quality tool? graphite + statsd will do the trick for basic infrastructure. However it's single host, doesn't scale and only basic ugly graphs are supported.
- vacri 10y ago
  Datadog for thousands of hosts costs tens of thousands of dollars per month. That is not a low barrier. But if you're arguing for external vendors, then Papertrail is to logs what Datadog is to metrics. And having tried both Papertrail is easier and quicker to set up (though Datadog isn't difficult). Similarly, basic infrastructure health is not giving you the same sort of information (what the software is actually doing) that logging does. In order to do time-series monitoring of your software rather than your system, you need to spend time thinking about what metrics you need to track and how you're going to obtain them. I run both an ELK stack and a Prometheus stack, and I find they're good for different things.
  - user5994461 10y ago
    thousands of hosts costs hundreds of thousands of dollars. a few thousands more is a low barrier. Sumologic, papertrail, logentries (and many more) for cloud logs. Graylog or ELK or Splunk for self hosted logs. However, logs should never be send to the cloud, they are too sensitive information to outsource. Server metrics + stats are more reasonable. Agree, stats and logs cover different things. Need both.
    - otisg 10y ago
      Both metrics and logs are needed for the full picture (plus alerts and a few other things, like RUM and error/exception capture, both on front-end and backend). Saying that logs should never be sent to the cloud is overly absolute. Some logs should indeed stay behind firewall, but lots of organizations have logs that can be shipped out to services whose features derive all kinds of interesting insights from logs.

by vacri 10y ago

jedberg 10y ago

> And your position is not that logs are useless ("turn on logs for that system until you've solved the problem"), but that retaining logs are useless - quite a significant difference between the two. That's an important distinction, one that I agree with, and I should make clearer. Logs do have a purpose, but I'm not sure that retaining them does. Sure, for a very small shop, throw them on a disk, use awk, sed, grep, and perl to look through them, and call it a day. But once you get to the point of "spinning up a cluster of log servers" or something like it, I'd say you're probably better off investing in monitoring instead.

jsmthrowaway 10y ago
Having a corpus of live "test" requests for rebuilding an API is a compelling enough use case to justify infinite retention, in my opinion. Perhaps relevantly to your prior gig: "did we handle that weird Sony Bravia API call from five years ago in this redesign? Yes we did." I'm implying a little more structure to logging, though, because it sounds like you're thinking about random strings emitted from binaries (which I think are also approachable, to an extent). This overlaps with PII and such, though, and there is no one-size-fits-all answer. More than once I've run the entire corpus of requests to a system, ever, through a dummy rebuild as a pretty great integration test. It's a powerful SRE tool. Spelunking through all historical data is just icing on that cake, honestly. As the author says, SRE is basically just an information factory; I'd be haaaaaard pressed to agree with you on throwing away a lot of information -- you don't know what you don't know until you want to know it -- and betting all-in on monitoring. Retaining logs is not the hardest problem SRE deals with, either, but SREs turn around and force unrealistic latency requirements on the query side (I see a lot of ELK deploys running into this). You have to look at it as an Oracle. Oh, great Oracle of a pile of meaningless logs, cook off this map/reduce and tell me an interesting number that I can put in a Keynote for executives. Definitely not dashboarding from logs data. That's an impedance mismatch that Google gets away with because of the nature of their logging.
- user5994461 10y ago
  Splunking is cool. Nonetheless, splunk is the most expensive software license on the planet. More than Oracle, yes.
  - jsmthrowaway 10y ago
    I meant spelunking, where they got their name. But yes, you're not wrong. (I've heard of $millions.)
zwischenzug 10y ago
Some envs require logs to be kept for regulatory purposes. Interestingly, we never used a cluster of log servers. I was always skeptical of their utility. It was grep, plus some hand-rolled utility scripts to interrogate. One was a thing of beauty I spent years on, which saved us a ton of time.

user5994461 10y ago

> Logs have a much, much, much lower barrier to entry than a fully-complete time-series monitoring system that covers everything. A monitoring system has a lower barrier to entry. http://datadoghq.com/ http://datadoghq.com/ => will do ALL of that and much more. You can deploy it in a few hours to thousands of hosts, no problem. Direct competitor: http://signalfx.com/ http://signalfx.com/ Have no money to pay for high quality tool? graphite + statsd will do the trick for basic infrastructure. However it's single host, doesn't scale and only basic ugly graphs are supported.

vacri 10y ago
Datadog for thousands of hosts costs tens of thousands of dollars per month. That is not a low barrier. But if you're arguing for external vendors, then Papertrail is to logs what Datadog is to metrics. And having tried both Papertrail is easier and quicker to set up (though Datadog isn't difficult). Similarly, basic infrastructure health is not giving you the same sort of information (what the software is actually doing) that logging does. In order to do time-series monitoring of your software rather than your system, you need to spend time thinking about what metrics you need to track and how you're going to obtain them. I run both an ELK stack and a Prometheus stack, and I find they're good for different things.
- user5994461 10y ago
  thousands of hosts costs hundreds of thousands of dollars. a few thousands more is a low barrier. Sumologic, papertrail, logentries (and many more) for cloud logs. Graylog or ELK or Splunk for self hosted logs. However, logs should never be send to the cloud, they are too sensitive information to outsource. Server metrics + stats are more reasonable. Agree, stats and logs cover different things. Need both.
  - otisg 10y ago
    Both metrics and logs are needed for the full picture (plus alerts and a few other things, like RUM and error/exception capture, both on front-end and backend). Saying that logs should never be sent to the cloud is overly absolute. Some logs should indeed stay behind firewall, but lots of organizations have logs that can be shipped out to services whose features derive all kinds of interesting insights from logs.