4 ms·

Really? We might be working in entirely different spheres and cultures, but I tend to look at (recently historical) logs pretty frequently for a bunch of reason

Really? We might be working in entirely different spheres and cultures, but I tend to look at (recently historical) logs pretty frequently for a bunch of reasons: helping to isolate infrequently encountered and hard to reproduce bugs, tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour, identifying whether a particular code path is actually in use and if so how heavily, etc etc. They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment! Edit: just realised who I was replying to. Netflix and Reddit might be very different environments with very different reliability requirements to a lot of other types of software.

jedberg 4y ago
> helping to isolate infrequently encountered and hard to reproduce bugs As I mentioned in my post, that's the one case where I find value in temporary turning on logging, but having old logs around is awfully costly for the off chance that they help. > tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour, That's request tracing, which is different. Just storing a request trace is useful, but that's not what I mean by logging. I'm talking about server application logs. > identifying whether a particular code path is actually in use and if so how heavily Again that would be request tracing, not application logging. > They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment! If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected. If they are affected, then your logs will keep showing it.
- idunno246 4y ago
  > Again that would be request tracing, not application logging. from what ive seen people who rely on logs do not have proper request tracing and/or metrics. its much simpler to dump a bunch of unformatted junk to stdout and throw elasticsearch at it, but not a good experience. i agree overall, i find logs to be a last resort with really bad signal to noise ratio.
- mjfisher 4y ago
  Gotcha. I think we're probably thinking about different kinds of environments - probably the key differentiation is the total volume of logs, the kinds of services we're monitoring, and how bad a given error is. > Having old logs around is awfully costly for the off chance that they help. Depends massively on the log volume. I've worked at places that generated terrabytes of logs over the course of a few days, and required very large ES clusters to try to keep on top of it. There was definitely value in having those logs, but it only made sense to retain them for a couple of weeks. It's much less costly for low volume services. > ...request tracing, which is different... Ye-es, until it's not. If your current setup isn't yet developed or grown enough to have full request tracing, you can get a long way filtering aggregated logs by a correlation ID. I know it's no substitute for a fully feature tracing system, but in situations where you have limited resourcing you need to prioritise it can get you a long way > If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected. That massively depends on the impact of the incident, I think. If it's a "There was a problem playing this title, please try again" kind of deal, it's no biggie. If it's a "my broadcast TV channel is off air" or "my financial system has crashed in an unexpected way trying to reconcile this transaction" type situation, you might be a little more worried - and the customer is likely going to be very upset for quite a while, and some people with very serious faces might come to talk to you about very large fines.

jedberg 4y ago

> helping to isolate infrequently encountered and hard to reproduce bugs As I mentioned in my post, that's the one case where I find value in temporary turning on logging, but having old logs around is awfully costly for the off chance that they help. > tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour, That's request tracing, which is different. Just storing a request trace is useful, but that's not what I mean by logging. I'm talking about server application logs. > identifying whether a particular code path is actually in use and if so how heavily Again that would be request tracing, not application logging. > They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment! If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected. If they are affected, then your logs will keep showing it.

idunno246 4y ago
> Again that would be request tracing, not application logging. from what ive seen people who rely on logs do not have proper request tracing and/or metrics. its much simpler to dump a bunch of unformatted junk to stdout and throw elasticsearch at it, but not a good experience. i agree overall, i find logs to be a last resort with really bad signal to noise ratio.
mjfisher 4y ago
Gotcha. I think we're probably thinking about different kinds of environments - probably the key differentiation is the total volume of logs, the kinds of services we're monitoring, and how bad a given error is. > Having old logs around is awfully costly for the off chance that they help. Depends massively on the log volume. I've worked at places that generated terrabytes of logs over the course of a few days, and required very large ES clusters to try to keep on top of it. There was definitely value in having those logs, but it only made sense to retain them for a couple of weeks. It's much less costly for low volume services. > ...request tracing, which is different... Ye-es, until it's not. If your current setup isn't yet developed or grown enough to have full request tracing, you can get a long way filtering aggregated logs by a correlation ID. I know it's no substitute for a fully feature tracing system, but in situations where you have limited resourcing you need to prioritise it can get you a long way > If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected. That massively depends on the impact of the incident, I think. If it's a "There was a problem playing this title, please try again" kind of deal, it's no biggie. If it's a "my broadcast TV channel is off air" or "my financial system has crashed in an unexpected way trying to reconcile this transaction" type situation, you might be a little more worried - and the customer is likely going to be very upset for quite a while, and some people with very serious faces might come to talk to you about very large fines.