3 ms·

I've been doing ops for a long time, and I can't remember the last time I looked at an historical log (edit: I'm referring to application logs specifically stor

I've been doing ops for a long time, and I can't remember the last time I looked at an historical log (edit: I'm referring to application logs specifically stored in a central location). When something is broken, I turn on logging and look at the logs in real-time. If something was broken in the past and no longer is, I turn on logging for that thing so that I have logs if it happens again. The only use I can think of for historical logs is security auditing. In which case there is only a subset of logs you really need to trace a breach and they don't need to be searchable online -- you can pull them back from cold storage if you need to check for an incident. Moving and storing logs and making them all instantly searchable is awfully expensive for little benefit. Edit2: Disclaimer: This opinion does not apply to government regulated services like financial services.

encoderer 4y ago
Counter-point is that when a customer reports a problem I don’t want to have to say “can you try that again after I enable logging”.
- jedberg 4y ago
  When a customer reports a problem the first thing I try is to reproduce it (with logging on). If I can't reproduce it, does it matter? I can leave logging on and let them know I can't reproduce their problem but if it happens again to let me know.
  - encoderer 4y ago
    In your background of consumer engineering, yes it probably does not matter. But it’s not the same in SaaS
    - jedberg 4y ago
      I've worked in SaaS for years, and I don't have any examples in my history where that's the case. Do you have an example of a case where historical logging was the only way to track down a problem?
      - encoderer 4y ago
        Yesterday I used logs to help a customer who had mis-configured their SAML SSO (the problem was on the IdP side). I use our database audit trail logs to understand the sequence of changes a customer makes that sometimes results in behavior that is surprising if you only look at the current state. We put a big emphasis on the second S in SaaS and we want a magical 1-email support experience whenever possible. Audit logs help this.
  - staticassertion 4y ago
    When do you turn the logs back off?
    - jedberg 4y ago
      When the problem is verified solved.
      - staticassertion 4y ago
        I can't imagine the logs are off very much of the time then
        jedberg 4y ago
        Well when you have microservices, hopefully only one or two of them need logging at a given time.
- macspoofing 4y ago
  You may have to. Debug logging can be very expensive to simply leave on in production.
  - falcolas 4y ago
    You can also prune logs based on level at different time intervals.
- carlmr 4y ago
  Exactly. Also sometimes you discover some issue and you want to see if this is never happened before, sometimes, often? Depends really how long your customers can wait.
  - jedberg 4y ago
    Logging is different than metrics. Metrics should tell you if a problem has happened in the past.
mjfisher 4y ago
Really? We might be working in entirely different spheres and cultures, but I tend to look at (recently historical) logs pretty frequently for a bunch of reasons: helping to isolate infrequently encountered and hard to reproduce bugs, tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour, identifying whether a particular code path is actually in use and if so how heavily, etc etc. They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment! Edit: just realised who I was replying to. Netflix and Reddit might be very different environments with very different reliability requirements to a lot of other types of software.
- jedberg 4y ago
  > helping to isolate infrequently encountered and hard to reproduce bugs As I mentioned in my post, that's the one case where I find value in temporary turning on logging, but having old logs around is awfully costly for the off chance that they help. > tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour, That's request tracing, which is different. Just storing a request trace is useful, but that's not what I mean by logging. I'm talking about server application logs. > identifying whether a particular code path is actually in use and if so how heavily Again that would be request tracing, not application logging. > They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment! If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected. If they are affected, then your logs will keep showing it.
  - idunno246 4y ago
    > Again that would be request tracing, not application logging. from what ive seen people who rely on logs do not have proper request tracing and/or metrics. its much simpler to dump a bunch of unformatted junk to stdout and throw elasticsearch at it, but not a good experience. i agree overall, i find logs to be a last resort with really bad signal to noise ratio.
closeparen 4y ago
As a software engineer responsible for building and operating various async and background workflows, I get a lot of inquiries like "I see cause X, why wasn't there side effect Y?" and I go and poke around the data to figure out what happened. Sometimes the cause didn't really happen, sometimes there are other criteria that the stakeholder wasn't thinking about that weren't met, sometimes the cause really happened but the data is missing or malformed upstream, and very occasionally my system errored out and dropped the ball. These are all deeply mysterious head-scratching issues if you don't have ready access to detailed logs. More mundanely, if I get paged about an error rate, the first thing I'm going to do is look at the logs to see what the errors are.
falcolas 4y ago
I, and my peers, look at historical logs daily. Our customer response team is often looking at them on a minute-by-minute basis, and sets up alerts on them. Our logs tell me the state of the application when the error occurred without having to reproduce it. They also tell me how many customers are having the issue, so we can set the severity of the incident at the appropriate level and reach out to those customers proactively if needed. Sure, we might be able to do this with metrics or tracing instead, but that's often just duplicating information we're already logging. For example, by adding a correlation ID in my log metadata, I suddenly have end-to-end request tracing. It's not as pretty as what New Relic can produce, but it definitely works.
- AishwaryaVenkat 4y ago
  Ofcourse, If your business is growing, chances are, you are collecting more and more logs. Your log management process is ultimately slowing down your troubleshooting. Atatus provides you with hassle-free log management delivered as a fully managed cloud service. You don't have to set up and maintain it. This instantly cuts costs, reduces IT load and improves both the speed of troubleshooting and resolution of issues.
Jugurtha 4y ago
We used Sentry[0] and it saved us a lot of time digging into lots of logs to find out some exception that went unnoticed. We still did rely on logs to troubleshoot and understand some issues. - [0]: https://sentry.io https://sentry.io

encoderer 4y ago

Counter-point is that when a customer reports a problem I don’t want to have to say “can you try that again after I enable logging”.

jedberg 4y ago
When a customer reports a problem the first thing I try is to reproduce it (with logging on). If I can't reproduce it, does it matter? I can leave logging on and let them know I can't reproduce their problem but if it happens again to let me know.
- encoderer 4y ago
  In your background of consumer engineering, yes it probably does not matter. But it’s not the same in SaaS
  - jedberg 4y ago
    I've worked in SaaS for years, and I don't have any examples in my history where that's the case. Do you have an example of a case where historical logging was the only way to track down a problem?
    - encoderer 4y ago
      Yesterday I used logs to help a customer who had mis-configured their SAML SSO (the problem was on the IdP side). I use our database audit trail logs to understand the sequence of changes a customer makes that sometimes results in behavior that is surprising if you only look at the current state. We put a big emphasis on the second S in SaaS and we want a magical 1-email support experience whenever possible. Audit logs help this.
- staticassertion 4y ago
  When do you turn the logs back off?
  - jedberg 4y ago
    When the problem is verified solved.
    - staticassertion 4y ago
      I can't imagine the logs are off very much of the time then
      - jedberg 4y ago
        Well when you have microservices, hopefully only one or two of them need logging at a given time.
macspoofing 4y ago
You may have to. Debug logging can be very expensive to simply leave on in production.
- falcolas 4y ago
  You can also prune logs based on level at different time intervals.
carlmr 4y ago
Exactly. Also sometimes you discover some issue and you want to see if this is never happened before, sometimes, often? Depends really how long your customers can wait.
- jedberg 4y ago
  Logging is different than metrics. Metrics should tell you if a problem has happened in the past.

mjfisher 4y ago

Really? We might be working in entirely different spheres and cultures, but I tend to look at (recently historical) logs pretty frequently for a bunch of reasons: helping to isolate infrequently encountered and hard to reproduce bugs, tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour, identifying whether a particular code path is actually in use and if so how heavily, etc etc. They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment! Edit: just realised who I was replying to. Netflix and Reddit might be very different environments with very different reliability requirements to a lot of other types of software.

jedberg 4y ago
> helping to isolate infrequently encountered and hard to reproduce bugs As I mentioned in my post, that's the one case where I find value in temporary turning on logging, but having old logs around is awfully costly for the off chance that they help. > tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour, That's request tracing, which is different. Just storing a request trace is useful, but that's not what I mean by logging. I'm talking about server application logs. > identifying whether a particular code path is actually in use and if so how heavily Again that would be request tracing, not application logging. > They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment! If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected. If they are affected, then your logs will keep showing it.
- idunno246 4y ago
  > Again that would be request tracing, not application logging. from what ive seen people who rely on logs do not have proper request tracing and/or metrics. its much simpler to dump a bunch of unformatted junk to stdout and throw elasticsearch at it, but not a good experience. i agree overall, i find logs to be a last resort with really bad signal to noise ratio.

closeparen 4y ago

As a software engineer responsible for building and operating various async and background workflows, I get a lot of inquiries like "I see cause X, why wasn't there side effect Y?" and I go and poke around the data to figure out what happened. Sometimes the cause didn't really happen, sometimes there are other criteria that the stakeholder wasn't thinking about that weren't met, sometimes the cause really happened but the data is missing or malformed upstream, and very occasionally my system errored out and dropped the ball. These are all deeply mysterious head-scratching issues if you don't have ready access to detailed logs. More mundanely, if I get paged about an error rate, the first thing I'm going to do is look at the logs to see what the errors are.

falcolas 4y ago

I, and my peers, look at historical logs daily. Our customer response team is often looking at them on a minute-by-minute basis, and sets up alerts on them. Our logs tell me the state of the application when the error occurred without having to reproduce it. They also tell me how many customers are having the issue, so we can set the severity of the incident at the appropriate level and reach out to those customers proactively if needed. Sure, we might be able to do this with metrics or tracing instead, but that's often just duplicating information we're already logging. For example, by adding a correlation ID in my log metadata, I suddenly have end-to-end request tracing. It's not as pretty as what New Relic can produce, but it definitely works.

AishwaryaVenkat 4y ago
Ofcourse, If your business is growing, chances are, you are collecting more and more logs. Your log management process is ultimately slowing down your troubleshooting. Atatus provides you with hassle-free log management delivered as a fully managed cloud service. You don't have to set up and maintain it. This instantly cuts costs, reduces IT load and improves both the speed of troubleshooting and resolution of issues.

Jugurtha 4y ago

We used Sentry[0] and it saved us a lot of time digging into lots of logs to find out some exception that went unnoticed. We still did rely on logs to troubleshoot and understand some issues. - [0]: https://sentry.io https://sentry.io