4 ms·

Counter-point is that when a customer reports a problem I don’t want to have to say “can you try that again after I enable logging”.

jedberg 4y ago
When a customer reports a problem the first thing I try is to reproduce it (with logging on). If I can't reproduce it, does it matter? I can leave logging on and let them know I can't reproduce their problem but if it happens again to let me know.
- encoderer 4y ago
  In your background of consumer engineering, yes it probably does not matter. But it’s not the same in SaaS
  - jedberg 4y ago
    I've worked in SaaS for years, and I don't have any examples in my history where that's the case. Do you have an example of a case where historical logging was the only way to track down a problem?
    - encoderer 4y ago
      Yesterday I used logs to help a customer who had mis-configured their SAML SSO (the problem was on the IdP side). I use our database audit trail logs to understand the sequence of changes a customer makes that sometimes results in behavior that is surprising if you only look at the current state. We put a big emphasis on the second S in SaaS and we want a magical 1-email support experience whenever possible. Audit logs help this.
- staticassertion 4y ago
  When do you turn the logs back off?
  - jedberg 4y ago
    When the problem is verified solved.
    - staticassertion 4y ago
      I can't imagine the logs are off very much of the time then
      - jedberg 4y ago
        Well when you have microservices, hopefully only one or two of them need logging at a given time.
macspoofing 4y ago
You may have to. Debug logging can be very expensive to simply leave on in production.
- falcolas 4y ago
  You can also prune logs based on level at different time intervals.
carlmr 4y ago
Exactly. Also sometimes you discover some issue and you want to see if this is never happened before, sometimes, often? Depends really how long your customers can wait.
- jedberg 4y ago
  Logging is different than metrics. Metrics should tell you if a problem has happened in the past.

jedberg 4y ago

When a customer reports a problem the first thing I try is to reproduce it (with logging on). If I can't reproduce it, does it matter? I can leave logging on and let them know I can't reproduce their problem but if it happens again to let me know.

encoderer 4y ago
In your background of consumer engineering, yes it probably does not matter. But it’s not the same in SaaS
- jedberg 4y ago
  I've worked in SaaS for years, and I don't have any examples in my history where that's the case. Do you have an example of a case where historical logging was the only way to track down a problem?
  - encoderer 4y ago
    Yesterday I used logs to help a customer who had mis-configured their SAML SSO (the problem was on the IdP side). I use our database audit trail logs to understand the sequence of changes a customer makes that sometimes results in behavior that is surprising if you only look at the current state. We put a big emphasis on the second S in SaaS and we want a magical 1-email support experience whenever possible. Audit logs help this.
staticassertion 4y ago
When do you turn the logs back off?
- jedberg 4y ago
  When the problem is verified solved.
  - staticassertion 4y ago
    I can't imagine the logs are off very much of the time then
    - jedberg 4y ago
      Well when you have microservices, hopefully only one or two of them need logging at a given time.

macspoofing 4y ago

You may have to. Debug logging can be very expensive to simply leave on in production.

falcolas 4y ago
You can also prune logs based on level at different time intervals.

carlmr 4y ago

Exactly. Also sometimes you discover some issue and you want to see if this is never happened before, sometimes, often? Depends really how long your customers can wait.

jedberg 4y ago
Logging is different than metrics. Metrics should tell you if a problem has happened in the past.