6 ms·

Logging is for events with identity, metrics are for events without identity. They are not the same thing. Sometimes I care about the bulk behaviour of my appl

Logging is for events with identity, metrics are for events without identity. They are not the same thing. Sometimes I care about the bulk behaviour of my application. I hit 2757 RPS and suddenly find a catastrophic gap in the performance landscape of my application -- that's useful. I don't care much about the particulars of the requests. What matters is the statistical behaviour. But sometimes the application crashes when a particular input is given. These are outliers, they don't show up clearly in statistical behaviour. In that case I care about the details. Metrics are warnings that smokers die younger, on average, than non-smokers. Logs are a biopsy showing which kind of lung cancer you have. No sensible production system or PaaS treats them as identical, even if you can derive metrics from logs.

jedberg 11y ago
> But sometimes the application crashes when a particular input is given. These are outliers, they don't show up clearly in statistical behaviour. In that case I care about the details. But my question is, why do you care about the outliers if the customer experience is not harmed by them? And if it is harmed, then it should should up in your 99th percentile metrics, even as an outlier, at which point you can turn on aggregated logging to look for that one thing. And if that one thing doesn't happen again, then did it really matter?
- mostlyjason 11y ago
  Imagine customer support where you really need a record of the individual transactions for that particular customer. A monitoring solution won't give you that level of detail.
  - jedberg 11y ago
    Individual transactions aren't found in application logs, they are found in the data store. Also, again, it's important to stress that central logging should be available, just not on all the time. If it's a specific sequence of events that triggers a problem, turn on logging and then ask them to do the sequence again. If they don't need to do the sequence again, then it wasn't really of consequence. And if it happens for multiple users, then customer service should be able to turn on let the developers know to try and find the cause which they can do by adding more metrics or perhaps some temporary central logging.
    - brazzledazzle 11y ago
      A better compromise would be always having logging on but only retaining very recent logs. You need the infrastructure to support logging and correlation anyway and if you have a short enough retention period you don't have to worry about dealing with growing infrastructure just to support it.
      - mostlyjason 11y ago
        Yeah I think most people store them for a limited time, such as a week or a month. Most issues that pop up will only be relevant that long. You can have permanent archives for individual data since storage is cheap, or even summary metrics. Better to start with ground truth because you often don't know what data you'll need in hindsight.
  - ethbro 11y ago
    I think that's the difference between a log and an audit trail. Parent makes a good point imo. Systems are now sufficiently complex that logging everything is a useless indicator. And if you're only using a subset of your logging anyway (by generating metrics), why collect anything more before you know there's a problem. Tl;dr: avoid premature log verbosity
- jacques_chester 11y ago
  > But my question is, why do you care about the outliers if the customer experience is not harmed by them? The effect of errors is not necessarily linear to their frequency. Distributed systems of any consequence are not meaningfully modelled with smooth, continuous functions suitable for highschool calculus. There is a great deal of chaos and catastrophe. Metrics -- including customer visible ones like "zero HTTP responses per second" -- raise the question. Logs provide the physical clues.
  - jedberg 11y ago
    > Metrics -- including customer visible ones like "zero HTTP responses per second" -- raise the question. Logs provide the physical clues. Exactly my point. First your monitoring tells you that there is zero HTTP response, then you turn on central logging until you find the problem. If the problem goes away before you turn the logging on, then it wasn't really a problem, right? And if it happens again, then you can turn on central logging until you find it.
    - felixgallo 11y ago
      turning on logging post-facto won't go back in time to catch a rare bug. And since you need the added resources that logging requires to be available at all times anyway, you might as well just leave logging on. What's your objection to having recent logs?
      - jedberg 11y ago
        > turning on logging post-facto won't go back in time to catch a rare bug I guess my point is that if a rare bug is rare enough, then it doesn't really matter, especially if the effect isn't catastrophic. > And since you need the added resources that logging requires to be available at all times anyway, you might as well just leave logging on. What's your objection to having recent logs? There are a lot more resources required to log everything all the time then to log a subset of things some of the time. Even having recent logs of everything is a huge overhead compared to more detailed logs of just some things.
- stouset 11y ago
  Mindsets like this are a large component of why security vulnerabilities are ubiquitous.
  - jedberg 11y ago
    That and their cost is relatively low from a business standpoint (and I say this as a security person myself). Even the worst breaches generally don't have much effect on the bottom line (Target is a great example). Ashely Madison and Code Spaces are the only ones I can think of off the top of my head that were bad enough to actually cause a material effect to the bottom line.
henrik_w 11y ago
Well put! Also, there are the cases when the system does not behave as expected (for example, mis-configuration, or wrong input from the user) - in those cases there are no exceptions or crashes, but a good log will tell you why it happens. I've given a more detailed example of this in example 2 here: http://henrikwarne.com/2013/05/05/great-programmers-write-debuggable-code/ http://henrikwarne.com/2013/05/05/great-programmers-write-de...
bbrazil 11y ago
An additional practical difference is that due to tracking less, that metrics can take much fewer resources allowing you to gain insight that's not practical from logs. I'd expect in a well instrumented system that a single request could easily affect hundreds of metrics. Trying to get that level of information from logging alone isn't practical due to the volume of data it'd require. If you want to track overall behaviour of hundreds of codepoints then metrics are good, if you want to track behaviour for all of your events at a handful of codepoints then logs are good.
kiyoto 11y ago
I think you are conflating between the needs of centralizing logs and having them around at all. The OP is saying that always centralizing them might not always make sense, and I tent to agree (and I say this as someone who maintains a popular open source log collector) If I were to play the devil's advocate, the real needs for raw log data in a centralized location is for folks outside of Ops: data analysts and data scientists.
- trjordan 11y ago
  Log data should always be centralized if the machines they're stored on are ephemeral.
  - jacques_chester 11y ago
    All machines are ephemeral. Some people just don't realise it.
    - kiyoto 11y ago
      You are preaching to the choir here. Fluentd (as a proxy for my logging-related beliefs) pushed Docker to have logging long before the logging driver, and now it is one of the officially supported logging drivers. What you don't seem to realize is that the cost of centralized logging is not always worth it. Machines are ephemeral and so are many application related problems. It's one thing to counter the OP saying that centralized logging has merits (and I believe the OP agrees with that statement) and another to say centralized logging is always a must.
  - jedberg 11y ago
    Why? My whole point is that the data isn't useful beyond the life of the machine so why store it?
    - derefr 11y ago
      If the logs might contain the stacktrace of a request that brought down your application, how do you look at them if your health-checker helpfully terminates the now-considered-wedged app VM, and your logs also "helpfully" disappear right along with it? Half the point of logs is that they're the examinable blast-wave of something that might no longer exist.

jedberg 11y ago

> But sometimes the application crashes when a particular input is given. These are outliers, they don't show up clearly in statistical behaviour. In that case I care about the details. But my question is, why do you care about the outliers if the customer experience is not harmed by them? And if it is harmed, then it should should up in your 99th percentile metrics, even as an outlier, at which point you can turn on aggregated logging to look for that one thing. And if that one thing doesn't happen again, then did it really matter?

mostlyjason 11y ago
Imagine customer support where you really need a record of the individual transactions for that particular customer. A monitoring solution won't give you that level of detail.
- jedberg 11y ago
  Individual transactions aren't found in application logs, they are found in the data store. Also, again, it's important to stress that central logging should be available, just not on all the time. If it's a specific sequence of events that triggers a problem, turn on logging and then ask them to do the sequence again. If they don't need to do the sequence again, then it wasn't really of consequence. And if it happens for multiple users, then customer service should be able to turn on let the developers know to try and find the cause which they can do by adding more metrics or perhaps some temporary central logging.
  - brazzledazzle 11y ago
    A better compromise would be always having logging on but only retaining very recent logs. You need the infrastructure to support logging and correlation anyway and if you have a short enough retention period you don't have to worry about dealing with growing infrastructure just to support it.
    - mostlyjason 11y ago
      Yeah I think most people store them for a limited time, such as a week or a month. Most issues that pop up will only be relevant that long. You can have permanent archives for individual data since storage is cheap, or even summary metrics. Better to start with ground truth because you often don't know what data you'll need in hindsight.
- ethbro 11y ago
  I think that's the difference between a log and an audit trail. Parent makes a good point imo. Systems are now sufficiently complex that logging everything is a useless indicator. And if you're only using a subset of your logging anyway (by generating metrics), why collect anything more before you know there's a problem. Tl;dr: avoid premature log verbosity
jacques_chester 11y ago
> But my question is, why do you care about the outliers if the customer experience is not harmed by them? The effect of errors is not necessarily linear to their frequency. Distributed systems of any consequence are not meaningfully modelled with smooth, continuous functions suitable for highschool calculus. There is a great deal of chaos and catastrophe. Metrics -- including customer visible ones like "zero HTTP responses per second" -- raise the question. Logs provide the physical clues.
- jedberg 11y ago
  > Metrics -- including customer visible ones like "zero HTTP responses per second" -- raise the question. Logs provide the physical clues. Exactly my point. First your monitoring tells you that there is zero HTTP response, then you turn on central logging until you find the problem. If the problem goes away before you turn the logging on, then it wasn't really a problem, right? And if it happens again, then you can turn on central logging until you find it.
  - felixgallo 11y ago
    turning on logging post-facto won't go back in time to catch a rare bug. And since you need the added resources that logging requires to be available at all times anyway, you might as well just leave logging on. What's your objection to having recent logs?
    - jedberg 11y ago
      > turning on logging post-facto won't go back in time to catch a rare bug I guess my point is that if a rare bug is rare enough, then it doesn't really matter, especially if the effect isn't catastrophic. > And since you need the added resources that logging requires to be available at all times anyway, you might as well just leave logging on. What's your objection to having recent logs? There are a lot more resources required to log everything all the time then to log a subset of things some of the time. Even having recent logs of everything is a huge overhead compared to more detailed logs of just some things.
stouset 11y ago
Mindsets like this are a large component of why security vulnerabilities are ubiquitous.
- jedberg 11y ago
  That and their cost is relatively low from a business standpoint (and I say this as a security person myself). Even the worst breaches generally don't have much effect on the bottom line (Target is a great example). Ashely Madison and Code Spaces are the only ones I can think of off the top of my head that were bad enough to actually cause a material effect to the bottom line.

henrik_w 11y ago

Well put! Also, there are the cases when the system does not behave as expected (for example, mis-configuration, or wrong input from the user) - in those cases there are no exceptions or crashes, but a good log will tell you why it happens. I've given a more detailed example of this in example 2 here: http://henrikwarne.com/2013/05/05/great-programmers-write-debuggable-code/ http://henrikwarne.com/2013/05/05/great-programmers-write-de...

bbrazil 11y ago

An additional practical difference is that due to tracking less, that metrics can take much fewer resources allowing you to gain insight that's not practical from logs. I'd expect in a well instrumented system that a single request could easily affect hundreds of metrics. Trying to get that level of information from logging alone isn't practical due to the volume of data it'd require. If you want to track overall behaviour of hundreds of codepoints then metrics are good, if you want to track behaviour for all of your events at a handful of codepoints then logs are good.

kiyoto 11y ago

I think you are conflating between the needs of centralizing logs and having them around at all. The OP is saying that always centralizing them might not always make sense, and I tent to agree (and I say this as someone who maintains a popular open source log collector) If I were to play the devil's advocate, the real needs for raw log data in a centralized location is for folks outside of Ops: data analysts and data scientists.

trjordan 11y ago
Log data should always be centralized if the machines they're stored on are ephemeral.
- jacques_chester 11y ago
  All machines are ephemeral. Some people just don't realise it.
  - kiyoto 11y ago
    You are preaching to the choir here. Fluentd (as a proxy for my logging-related beliefs) pushed Docker to have logging long before the logging driver, and now it is one of the officially supported logging drivers. What you don't seem to realize is that the cost of centralized logging is not always worth it. Machines are ephemeral and so are many application related problems. It's one thing to counter the OP saying that centralized logging has merits (and I believe the OP agrees with that statement) and another to say centralized logging is always a must.
- jedberg 11y ago
  Why? My whole point is that the data isn't useful beyond the life of the machine so why store it?
  - derefr 11y ago
    If the logs might contain the stacktrace of a request that brought down your application, how do you look at them if your health-checker helpfully terminates the now-considered-wedged app VM, and your logs also "helpfully" disappear right along with it? Half the point of logs is that they're the examinable blast-wave of something that might no longer exist.