9 ms·

Interesting, I'd think logging is one of the clearest situations when you want best effort. Logging is, almost by definition, not the "core" of your application

by msm_ 3y ago

Interesting, I'd think logging is one of the clearest situations when you want best effort. Logging is, almost by definition, not the "core" of your application, so failure to log properly should not prevent the core of the program from working. Killing the whole program because logging server is clearly throwing the baby out with the bathwater. What people probably mean is "logging is important, let's avoid losing log messages if possible", which is what "best" in "best effort" stands for. For example it's often a good idea to have a local log queue, to avoid data loss in case of a temporary log server downtime.

cduzz 3y ago
People use logging (appropriately or inappropriately; not my bucket of monkeys) for a variety of things including audit and billing records, which are likely a good case for a guaranteed delivery API. People often don't think precisely about what they say or want, and also often don't think through corner cases such as "what if XYZ breaks or gets slow?" And don't get me started on "log" messages that are 300mb events. Per log. Sigh.
insanitybit 3y ago
If you lose logs when your service crashes you're losing logs at the time they are most important.
- _w1tm 3y ago
  No, you’re losing client logs when your logging service crashes. Your logging service should probably not be logging through calls to itself.
- tux1968 3y ago
  That's unavoidable if the logging service is down when your server crashes. Having a local queue doesn't mean logging to the service is delayed, it can be sent immediately. All the local queue does is give you some resiliency, by being able to retry if the first logging attempt fails.
  - insanitybit 3y ago
    If your logging service is down all bets are off. But by buffering logs you're now accepting that problems not related to the logging service will also cause you to drop logs - as I mentioned, your service crashing, or being OOM'd, would be one example.
    - tux1968 3y ago
      What's more likely? An intermittent network issue, the logging service being momentarily down, or a local crash that only affects your buffering queue? If an OOM happens, all bets are off anyway, since it has as much likelihood of taking out your application as it does your buffering code. The local buffering code might very well be part of the application in the first place, so the fate of the buffering code is the same as the application anyway. It seems you're trying very hard to contrive a situation where doing nothing is better than taking reasonable steps to counter occasional network hiccups.
      - insanitybit 3y ago
        > It seems you're trying very hard to contrive a situation where doing nothing is better than taking reasonable steps to counter occasional network hiccups. I think you've completely misunderstood me then. I haven't taken a stance at all on what should be done. I'm only trying to agree with the grandparent poster about logging ultimately reflecting CAP Theorem.
- tremon 3y ago
  But if your service has downtime because the logs could not be written, that seems strictly inferior. As someone else wrote upthread, you only want guaranteed delivery for logs if they're required under a strict audit regime and the cost of noncompliance is higher than the cost of a service outage.
  - insanitybit 3y ago
    FWIW I agree, I'm just trying to be clear that you are choosing one or the other, as the grandparent was stating.
linuxdude314 3y ago
It’s not the core of the application, but it can be the core of the business. For companies that sell API access logs in one form or another are how bills are reconciled and usage metered.
wolverine876 3y ago
Logging can be essential to security (to auditing). It's your record of what happened. If an attacker can cause logging to fail, they can cover their tracks more easily.
- deathanatos 3y ago
  To me audit logs aren't "logs" (in the normal sense), despite the name. They tend to have different requirements; e.g., in my industry, they must be retained, by law, and for far longer than our normal logs. To me, those different requirements imply that they should be treated differently by the code, probably even under distinct flows: synchronously, and ideally to somewhere that I can later compress like hell and store in some very cheap long term storage. Whereas the debug logs that I use for debugging? Rotate out after 30 to 90d, … and yeah, best effort is fine. (The audit logs might also end up in one's normal logs too, for convenience.)
  - wolverine876 3y ago
    While I generally agree, I'll add that the debug logs can be useful in security incidents.
fnordpiglet 3y ago
It depends. Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant. I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely to cause blocking at some OS level or other sorts of jitter that’s harder to mask. Typically I develop logging as an in memory thing that offloads to a remote service over the network. The durability of the memory store can be an issue in highly sensitive workloads, and you’ll want to do synchronous disk IO for that case to ensure durability and consistent time budgets, but for almost all application disk less logging is preferable.
- shawnz 3y ago
  If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? And if you are, then you suffer all the problems of local disk logging but also all the extra failure modes introduced by the network, too
  - fnordpiglet 3y ago
    The difference is that network IO can be more easily masked by the operating system than block device IO. When you offload your logging to another thread the story isn’t over because your disk logging can interfere at a system level. Network IO isn’t as noisy. If durability is important you might still need to wait for an ACK before freeing the buffer for the message which might lead to more overall memory use, all the operations play nicely in a preemptable scheduling system. Also, the failure modes of systems are very tied to durable storage devices attached to the system and very rarely to network devices. By reducing the number of things that need a disk (ideally to zero) you can remove disks from the system and its availability story. Once you get to fully disk less systems the system failure modes are actually almost nothing. But even with disks attached reducing the times you interact with the disk (especially for chatty things like logs!) reduces the likelihood the entire system fails due to a disk issue.
  - 3y ago

by msm_ 3y ago

cduzz 3y ago

People use logging (appropriately or inappropriately; not my bucket of monkeys) for a variety of things including audit and billing records, which are likely a good case for a guaranteed delivery API. People often don't think precisely about what they say or want, and also often don't think through corner cases such as "what if XYZ breaks or gets slow?" And don't get me started on "log" messages that are 300mb events. Per log. Sigh.

insanitybit 3y ago

If you lose logs when your service crashes you're losing logs at the time they are most important.

_w1tm 3y ago
No, you’re losing client logs when your logging service crashes. Your logging service should probably not be logging through calls to itself.
tux1968 3y ago
That's unavoidable if the logging service is down when your server crashes. Having a local queue doesn't mean logging to the service is delayed, it can be sent immediately. All the local queue does is give you some resiliency, by being able to retry if the first logging attempt fails.
- insanitybit 3y ago
  If your logging service is down all bets are off. But by buffering logs you're now accepting that problems not related to the logging service will also cause you to drop logs - as I mentioned, your service crashing, or being OOM'd, would be one example.
  - tux1968 3y ago
    What's more likely? An intermittent network issue, the logging service being momentarily down, or a local crash that only affects your buffering queue? If an OOM happens, all bets are off anyway, since it has as much likelihood of taking out your application as it does your buffering code. The local buffering code might very well be part of the application in the first place, so the fate of the buffering code is the same as the application anyway. It seems you're trying very hard to contrive a situation where doing nothing is better than taking reasonable steps to counter occasional network hiccups.
    - insanitybit 3y ago
      > It seems you're trying very hard to contrive a situation where doing nothing is better than taking reasonable steps to counter occasional network hiccups. I think you've completely misunderstood me then. I haven't taken a stance at all on what should be done. I'm only trying to agree with the grandparent poster about logging ultimately reflecting CAP Theorem.
tremon 3y ago
But if your service has downtime because the logs could not be written, that seems strictly inferior. As someone else wrote upthread, you only want guaranteed delivery for logs if they're required under a strict audit regime and the cost of noncompliance is higher than the cost of a service outage.
- insanitybit 3y ago
  FWIW I agree, I'm just trying to be clear that you are choosing one or the other, as the grandparent was stating.

linuxdude314 3y ago

It’s not the core of the application, but it can be the core of the business. For companies that sell API access logs in one form or another are how bills are reconciled and usage metered.

wolverine876 3y ago

Logging can be essential to security (to auditing). It's your record of what happened. If an attacker can cause logging to fail, they can cover their tracks more easily.

deathanatos 3y ago
To me audit logs aren't "logs" (in the normal sense), despite the name. They tend to have different requirements; e.g., in my industry, they must be retained, by law, and for far longer than our normal logs. To me, those different requirements imply that they should be treated differently by the code, probably even under distinct flows: synchronously, and ideally to somewhere that I can later compress like hell and store in some very cheap long term storage. Whereas the debug logs that I use for debugging? Rotate out after 30 to 90d, … and yeah, best effort is fine. (The audit logs might also end up in one's normal logs too, for convenience.)
- wolverine876 3y ago
  While I generally agree, I'll add that the debug logs can be useful in security incidents.

fnordpiglet 3y ago

It depends. Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant. I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely to cause blocking at some OS level or other sorts of jitter that’s harder to mask. Typically I develop logging as an in memory thing that offloads to a remote service over the network. The durability of the memory store can be an issue in highly sensitive workloads, and you’ll want to do synchronous disk IO for that case to ensure durability and consistent time budgets, but for almost all application disk less logging is preferable.

shawnz 3y ago
If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? And if you are, then you suffer all the problems of local disk logging but also all the extra failure modes introduced by the network, too
- fnordpiglet 3y ago
  The difference is that network IO can be more easily masked by the operating system than block device IO. When you offload your logging to another thread the story isn’t over because your disk logging can interfere at a system level. Network IO isn’t as noisy. If durability is important you might still need to wait for an ACK before freeing the buffer for the message which might lead to more overall memory use, all the operations play nicely in a preemptable scheduling system. Also, the failure modes of systems are very tied to durable storage devices attached to the system and very rarely to network devices. By reducing the number of things that need a disk (ideally to zero) you can remove disks from the system and its availability story. Once you get to fully disk less systems the system failure modes are actually almost nothing. But even with disks attached reducing the times you interact with the disk (especially for chatty things like logs!) reduces the likelihood the entire system fails due to a disk issue.
- 3y ago