5 ms·

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging inte

by cduzz 3y ago

I have discussions with cow-orkers around logging; "We have Best-Effort and Guaranteed-Delivery APIs" "I want Guaranteed Delivery!!!" "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" "NO NO Must not take downtime!" "If you need it logged, and can't log it, what do you do?" These days I just point to the CAP theorem and suggest that logging is the same as any other distributed system. Because there's a wikipedia article with a triangle and the word "theorem" people seem to accept that. [edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API]

loeg 3y ago
I read GD as “god damn,” which also seems to fit.
- rezonant 3y ago
  aw you beat me to it
msm_ 3y ago
Interesting, I'd think logging is one of the clearest situations when you want best effort. Logging is, almost by definition, not the "core" of your application, so failure to log properly should not prevent the core of the program from working. Killing the whole program because logging server is clearly throwing the baby out with the bathwater. What people probably mean is "logging is important, let's avoid losing log messages if possible", which is what "best" in "best effort" stands for. For example it's often a good idea to have a local log queue, to avoid data loss in case of a temporary log server downtime.
- cduzz 3y ago
  People use logging (appropriately or inappropriately; not my bucket of monkeys) for a variety of things including audit and billing records, which are likely a good case for a guaranteed delivery API. People often don't think precisely about what they say or want, and also often don't think through corner cases such as "what if XYZ breaks or gets slow?" And don't get me started on "log" messages that are 300mb events. Per log. Sigh.
- insanitybit 3y ago
  If you lose logs when your service crashes you're losing logs at the time they are most important.
  - _w1tm 3y ago
    No, you’re losing client logs when your logging service crashes. Your logging service should probably not be logging through calls to itself.
  - tux1968 3y ago
    That's unavoidable if the logging service is down when your server crashes. Having a local queue doesn't mean logging to the service is delayed, it can be sent immediately. All the local queue does is give you some resiliency, by being able to retry if the first logging attempt fails.
    - insanitybit 3y ago
      If your logging service is down all bets are off. But by buffering logs you're now accepting that problems not related to the logging service will also cause you to drop logs - as I mentioned, your service crashing, or being OOM'd, would be one example.
      - tux1968 3y ago
        What's more likely? An intermittent network issue, the logging service being momentarily down, or a local crash that only affects your buffering queue? If an OOM happens, all bets are off anyway, since it has as much likelihood of taking out your application as it does your buffering code. The local buffering code might very well be part of the application in the first place, so the fate of the buffering code is the same as the application anyway. It seems you're trying very hard to contrive a situation where doing nothing is better than taking reasonable steps to counter occasional network hiccups.
        insanitybit 3y ago
- linuxdude314 3y ago
  It’s not the core of the application, but it can be the core of the business. For companies that sell API access logs in one form or another are how bills are reconciled and usage metered.
- wolverine876 3y ago
  Logging can be essential to security (to auditing). It's your record of what happened. If an attacker can cause logging to fail, they can cover their tracks more easily.
  - deathanatos 3y ago
    To me audit logs aren't "logs" (in the normal sense), despite the name. They tend to have different requirements; e.g., in my industry, they must be retained, by law, and for far longer than our normal logs. To me, those different requirements imply that they should be treated differently by the code, probably even under distinct flows: synchronously, and ideally to somewhere that I can later compress like hell and store in some very cheap long term storage. Whereas the debug logs that I use for debugging? Rotate out after 30 to 90d, … and yeah, best effort is fine. (The audit logs might also end up in one's normal logs too, for convenience.)
    - wolverine876 3y ago
      While I generally agree, I'll add that the debug logs can be useful in security incidents.
- fnordpiglet 3y ago
  It depends. Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant. I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely to cause blocking at some OS level or other sorts of jitter that’s harder to mask. Typically I develop logging as an in memory thing that offloads to a remote service over the network. The durability of the memory store can be an issue in highly sensitive workloads, and you’ll want to do synchronous disk IO for that case to ensure durability and consistent time budgets, but for almost all application disk less logging is preferable.
  - shawnz 3y ago
    If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? And if you are, then you suffer all the problems of local disk logging but also all the extra failure modes introduced by the network, too
    - fnordpiglet 3y ago
      The difference is that network IO can be more easily masked by the operating system than block device IO. When you offload your logging to another thread the story isn’t over because your disk logging can interfere at a system level. Network IO isn’t as noisy. If durability is important you might still need to wait for an ACK before freeing the buffer for the message which might lead to more overall memory use, all the operations play nicely in a preemptable scheduling system. Also, the failure modes of systems are very tied to durable storage devices attached to the system and very rarely to network devices. By reducing the number of things that need a disk (ideally to zero) you can remove disks from the system and its availability story. Once you get to fully disk less systems the system failure modes are actually almost nothing. But even with disks attached reducing the times you interact with the disk (especially for chatty things like logs!) reduces the likelihood the entire system fails due to a disk issue.
    - 3y ago
supriyo-biswas 3y ago
The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to the logging service, allowing for retries if the logging service is down or slow (for moderately short values of down/slow). Promtail[1] can do this if you're using Loki for logging. [1] https://grafana.com/docs/loki/latest/send-data/promtail/ https://grafana.com/docs/loki/latest/send-data/promtail/
- insanitybit 3y ago
  But that's still not guaranteed delivery. You're doing what the OP presented - choosing to drop logs under some circumstances when the system is down. a) If your service crashes and it's in-memory, you lose logs b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block
  - hgfghui7 3y ago
    You are thinking too much in terms of the stated requirements instead of what people actually want: good uptime and good debugability. Falling back to local logging means a blip in logging availability doesn't turn into all hands on deck everything is on fire. And it means that logs will very likely be available for any failures. In other words it's good enough.
    - mort96 3y ago
      "Good uptime and good reliability but no guarantees" is just a good best effort system.
    - insanitybit 3y ago
      Good enough is literally "best effort delivery", you're just agreeing with them that this is ultimately a distributed systems problem and you either choose CP or AP.
  - kbenson 3y ago
    Yeah, what the "best effort" actually means in practice is usually a result of how much resources you want to throw at the problem. Those give you runway on how much of a problem you can withstand and perhaps recover from without any loss of data (logs), but in the end you're usually still just buying time. That's usually enough though.
rezonant 3y ago
> "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" > [edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API] i read GD as god-damned :-)
- salamanderman 3y ago
  me too [EDIT: and I totally empathized]
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]
Zondartul 3y ago
I have some wishfull thinking ideas on this, but it should be possible to have both at least in an imaginary, theoretical scenario. You can have both guaranteed delivery and no downtime if your whole system is so deterministic that anything that normally would result in blocking just will not, cannot happen. In other words it should be a hard real-time system that is formally verified top to bottom, down to the last transistor. Does anyone actually do that? Verify the program and the hardware to prove that it will never run out of memory for logs and such? Continuing this thought, logs are probably generated endlessly, so either whoever wants them has to also guarantee that that they are processedand disposed of right after being logged... or there is a finite ammount of log messages that can be stored (arbitrary number like 10 000) but the user (of logs) has to guarantee that they will take the "mail" out of the box sooner than it overfills (at some predictable, deterministic rate). So really that means even if OUR system is mathematically perfect, we're just making the downtime someone elses problem - namely, the consumer of the infinite logs. That, or we guarantee that the final resources of our self-contained, verified system will last longer than the finite shelf life of the system as a whole (like maybe 5 years for another arbitrary number)
- morelisp 3y ago
  PACELC says you get blocking or unavailability or inconsistency.
- ElectricalUnion 3y ago
  From a hardware point of view, this system is unlikely to exist, because you need a system with components that never have any reliability issues ever to have a totally deterministic system. From a software point of view, this system is unlikely to exist as it doesn't matter that the cause of your downtime is "something else that isn't our system". As a result, you're gonna end up requiring infinite reliable storage to upkeep your promises.

by cduzz 3y ago

loeg 3y ago

I read GD as “god damn,” which also seems to fit.

rezonant 3y ago
aw you beat me to it

msm_ 3y ago

Interesting, I'd think logging is one of the clearest situations when you want best effort. Logging is, almost by definition, not the "core" of your application, so failure to log properly should not prevent the core of the program from working. Killing the whole program because logging server is clearly throwing the baby out with the bathwater. What people probably mean is "logging is important, let's avoid losing log messages if possible", which is what "best" in "best effort" stands for. For example it's often a good idea to have a local log queue, to avoid data loss in case of a temporary log server downtime.

cduzz 3y ago
People use logging (appropriately or inappropriately; not my bucket of monkeys) for a variety of things including audit and billing records, which are likely a good case for a guaranteed delivery API. People often don't think precisely about what they say or want, and also often don't think through corner cases such as "what if XYZ breaks or gets slow?" And don't get me started on "log" messages that are 300mb events. Per log. Sigh.
insanitybit 3y ago
If you lose logs when your service crashes you're losing logs at the time they are most important.
- _w1tm 3y ago
  No, you’re losing client logs when your logging service crashes. Your logging service should probably not be logging through calls to itself.
- tux1968 3y ago
  That's unavoidable if the logging service is down when your server crashes. Having a local queue doesn't mean logging to the service is delayed, it can be sent immediately. All the local queue does is give you some resiliency, by being able to retry if the first logging attempt fails.
  - insanitybit 3y ago
    If your logging service is down all bets are off. But by buffering logs you're now accepting that problems not related to the logging service will also cause you to drop logs - as I mentioned, your service crashing, or being OOM'd, would be one example.
    - tux1968 3y ago
      What's more likely? An intermittent network issue, the logging service being momentarily down, or a local crash that only affects your buffering queue? If an OOM happens, all bets are off anyway, since it has as much likelihood of taking out your application as it does your buffering code. The local buffering code might very well be part of the application in the first place, so the fate of the buffering code is the same as the application anyway. It seems you're trying very hard to contrive a situation where doing nothing is better than taking reasonable steps to counter occasional network hiccups.
      - insanitybit 3y ago
linuxdude314 3y ago
It’s not the core of the application, but it can be the core of the business. For companies that sell API access logs in one form or another are how bills are reconciled and usage metered.
wolverine876 3y ago
Logging can be essential to security (to auditing). It's your record of what happened. If an attacker can cause logging to fail, they can cover their tracks more easily.
- deathanatos 3y ago
  To me audit logs aren't "logs" (in the normal sense), despite the name. They tend to have different requirements; e.g., in my industry, they must be retained, by law, and for far longer than our normal logs. To me, those different requirements imply that they should be treated differently by the code, probably even under distinct flows: synchronously, and ideally to somewhere that I can later compress like hell and store in some very cheap long term storage. Whereas the debug logs that I use for debugging? Rotate out after 30 to 90d, … and yeah, best effort is fine. (The audit logs might also end up in one's normal logs too, for convenience.)
  - wolverine876 3y ago
    While I generally agree, I'll add that the debug logs can be useful in security incidents.
fnordpiglet 3y ago
It depends. Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant. I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely to cause blocking at some OS level or other sorts of jitter that’s harder to mask. Typically I develop logging as an in memory thing that offloads to a remote service over the network. The durability of the memory store can be an issue in highly sensitive workloads, and you’ll want to do synchronous disk IO for that case to ensure durability and consistent time budgets, but for almost all application disk less logging is preferable.
- shawnz 3y ago
  If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? And if you are, then you suffer all the problems of local disk logging but also all the extra failure modes introduced by the network, too
  - fnordpiglet 3y ago
    The difference is that network IO can be more easily masked by the operating system than block device IO. When you offload your logging to another thread the story isn’t over because your disk logging can interfere at a system level. Network IO isn’t as noisy. If durability is important you might still need to wait for an ACK before freeing the buffer for the message which might lead to more overall memory use, all the operations play nicely in a preemptable scheduling system. Also, the failure modes of systems are very tied to durable storage devices attached to the system and very rarely to network devices. By reducing the number of things that need a disk (ideally to zero) you can remove disks from the system and its availability story. Once you get to fully disk less systems the system failure modes are actually almost nothing. But even with disks attached reducing the times you interact with the disk (especially for chatty things like logs!) reduces the likelihood the entire system fails due to a disk issue.
  - 3y ago

supriyo-biswas 3y ago

The better way to do this is to write the logs to a file or an in-memory ring buffer and have a separate thread/process push logs from the file/ring-buffer to the logging service, allowing for retries if the logging service is down or slow (for moderately short values of down/slow). Promtail[1] can do this if you're using Loki for logging. [1] https://grafana.com/docs/loki/latest/send-data/promtail/ https://grafana.com/docs/loki/latest/send-data/promtail/

insanitybit 3y ago
But that's still not guaranteed delivery. You're doing what the OP presented - choosing to drop logs under some circumstances when the system is down. a) If your service crashes and it's in-memory, you lose logs b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block
- hgfghui7 3y ago
  You are thinking too much in terms of the stated requirements instead of what people actually want: good uptime and good debugability. Falling back to local logging means a blip in logging availability doesn't turn into all hands on deck everything is on fire. And it means that logs will very likely be available for any failures. In other words it's good enough.
  - mort96 3y ago
    "Good uptime and good reliability but no guarantees" is just a good best effort system.
  - insanitybit 3y ago
    Good enough is literally "best effort delivery", you're just agreeing with them that this is ultimately a distributed systems problem and you either choose CP or AP.
- kbenson 3y ago
  Yeah, what the "best effort" actually means in practice is usually a result of how much resources you want to throw at the problem. Those give you runway on how much of a problem you can withstand and perhaps recover from without any loss of data (logs), but in the end you're usually still just buying time. That's usually enough though.

rezonant 3y ago

> "If the GD logging interface is offline or slow, you'll take downtime; is that okay?" > [edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API] i read GD as god-damned :-)

salamanderman 3y ago
me too [EDIT: and I totally empathized]
deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

Zondartul 3y ago

I have some wishfull thinking ideas on this, but it should be possible to have both at least in an imaginary, theoretical scenario. You can have both guaranteed delivery and no downtime if your whole system is so deterministic that anything that normally would result in blocking just will not, cannot happen. In other words it should be a hard real-time system that is formally verified top to bottom, down to the last transistor. Does anyone actually do that? Verify the program and the hardware to prove that it will never run out of memory for logs and such? Continuing this thought, logs are probably generated endlessly, so either whoever wants them has to also guarantee that that they are processedand disposed of right after being logged... or there is a finite ammount of log messages that can be stored (arbitrary number like 10 000) but the user (of logs) has to guarantee that they will take the "mail" out of the box sooner than it overfills (at some predictable, deterministic rate). So really that means even if OUR system is mathematically perfect, we're just making the downtime someone elses problem - namely, the consumer of the infinite logs. That, or we guarantee that the final resources of our self-contained, verified system will last longer than the finite shelf life of the system as a whole (like maybe 5 years for another arbitrary number)

morelisp 3y ago
PACELC says you get blocking or unavailability or inconsistency.
ElectricalUnion 3y ago
From a hardware point of view, this system is unlikely to exist, because you need a system with components that never have any reliability issues ever to have a totally deterministic system. From a software point of view, this system is unlikely to exist as it doesn't matter that the cause of your downtime is "something else that isn't our system". As a result, you're gonna end up requiring infinite reliable storage to upkeep your promises.