4 ms·

It depends. Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are n

It depends. Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant. I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely to cause blocking at some OS level or other sorts of jitter that’s harder to mask. Typically I develop logging as an in memory thing that offloads to a remote service over the network. The durability of the memory store can be an issue in highly sensitive workloads, and you’ll want to do synchronous disk IO for that case to ensure durability and consistent time budgets, but for almost all application disk less logging is preferable.

shawnz 3y ago
If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? And if you are, then you suffer all the problems of local disk logging but also all the extra failure modes introduced by the network, too
- fnordpiglet 3y ago
  The difference is that network IO can be more easily masked by the operating system than block device IO. When you offload your logging to another thread the story isn’t over because your disk logging can interfere at a system level. Network IO isn’t as noisy. If durability is important you might still need to wait for an ACK before freeing the buffer for the message which might lead to more overall memory use, all the operations play nicely in a preemptable scheduling system. Also, the failure modes of systems are very tied to durable storage devices attached to the system and very rarely to network devices. By reducing the number of things that need a disk (ideally to zero) you can remove disks from the system and its availability story. Once you get to fully disk less systems the system failure modes are actually almost nothing. But even with disks attached reducing the times you interact with the disk (especially for chatty things like logs!) reduces the likelihood the entire system fails due to a disk issue.
- lmm 3y ago
  > If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? Depends on your failure model. I'd consider e.g. "received in memory by at least 3/5 remote servers in separate datacenters" to be safer than "committed to local disk".
  - cduzz 3y ago
    You're still on one side or another of the CAP triangle. In a network partition, you are either offline or your data is not consistent. If you're writing local to your system, you're losing data if there's a single device failure. https://en.wikipedia.org/wiki/CAP_theorem https://en.wikipedia.org/wiki/CAP_theorem
    - fnordpiglet 3y ago
      For logs, which are immutable time series journals, any copy is entirely sufficient. The first write is a quorum. Also from a systems POV reads are not a feature of logs.
    - lmm 3y ago
      CAP is irrelevant, consistency does not matter for logs.
      - cduzz 3y ago
        Consistency is a synonym for "guaranteed", and means "written to 2 remote, reliable, append-only storage endpoints" (for any reasonable definition of reliability) So -- a single system collecting a log event -- it is not reliable (guaranteed) if written just to some device on that system. Instances can be de-provisioned (and logs lost), filesystems or databases can be scrambled, badguys can encrypt your data, etc. In this context, a "network partition" prevents consistency (data not written to reliable media) or prevents availability (won't accept new requests until their activity can be logged reliably). If you define "reliably" differently, you may have a different interpretation of log consistency.
ReactiveJelly 3y ago
If it's a journaled record for the business then I think I'd write it to SQLite or something with good transactions and not mix it in the debug logs
- fnordpiglet 3y ago
  There are more logs than debug logs, and using SQLite as the encoding store for your logs doesn’t make it not logging.

shawnz 3y ago

If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? And if you are, then you suffer all the problems of local disk logging but also all the extra failure modes introduced by the network, too

fnordpiglet 3y ago
The difference is that network IO can be more easily masked by the operating system than block device IO. When you offload your logging to another thread the story isn’t over because your disk logging can interfere at a system level. Network IO isn’t as noisy. If durability is important you might still need to wait for an ACK before freeing the buffer for the message which might lead to more overall memory use, all the operations play nicely in a preemptable scheduling system. Also, the failure modes of systems are very tied to durable storage devices attached to the system and very rarely to network devices. By reducing the number of things that need a disk (ideally to zero) you can remove disks from the system and its availability story. Once you get to fully disk less systems the system failure modes are actually almost nothing. But even with disks attached reducing the times you interact with the disk (especially for chatty things like logs!) reduces the likelihood the entire system fails due to a disk issue.
lmm 3y ago
> If you're not waiting for the remote log server to write the messages to its disk before proceeding, then it seems like that's not guaranteed to me? Depends on your failure model. I'd consider e.g. "received in memory by at least 3/5 remote servers in separate datacenters" to be safer than "committed to local disk".
- cduzz 3y ago
  You're still on one side or another of the CAP triangle. In a network partition, you are either offline or your data is not consistent. If you're writing local to your system, you're losing data if there's a single device failure. https://en.wikipedia.org/wiki/CAP_theorem https://en.wikipedia.org/wiki/CAP_theorem
  - fnordpiglet 3y ago
    For logs, which are immutable time series journals, any copy is entirely sufficient. The first write is a quorum. Also from a systems POV reads are not a feature of logs.
  - lmm 3y ago
    CAP is irrelevant, consistency does not matter for logs.
    - cduzz 3y ago
      Consistency is a synonym for "guaranteed", and means "written to 2 remote, reliable, append-only storage endpoints" (for any reasonable definition of reliability) So -- a single system collecting a log event -- it is not reliable (guaranteed) if written just to some device on that system. Instances can be de-provisioned (and logs lost), filesystems or databases can be scrambled, badguys can encrypt your data, etc. In this context, a "network partition" prevents consistency (data not written to reliable media) or prevents availability (won't accept new requests until their activity can be logged reliably). If you define "reliably" differently, you may have a different interpretation of log consistency.

ReactiveJelly 3y ago

If it's a journaled record for the business then I think I'd write it to SQLite or something with good transactions and not mix it in the debug logs

fnordpiglet 3y ago
There are more logs than debug logs, and using SQLite as the encoding store for your logs doesn’t make it not logging.