5 ms·

Given that they consider durable logging to be a key non-functional requirement, I'm not sure how ditching durability is a win. Single ordered delivery is hard

Given that they consider durable logging to be a key non-functional requirement, I'm not sure how ditching durability is a win. Single ordered delivery is hard. Really hard. It's easier to allow multiple delivery and/or unordered delivery -- if your log entries are given various correlating UUIDs and signed, this makes the whole problem about fifty hojillion times easier. Loggregator[1] and Doppler[2] use UUIDs to identify request, app and agent, which makes it easier to allow logs to arrive out of order from multiple sources and then be reassembled. One thing that helps a lot is to separate metrics from logs. Logging frameworks get overloaded into metrics systems. Logs are useful if you are recording essentially unique events ("I started at ...", "there was a request for /foo at..."). They are wasteful of high QOS resources if you're looking at statistical data in which fine-grained event identity isn't relevant ("57Mb is used by this process", "this request took 100ms"). For metrics it is better to use lossy sampling and derive a statistical view of goings on, rather than trying to capture every single data point. A log entry should be like a page in your personal diary on your birthday. A metric is a phone survey asking your age. If you don't answer the survey, the data is still useful even with standard error. Edit: I forgot my usual disclaimer that I work for Pivotal Labs, a division of Pivotal, which is the leading contributor of engineering effort on Cloud Foundry. [1] https://github.com/cloudfoundry/loggregator https://github.com/cloudfoundry/loggregator [2] https://github.com/cloudfoundry/loggregator/tree/develop/src/doppler https://github.com/cloudfoundry/loggregator/tree/develop/src...

cwp 11y ago
Read more closely. Durability was NOT a requirement. Since we've decided to scope out access to historical logs from the problem we were trying to solve and focus only on real-time log consolidation, that feature of Kafka became an unnecessary penalty without providing any benefits.
- ZielenMiejska 11y ago
  And that puzzles me: how useful is a logging solution which can loose log messages by design? In an environment where "babies will die"? Maybe there is durable logging somewhere else? What am I missing here?
  - vidarh 11y ago
    Consider the situations where it's likely to lose log messages: When the system is overloaded. Exactly the type of situations where you don't want to slow the system down further by spending resources on trying to let non-essential services survive. Presumably their tradeoff is that it's more important for the system to remain available than for every log message to be delivered. Then secondly you try to deliver log messages with as high reliability as possible. Often it is better to design for non-essentially components to fail early, or at least prevent their resource usage from escalating and dragging down other parts of the system (in this case, fixed buffers in 0MQ lets them isolate load in one part of the system by simply locking the rate the drain the buffers at below a suitable threshold that's normally fast enough).
    - acconsta 11y ago
      The problem is that overloaded 0MQ pub sockets drop exactly the data you don't want to drop — the newest data. Maybe they handle that in the application layer, but it's not clear.
      - PieterH 11y ago
        It's a problem in theory, not in practice. The notion of throwing away old data (or coalescing new with old) is relevant to slow queues. It turns out to be irrelevant with ZeroMQ. First, message rates with ZeroMQ are often hundreds of thousands per second. The architecture must be designed so that no buffers, anywhere, overflow. If they do, you have a problem, usually a slow subscriber. Throwing out older data doesn't cure the problem. What ZeroMQ does is punish the slow subscriber by dropping so that the publisher doesn't crash. It's not recovery for the subscriber, it's protection for the publisher (and thus for other subscribers). Second, trying to delete old messages is complex and sometimes impossible (if they're already in system buffers). The design of ZeroMQ's internal pipes has one writer and one reader, without locks. For the writer to mess with the reader would slow down everything and introduce risk of bugs. Dropping new incoming data is the only way anyone has ever found to keep things running at full speed. These design choices were often delicate and counter-intuitive, yet they have turned out to be mostly accurate.
- jacques_chester 11y ago
  I read it differently, based on: At Auth0, high availability is the high order bit. We don't stress as much about throughput or performance as we do about high availability. We don't have a concept of a maintenance period. We need to be available 24/7, year round. Given the nature of some of our customers, if we are down, "babies will die". Though, as you point out, they simply dropped durability from their requirements.

cwp 11y ago

Read more closely. Durability was NOT a requirement. Since we've decided to scope out access to historical logs from the problem we were trying to solve and focus only on real-time log consolidation, that feature of Kafka became an unnecessary penalty without providing any benefits.

ZielenMiejska 11y ago
And that puzzles me: how useful is a logging solution which can loose log messages by design? In an environment where "babies will die"? Maybe there is durable logging somewhere else? What am I missing here?
- vidarh 11y ago
  Consider the situations where it's likely to lose log messages: When the system is overloaded. Exactly the type of situations where you don't want to slow the system down further by spending resources on trying to let non-essential services survive. Presumably their tradeoff is that it's more important for the system to remain available than for every log message to be delivered. Then secondly you try to deliver log messages with as high reliability as possible. Often it is better to design for non-essentially components to fail early, or at least prevent their resource usage from escalating and dragging down other parts of the system (in this case, fixed buffers in 0MQ lets them isolate load in one part of the system by simply locking the rate the drain the buffers at below a suitable threshold that's normally fast enough).
  - acconsta 11y ago
    The problem is that overloaded 0MQ pub sockets drop exactly the data you don't want to drop — the newest data. Maybe they handle that in the application layer, but it's not clear.
    - PieterH 11y ago
      It's a problem in theory, not in practice. The notion of throwing away old data (or coalescing new with old) is relevant to slow queues. It turns out to be irrelevant with ZeroMQ. First, message rates with ZeroMQ are often hundreds of thousands per second. The architecture must be designed so that no buffers, anywhere, overflow. If they do, you have a problem, usually a slow subscriber. Throwing out older data doesn't cure the problem. What ZeroMQ does is punish the slow subscriber by dropping so that the publisher doesn't crash. It's not recovery for the subscriber, it's protection for the publisher (and thus for other subscribers). Second, trying to delete old messages is complex and sometimes impossible (if they're already in system buffers). The design of ZeroMQ's internal pipes has one writer and one reader, without locks. For the writer to mess with the reader would slow down everything and introduce risk of bugs. Dropping new incoming data is the only way anyone has ever found to keep things running at full speed. These design choices were often delicate and counter-intuitive, yet they have turned out to be mostly accurate.
jacques_chester 11y ago
I read it differently, based on: At Auth0, high availability is the high order bit. We don't stress as much about throughput or performance as we do about high availability. We don't have a concept of a maintenance period. We need to be available 24/7, year round. Given the nature of some of our customers, if we are down, "babies will die". Though, as you point out, they simply dropped durability from their requirements.