4 ms·

And that puzzles me: how useful is a logging solution which can loose log messages by design? In an environment where "babies will die"? Maybe there is durable

And that puzzles me: how useful is a logging solution which can loose log messages by design? In an environment where "babies will die"? Maybe there is durable logging somewhere else? What am I missing here?

vidarh 11y ago
Consider the situations where it's likely to lose log messages: When the system is overloaded. Exactly the type of situations where you don't want to slow the system down further by spending resources on trying to let non-essential services survive. Presumably their tradeoff is that it's more important for the system to remain available than for every log message to be delivered. Then secondly you try to deliver log messages with as high reliability as possible. Often it is better to design for non-essentially components to fail early, or at least prevent their resource usage from escalating and dragging down other parts of the system (in this case, fixed buffers in 0MQ lets them isolate load in one part of the system by simply locking the rate the drain the buffers at below a suitable threshold that's normally fast enough).
- acconsta 11y ago
  The problem is that overloaded 0MQ pub sockets drop exactly the data you don't want to drop — the newest data. Maybe they handle that in the application layer, but it's not clear.
  - PieterH 11y ago
    It's a problem in theory, not in practice. The notion of throwing away old data (or coalescing new with old) is relevant to slow queues. It turns out to be irrelevant with ZeroMQ. First, message rates with ZeroMQ are often hundreds of thousands per second. The architecture must be designed so that no buffers, anywhere, overflow. If they do, you have a problem, usually a slow subscriber. Throwing out older data doesn't cure the problem. What ZeroMQ does is punish the slow subscriber by dropping so that the publisher doesn't crash. It's not recovery for the subscriber, it's protection for the publisher (and thus for other subscribers). Second, trying to delete old messages is complex and sometimes impossible (if they're already in system buffers). The design of ZeroMQ's internal pipes has one writer and one reader, without locks. For the writer to mess with the reader would slow down everything and introduce risk of bugs. Dropping new incoming data is the only way anyone has ever found to keep things running at full speed. These design choices were often delicate and counter-intuitive, yet they have turned out to be mostly accurate.
    - acconsta 11y ago
      >trying to delete old messages is complex and sometimes impossible I'm not familiar with ZeroMQ's data structures, so forgive my ignorance. At the high water mark, why can't the consumer throw away old messages instead of the producer throwing away new messages? There are no locks or bugs — that's what the consumer does anyway. I'm not saying that should be the default behavior, but perhaps an option. It's cleaner than silently dropping new messages, then sending an entire buffer of old messages if the subscriber recovers.
    - vidarh 11y ago
      I love the thinking behind ZeroMQ... I've been a big fan of the work you guys have been doing all the way back to Libero (I used it to clean up a blazing fast multiplexed NNTP server with it back in '96 or '97; the first version did all the state transitions manually)
- jacques_chester 11y ago
  This is why you should drop metrics first, then logs. Also, as you've alluded to, guaranteeing log availability provides an enticing cross-tenant denial of service vector. Vomit enough log messages into a shared fabric and you can begin to affect your neighbours. ZeroMQ, if I read right, diminishes some of this risk.

vidarh 11y ago

Consider the situations where it's likely to lose log messages: When the system is overloaded. Exactly the type of situations where you don't want to slow the system down further by spending resources on trying to let non-essential services survive. Presumably their tradeoff is that it's more important for the system to remain available than for every log message to be delivered. Then secondly you try to deliver log messages with as high reliability as possible. Often it is better to design for non-essentially components to fail early, or at least prevent their resource usage from escalating and dragging down other parts of the system (in this case, fixed buffers in 0MQ lets them isolate load in one part of the system by simply locking the rate the drain the buffers at below a suitable threshold that's normally fast enough).

acconsta 11y ago
The problem is that overloaded 0MQ pub sockets drop exactly the data you don't want to drop — the newest data. Maybe they handle that in the application layer, but it's not clear.
- PieterH 11y ago
  It's a problem in theory, not in practice. The notion of throwing away old data (or coalescing new with old) is relevant to slow queues. It turns out to be irrelevant with ZeroMQ. First, message rates with ZeroMQ are often hundreds of thousands per second. The architecture must be designed so that no buffers, anywhere, overflow. If they do, you have a problem, usually a slow subscriber. Throwing out older data doesn't cure the problem. What ZeroMQ does is punish the slow subscriber by dropping so that the publisher doesn't crash. It's not recovery for the subscriber, it's protection for the publisher (and thus for other subscribers). Second, trying to delete old messages is complex and sometimes impossible (if they're already in system buffers). The design of ZeroMQ's internal pipes has one writer and one reader, without locks. For the writer to mess with the reader would slow down everything and introduce risk of bugs. Dropping new incoming data is the only way anyone has ever found to keep things running at full speed. These design choices were often delicate and counter-intuitive, yet they have turned out to be mostly accurate.
  - acconsta 11y ago
    >trying to delete old messages is complex and sometimes impossible I'm not familiar with ZeroMQ's data structures, so forgive my ignorance. At the high water mark, why can't the consumer throw away old messages instead of the producer throwing away new messages? There are no locks or bugs — that's what the consumer does anyway. I'm not saying that should be the default behavior, but perhaps an option. It's cleaner than silently dropping new messages, then sending an entire buffer of old messages if the subscriber recovers.
  - vidarh 11y ago
    I love the thinking behind ZeroMQ... I've been a big fan of the work you guys have been doing all the way back to Libero (I used it to clean up a blazing fast multiplexed NNTP server with it back in '96 or '97; the first version did all the state transitions manually)
jacques_chester 11y ago
This is why you should drop metrics first, then logs. Also, as you've alluded to, guaranteeing log availability provides an enticing cross-tenant denial of service vector. Vomit enough log messages into a shared fabric and you can begin to affect your neighbours. ZeroMQ, if I read right, diminishes some of this risk.