3 ms·

To me it sounds like Kafka was not understood in full detail (maybe because missing documentation or the high complexity) and they switched to a system they bui

by buster 10y ago

To me it sounds like Kafka was not understood in full detail (maybe because missing documentation or the high complexity) and they switched to a system they build themselves. Naturally they know in full detail what is going on and can set up the system as needed. I am wondering if working on solving the actual problems with Kafka would have been the better route. I've never used Kafka and i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations. How can you claim to run a serious business and say "babies will die" when you can't be sure to be able to find problems? Because, when will you lose logs? Not in normal operation, but when weird things happen. When networking has a hiccup. When Load on the system is too high, so most likely when many people are using your service. Exactly when shit hits the fan. And you just made the decision that it's ok to drop log messages in such cases? That's not good. I think you should either dive into Kafka/Zookeeper and fix your problems or switch to another logging solution. You should probably just drop that non-sense "streaming and real-time logs" requirement and live with a log delay of a few seconds and build something really stable instead of building something inherently unstable. Honestly, just collecting syslogs on the core vm and sending them to a central server would have been the better solution. Better then looking into fancy real-time, streaming logs on a sunday night because the system is having a breakdown and you can't even be sure that you are not missing essential logs.

macns 10y ago
[..] i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations. ZMQ silently drops messages when subscribers fail or not listening or when buffers fill up, but as they describe later on "access to historical logs", it's much easier to set up separate process/es for just that. It seems that when shit hits the fan for this reason ZMQ really is a more reliable choice because it's more flexible.
- buster 10y ago
  No. When you can't rely on your collected ZMQ logs and need to "access historical logs" by some other means, why use the ZMQ logs at all? You usually don't know that something was not logged. Also, as he describes in the article, historical logs are scoped out and it is "likely" they they will develop something for those logs in the future. Again it looks like the plan is to use ZMQ and a subscriber to put those logs into logstash. That doesn't solve the problem i mentioned at all. ZMQ may still drop the logs! So, as far as i understand they don't have a plan for reliable logging. Even if they would, they'd have one reliable solution and an unreliable solution. The unreliable ZMQ based approach is probably neat and leads to fancy realtime dashboard stuff, but since it's not a reliable source of information it's not a good solution for operating a system where "babies will die".
  - macns 10y ago
    AFAIK there's no ZMQ logs, ZMQ is just messaging patterns over various protocols, logging is your responsibility.
AYBABTME 10y ago
One has only two choices in those situations: drop logs or block receiving more logs. Given their availability requirements, I don't think that blocking is a viable choice. So dropping logs seems to be the only sane choice here. There's no other alternative really so I'm not sure about the consternation.
- buster 10y ago
  What? No. You can just save logs on the disk and buffer them. Just dropping logs or blocking because some network resource is not available are both terrible choices and that's not how logging worked for the last decades. Throwing away logs ist a major step backwards.
  - AYBABTME 10y ago
    If you buffer to disk, the same problem will eventually show up. Queues (in memory, on disk, anywhere) are all ultimately bounded, and when they are full, you have 2 choices: block or drop. Somehow you need to make the choice, there's no getting away from it. If you don't make the choice consciously, say by assuming that you can buffer to disk and avoid the problem, at some point you'll fill up your disks and your system will block: you'll have unknowingly picked the "block" option. If you decide to rotate logs and delete old rotations when too many logs are present, then you're picking the "drop" option...
    - buster 10y ago
      That's why you aggregate logs in a central service. I was writing about sending logs to a central service and not about how your disks fill up with more logs. There is log rotation for that and usually your logs will have been sent way before any log rotates. If your log rotation deletes logs before you aggregated them or if you let your disks fill up with logs you have a much bigger problem you should fix, of course.
      - AYBABTME 10y ago
        That's again the same problem; if your centralized service isn't reachable for whatever reason, your nodes can buffer for a while (in memory or on disk) but eventually the problem always will boil down to 'drop' or 'block'. However you construct it, somewhere you need to make that call. They made the call to drop logs, it's totally fine.

by buster 10y ago

macns 10y ago

[..] i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations. ZMQ silently drops messages when subscribers fail or not listening or when buffers fill up, but as they describe later on "access to historical logs", it's much easier to set up separate process/es for just that. It seems that when shit hits the fan for this reason ZMQ really is a more reliable choice because it's more flexible.

buster 10y ago
No. When you can't rely on your collected ZMQ logs and need to "access historical logs" by some other means, why use the ZMQ logs at all? You usually don't know that something was not logged. Also, as he describes in the article, historical logs are scoped out and it is "likely" they they will develop something for those logs in the future. Again it looks like the plan is to use ZMQ and a subscriber to put those logs into logstash. That doesn't solve the problem i mentioned at all. ZMQ may still drop the logs! So, as far as i understand they don't have a plan for reliable logging. Even if they would, they'd have one reliable solution and an unreliable solution. The unreliable ZMQ based approach is probably neat and leads to fancy realtime dashboard stuff, but since it's not a reliable source of information it's not a good solution for operating a system where "babies will die".
- macns 10y ago
  AFAIK there's no ZMQ logs, ZMQ is just messaging patterns over various protocols, logging is your responsibility.

AYBABTME 10y ago

One has only two choices in those situations: drop logs or block receiving more logs. Given their availability requirements, I don't think that blocking is a viable choice. So dropping logs seems to be the only sane choice here. There's no other alternative really so I'm not sure about the consternation.

buster 10y ago
What? No. You can just save logs on the disk and buffer them. Just dropping logs or blocking because some network resource is not available are both terrible choices and that's not how logging worked for the last decades. Throwing away logs ist a major step backwards.
- AYBABTME 10y ago
  If you buffer to disk, the same problem will eventually show up. Queues (in memory, on disk, anywhere) are all ultimately bounded, and when they are full, you have 2 choices: block or drop. Somehow you need to make the choice, there's no getting away from it. If you don't make the choice consciously, say by assuming that you can buffer to disk and avoid the problem, at some point you'll fill up your disks and your system will block: you'll have unknowingly picked the "block" option. If you decide to rotate logs and delete old rotations when too many logs are present, then you're picking the "drop" option...
  - buster 10y ago
    That's why you aggregate logs in a central service. I was writing about sending logs to a central service and not about how your disks fill up with more logs. There is log rotation for that and usually your logs will have been sent way before any log rotates. If your log rotation deletes logs before you aggregated them or if you let your disks fill up with logs you have a much bigger problem you should fix, of course.
    - AYBABTME 10y ago
      That's again the same problem; if your centralized service isn't reachable for whatever reason, your nodes can buffer for a while (in memory or on disk) but eventually the problem always will boil down to 'drop' or 'block'. However you construct it, somewhere you need to make that call. They made the call to drop logs, it's totally fine.