3 ms·

One has only two choices in those situations: drop logs or block receiving more logs. Given their availability requirements, I don't think that blocking is a vi

One has only two choices in those situations: drop logs or block receiving more logs. Given their availability requirements, I don't think that blocking is a viable choice. So dropping logs seems to be the only sane choice here. There's no other alternative really so I'm not sure about the consternation.

buster 10y ago
What? No. You can just save logs on the disk and buffer them. Just dropping logs or blocking because some network resource is not available are both terrible choices and that's not how logging worked for the last decades. Throwing away logs ist a major step backwards.
- AYBABTME 10y ago
  If you buffer to disk, the same problem will eventually show up. Queues (in memory, on disk, anywhere) are all ultimately bounded, and when they are full, you have 2 choices: block or drop. Somehow you need to make the choice, there's no getting away from it. If you don't make the choice consciously, say by assuming that you can buffer to disk and avoid the problem, at some point you'll fill up your disks and your system will block: you'll have unknowingly picked the "block" option. If you decide to rotate logs and delete old rotations when too many logs are present, then you're picking the "drop" option...
  - buster 10y ago
    That's why you aggregate logs in a central service. I was writing about sending logs to a central service and not about how your disks fill up with more logs. There is log rotation for that and usually your logs will have been sent way before any log rotates. If your log rotation deletes logs before you aggregated them or if you let your disks fill up with logs you have a much bigger problem you should fix, of course.
    - AYBABTME 10y ago
      That's again the same problem; if your centralized service isn't reachable for whatever reason, your nodes can buffer for a while (in memory or on disk) but eventually the problem always will boil down to 'drop' or 'block'. However you construct it, somewhere you need to make that call. They made the call to drop logs, it's totally fine.
      - buster 10y ago
        If your central infrastructure is down for many days you have other problems. Buffering logs on disks. Rotating and zipping files doesn't take much space. You can buffer a lot before you run into trouble. That's my point. You sound like buffering on the node is only possible for seconds whereas in real world scenarios log files are written for days, weeks or even months. Even in very large deployments and lots of logs. You would make the decision to don't use this advantage and throw logs away for no good reason.
hinkley 10y ago
I come to this discussion much like one would sit down at a bar and find their two friends are deep in a coding discussion they didn't hear the beginning of. How do you end up with a system design that can generate logs so fast that you can't keep up? It seems to be that some fundamental element of capacity planning was missed long ago and we're trying to fix the symptoms and not the cause. If I have a geographically distributed system, I'm going to have bandwidth and latency issues if I try this, sure. But why do I care? A request to the SF data center shouldn't involve the Munich data center. That is, if I care about response times, and if I don't care about that, then why do I care about instantaneous log availability? I think sometimes we get so bored with the problems we have that we invent new things to get upset about. Or management does, which is always worse.

buster 10y ago

What? No. You can just save logs on the disk and buffer them. Just dropping logs or blocking because some network resource is not available are both terrible choices and that's not how logging worked for the last decades. Throwing away logs ist a major step backwards.

AYBABTME 10y ago
If you buffer to disk, the same problem will eventually show up. Queues (in memory, on disk, anywhere) are all ultimately bounded, and when they are full, you have 2 choices: block or drop. Somehow you need to make the choice, there's no getting away from it. If you don't make the choice consciously, say by assuming that you can buffer to disk and avoid the problem, at some point you'll fill up your disks and your system will block: you'll have unknowingly picked the "block" option. If you decide to rotate logs and delete old rotations when too many logs are present, then you're picking the "drop" option...
- buster 10y ago
  That's why you aggregate logs in a central service. I was writing about sending logs to a central service and not about how your disks fill up with more logs. There is log rotation for that and usually your logs will have been sent way before any log rotates. If your log rotation deletes logs before you aggregated them or if you let your disks fill up with logs you have a much bigger problem you should fix, of course.
  - AYBABTME 10y ago
    That's again the same problem; if your centralized service isn't reachable for whatever reason, your nodes can buffer for a while (in memory or on disk) but eventually the problem always will boil down to 'drop' or 'block'. However you construct it, somewhere you need to make that call. They made the call to drop logs, it's totally fine.
    - buster 10y ago
      If your central infrastructure is down for many days you have other problems. Buffering logs on disks. Rotating and zipping files doesn't take much space. You can buffer a lot before you run into trouble. That's my point. You sound like buffering on the node is only possible for seconds whereas in real world scenarios log files are written for days, weeks or even months. Even in very large deployments and lots of logs. You would make the decision to don't use this advantage and throw logs away for no good reason.

hinkley 10y ago

I come to this discussion much like one would sit down at a bar and find their two friends are deep in a coding discussion they didn't hear the beginning of. How do you end up with a system design that can generate logs so fast that you can't keep up? It seems to be that some fundamental element of capacity planning was missed long ago and we're trying to fix the symptoms and not the cause. If I have a geographically distributed system, I'm going to have bandwidth and latency issues if I try this, sure. But why do I care? A request to the SF data center shouldn't involve the Munich data center. That is, if I care about response times, and if I don't care about that, then why do I care about instantaneous log availability? I think sometimes we get so bored with the problems we have that we invent new things to get upset about. Or management does, which is always worse.