7 ms·

Inbuilt docker + fluentd driver + plain ES falls flat on its face under scale. Not to mention docker daemon keeps dying :(. Whats working for us now is local d

by r4um 10y ago

Inbuilt docker + fluentd driver + plain ES falls flat on its face under scale. Not to mention docker daemon keeps dying :(. Whats working for us now is local docker json logs -> heka -> kafka -> graylog -> ES. We even dockerized graylog to scale the processing up and down on demand. Does 250k messages/sec at peak easily, More details in this presentation https://www.youtube.com/watch?v=PB8dBnpaP8s https://www.youtube.com/watch?v=PB8dBnpaP8s

edu115 10y ago
Too bad that Heka has been officially deprecated, sounds like a similar use case to what we have.
vardump 10y ago
> Does 250k messages/sec at peak easily, More details in this presentation https://www.youtube.com/watch?v=PB8dBnpaP8s https://www.youtube.com/watch?v=PB8dBnpaP8s 250k messages/sec is pretty low. It'd be useful (not to mention entirely feasible) to be able to handle 5M/sec bursts on a single system and aggregate sustained 50M/s+ (10Gbps network). The system should be able to establish total order (serialization) locally and causal order over the network. I did some tests and found out hardware limits are somewhere between 20-50M messages per second (serialized) on current consumer grade X86. Practical implementation would of course be slower. Of course you need to go binary logging at that point, adhere to cache line boundaries, etc. mechanical sympathy. Maybe even do usermode networking at the aggregator server. Binary logging is a must, because even something like string formatting is simply way too slow, by an order of magnitude. In my quick tests I found the string processing hit to be surprisingly high, 10-50x. C "sprintf" and C++ stringstream are atrociously slow. (Surprisingly, considering sprintf has a pretty complicated "bytecode" format specifier parsing loop, it was still significantly faster than stringstream implementation.) Timestamps are another huge performance issue. Typical system calls for high precision (a few microseconds or better) timestamp take several microseconds each - using one of those will alone drop performance to 100-500k range. It's of course much faster to use RDTSC, but then you have the issue that different CPU sockets have different offsets (sometimes large) and possibly also some frequency difference between them. Also older CPUs don't have invariant TSC, so their speed changes when CPU frequency changes. (Some word of warning about Windows QueryPerformanceCounter: When you test it on your development laptop, it appears fast, because it's using RDTSC behind the scenes. But when it runs on a NUMA server, it often changes behavior and becomes 20-100x slower, because Windows starts to use HPET instead. RDTSC takes maybe ~10 ns to execute and it's not a shared resource. HPET takes 1-2 microseconds to read and is a shared system resource, concurrent access from multiple cores will make it slower.)
- r4um 10y ago
  That is the peak we see right now at ~20-30% utilization overall network/cpu wise, it can do much more easily. > Binary logging is a must, because even something like string formatting is simply way too slow, by an order of magnitude. true, string processing takes most of the resources.
  - vardump 10y ago
    Timestamps are even worse. They're slow to get and have a multitude of other problems, like timezones, system clock adjustments (manual or NTP), rate drift, offset errors, etc. Some solve it by caching a low res timestamp, but that means losing ability to get side channel information about precisely how long time passed between two events. High resolution (micro/nanosecond range) timestamps give so much more information than just time.
    - r4um 10y ago
      We haven't dealt with those yet. But in general we are always pushing everyone to log judicially, use metrics wherever possible (numbers are still easier to manage than strings), use sentry for exceptions logging and introducing zipkin for tracing.
- user5994461 10y ago
  What usage could possibly need 5M log/s on a single system? Are you trying to log every TCP and UDP packet? To give a rough point of comparison, CloudFlare is taking 4M page views per second and that's 5% of the global internet traffic.
  - vardump 10y ago
    > Are you trying to log every TCP and UDP packet? No. Although I've done that too... To debug timing critical code. Or Heisenbugs that disappear when you log. Ever had a bug that vanishes when you enable logging? Not 5M sustained, just short sub-millisecond bursts here and there.
chrissnell 10y ago
Heka proved to be the weak link in our logging stack. We get far better performance from fluentd. We use a multi-tiered fluentd architecture. One tier accepts logs from Docker and (optionally) sends a container's log stream to a developer-managed custom fluentd parser. From here, logs are sent to a fluentd "sink" for storage in Elasticsearch. Using multiple tiers helps prevent overload and acts as a buffer if ES slows down temporarily.
- forgotpwtomain 10y ago
  > Heka proved to be the weak link in our logging stack. We get far better performance from fluentd. I'd be curious to know some more details on this. I guess Go channels do copying rather than sharing, but I'd still expect Heka to perform better. Personally I've found Heka to be more robust - I've had to fix bugs in plugins for the former two and generally I haven't found them to be architecturally sound. For example with logstash I've had bugs in plugins which would crash the entire daemon -- that's just inexcusable for a core infrastructure service. The multi-tiered thing sounds very strange, why not just hold an on-disk buffer?
  - r4um 10y ago
    Same for us fluentd was very resource hungry, heka brought down utilization significantly. Regarding the buffering we just added kafka in between for graylog/ES scaling/maintenance.
  - chrissnell 10y ago
    Believe me, I wanted Heka to work. We're a big Go shop and I understood Heka better than any of the competing log shippers. We got very deep into Heka and ultimately, it just couldn't do the job. It performed very poorly when there was any sort of a bottleneck into Elasticsearch. It would enter an unrecoverable state when it got choked up We spent a lot of time on the Heka IRC channel talking to the devs. These were known issues. Some of the Heka team are/were working on a replacement written in C. It sounded promising but the ES output plugin did not exist yet so we couldn't use it. The need for a performant and powerful log shipper is still there. I hope to see some new options come around soon that can achieve 1MM+ lines/sec from a single daemon without requiring multiple tiers, receivers, etc.

by r4um 10y ago

edu115 10y ago

Too bad that Heka has been officially deprecated, sounds like a similar use case to what we have.

vardump 10y ago

> Does 250k messages/sec at peak easily, More details in this presentation https://www.youtube.com/watch?v=PB8dBnpaP8s https://www.youtube.com/watch?v=PB8dBnpaP8s 250k messages/sec is pretty low. It'd be useful (not to mention entirely feasible) to be able to handle 5M/sec bursts on a single system and aggregate sustained 50M/s+ (10Gbps network). The system should be able to establish total order (serialization) locally and causal order over the network. I did some tests and found out hardware limits are somewhere between 20-50M messages per second (serialized) on current consumer grade X86. Practical implementation would of course be slower. Of course you need to go binary logging at that point, adhere to cache line boundaries, etc. mechanical sympathy. Maybe even do usermode networking at the aggregator server. Binary logging is a must, because even something like string formatting is simply way too slow, by an order of magnitude. In my quick tests I found the string processing hit to be surprisingly high, 10-50x. C "sprintf" and C++ stringstream are atrociously slow. (Surprisingly, considering sprintf has a pretty complicated "bytecode" format specifier parsing loop, it was still significantly faster than stringstream implementation.) Timestamps are another huge performance issue. Typical system calls for high precision (a few microseconds or better) timestamp take several microseconds each - using one of those will alone drop performance to 100-500k range. It's of course much faster to use RDTSC, but then you have the issue that different CPU sockets have different offsets (sometimes large) and possibly also some frequency difference between them. Also older CPUs don't have invariant TSC, so their speed changes when CPU frequency changes. (Some word of warning about Windows QueryPerformanceCounter: When you test it on your development laptop, it appears fast, because it's using RDTSC behind the scenes. But when it runs on a NUMA server, it often changes behavior and becomes 20-100x slower, because Windows starts to use HPET instead. RDTSC takes maybe ~10 ns to execute and it's not a shared resource. HPET takes 1-2 microseconds to read and is a shared system resource, concurrent access from multiple cores will make it slower.)

r4um 10y ago
That is the peak we see right now at ~20-30% utilization overall network/cpu wise, it can do much more easily. > Binary logging is a must, because even something like string formatting is simply way too slow, by an order of magnitude. true, string processing takes most of the resources.
- vardump 10y ago
  Timestamps are even worse. They're slow to get and have a multitude of other problems, like timezones, system clock adjustments (manual or NTP), rate drift, offset errors, etc. Some solve it by caching a low res timestamp, but that means losing ability to get side channel information about precisely how long time passed between two events. High resolution (micro/nanosecond range) timestamps give so much more information than just time.
  - r4um 10y ago
    We haven't dealt with those yet. But in general we are always pushing everyone to log judicially, use metrics wherever possible (numbers are still easier to manage than strings), use sentry for exceptions logging and introducing zipkin for tracing.
user5994461 10y ago
What usage could possibly need 5M log/s on a single system? Are you trying to log every TCP and UDP packet? To give a rough point of comparison, CloudFlare is taking 4M page views per second and that's 5% of the global internet traffic.
- vardump 10y ago
  > Are you trying to log every TCP and UDP packet? No. Although I've done that too... To debug timing critical code. Or Heisenbugs that disappear when you log. Ever had a bug that vanishes when you enable logging? Not 5M sustained, just short sub-millisecond bursts here and there.

chrissnell 10y ago

Heka proved to be the weak link in our logging stack. We get far better performance from fluentd. We use a multi-tiered fluentd architecture. One tier accepts logs from Docker and (optionally) sends a container's log stream to a developer-managed custom fluentd parser. From here, logs are sent to a fluentd "sink" for storage in Elasticsearch. Using multiple tiers helps prevent overload and acts as a buffer if ES slows down temporarily.

forgotpwtomain 10y ago
> Heka proved to be the weak link in our logging stack. We get far better performance from fluentd. I'd be curious to know some more details on this. I guess Go channels do copying rather than sharing, but I'd still expect Heka to perform better. Personally I've found Heka to be more robust - I've had to fix bugs in plugins for the former two and generally I haven't found them to be architecturally sound. For example with logstash I've had bugs in plugins which would crash the entire daemon -- that's just inexcusable for a core infrastructure service. The multi-tiered thing sounds very strange, why not just hold an on-disk buffer?
- r4um 10y ago
  Same for us fluentd was very resource hungry, heka brought down utilization significantly. Regarding the buffering we just added kafka in between for graylog/ES scaling/maintenance.
- chrissnell 10y ago
  Believe me, I wanted Heka to work. We're a big Go shop and I understood Heka better than any of the competing log shippers. We got very deep into Heka and ultimately, it just couldn't do the job. It performed very poorly when there was any sort of a bottleneck into Elasticsearch. It would enter an unrecoverable state when it got choked up We spent a lot of time on the Heka IRC channel talking to the devs. These were known issues. Some of the Heka team are/were working on a replacement written in C. It sounded promising but the ES output plugin did not exist yet so we couldn't use it. The need for a performant and powerful log shipper is still there. I hope to see some new options come around soon that can achieve 1MM+ lines/sec from a single daemon without requiring multiple tiers, receivers, etc.