4 ms·

Great article based on real life experience. I have been building logging and protective monitoring pipelines for a while now. From my experience if it comes t

Great article based on real life experience. I have been building logging and protective monitoring pipelines for a while now. From my experience if it comes to log shipping from hosts rsyslog + relp + disk assited in-memory asynchronous queues are preferred, most of the time you just only have network i/o as logs would not touch disk. The idea is to ship logs off the device ASAP as well as destination acts as a sink server capable to handle most of the spikes withouth stressing local source. All done via rsyslog which also wraps actual logs into json format locally. The glue could be syslog-tag. At the other end you could have ELK stack and logstash using json_lines codec input (pretty fast) structuring data further to your likings. Just looking into metrics now the avg time for logs showing in ELK is 7-200ms (the latency comes mostly from specific reads happening against the ES cluster). As ELK is always the slowest component, dropping logs compressed in-memory directly onto disk is also an option. One thing to note is that RELP can produce extra duplicates which are easily handled by inserting into Elasticsearch using specific document ID (some performance penalty) which could be some unique hash computed on (log content, timestamp, host) etc. With this in place you can also easily "replay" stream of logs to fill potential gaps. This type of setup scales really good as well. Edit: typos

wwright 6y ago
We have a similar setup, but I've been generating UUIDs for each event at logging time to avoid any computational overhead for deduping.
- cbsmith 6y ago
  Unless they are type1 uuid's, there's probably just as much computational overhead going on in the uuid creation. ;-)
  - wwright 6y ago
    Well, in our case, the event generators are almost never CPU bound but the logstash nodes are. Generating a UUID (which is mostly just waiting for the kernel to hand you some bytes, which allows other threads time to do useful work) before sending has no opportunity cost and frees up the more-valuable logstash resources. (These are micro-optimizations anyway.)
    - cbsmith 6y ago
      Yeah... I'd definitely remove logstash from that pipeline for that reason.
- fluential 6y ago
  The way to approach that in logstash is to use fingerprint plugin which can do various formats (including uuids), MURMUR3 seems to be the fastest here. https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html https://www.elastic.co/guide/en/logstash/current/plugins-fil.... Edit: typo
cbsmith 6y ago
Except you can scale up and beat that performance significantly if you just the standard syslog wire protocol to a central log server. That avoids a lot of independent append operations that you are counting on staying in the OS buffer despite whatever other IO might be going on, and you're also burning a lot of CPU (and space) transforming things to JSON using an inefficient JRuby runtime. We make this stuff harder by rebuilding functionality on top of systems that already have the functionality.
- foota 6y ago
  That works fine until your centralized server falls over, either because you're scaling to hundreds or thousands of machines and your logs are chatty or someone goofs and starts logging kilobytes of data on every request.
  - cbsmith 6y ago
    If you care about that, it's very easy to have redundancy there. In fact, the target pipe you route to on each container/system/etc. can be fully virtualized and routed as needed by... syslog. ;-) If you want to go crazy you can wrap it in rate limiting which is done for you by... syslog.
  - fluential 6y ago
    There would be usually multiple syslog sink servers so there is never a single point of failure. This can be configured in various ways either dns round robin, load balancers or even multiple destinations configured on the client side, rsyslog can iterate over a list in case of failures. If there is log storm coming from misconfigured app depending on your traffic levels the sink servers should give you a lot of space for an on-call engineer to be notified that something is off and adjust / rate limit / drop accordingly. Edit: udated with more info
- fluential 6y ago
  RELP is very much standard syslog wire protocol(which is just plain tcp anyway), but with some extra stuff to avoid losing data. Just fixing tcp issue here, where data could be ACK-ed into OS receive buffer and the app could be killed before offloading into a persisten storage. Not losing data is very important in log shipping. The json that syslog wraps data in is actually using the same syslog protocol - nothing is chaging here. It's encapsulated within the same protocol. The only reason to unwrap json is when you need to start structuring your data for BI views, your milage may vary. The key is that these two are de-coupled. Because you work on a basic level networking / storage layers its very easy to reason and build reliable pipelines according to your needs. As you say — you can't beat that. PS: The pain starts when you need to deal with multiline logs like java stack traces but then you just work with devs to log everything into json. Edit: split comment, more info

wwright 6y ago

We have a similar setup, but I've been generating UUIDs for each event at logging time to avoid any computational overhead for deduping.

cbsmith 6y ago
Unless they are type1 uuid's, there's probably just as much computational overhead going on in the uuid creation. ;-)
- wwright 6y ago
  Well, in our case, the event generators are almost never CPU bound but the logstash nodes are. Generating a UUID (which is mostly just waiting for the kernel to hand you some bytes, which allows other threads time to do useful work) before sending has no opportunity cost and frees up the more-valuable logstash resources. (These are micro-optimizations anyway.)
  - cbsmith 6y ago
    Yeah... I'd definitely remove logstash from that pipeline for that reason.
fluential 6y ago
The way to approach that in logstash is to use fingerprint plugin which can do various formats (including uuids), MURMUR3 seems to be the fastest here. https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html https://www.elastic.co/guide/en/logstash/current/plugins-fil.... Edit: typo

cbsmith 6y ago

Except you can scale up and beat that performance significantly if you just the standard syslog wire protocol to a central log server. That avoids a lot of independent append operations that you are counting on staying in the OS buffer despite whatever other IO might be going on, and you're also burning a lot of CPU (and space) transforming things to JSON using an inefficient JRuby runtime. We make this stuff harder by rebuilding functionality on top of systems that already have the functionality.

foota 6y ago
That works fine until your centralized server falls over, either because you're scaling to hundreds or thousands of machines and your logs are chatty or someone goofs and starts logging kilobytes of data on every request.
- cbsmith 6y ago
  If you care about that, it's very easy to have redundancy there. In fact, the target pipe you route to on each container/system/etc. can be fully virtualized and routed as needed by... syslog. ;-) If you want to go crazy you can wrap it in rate limiting which is done for you by... syslog.
- fluential 6y ago
  There would be usually multiple syslog sink servers so there is never a single point of failure. This can be configured in various ways either dns round robin, load balancers or even multiple destinations configured on the client side, rsyslog can iterate over a list in case of failures. If there is log storm coming from misconfigured app depending on your traffic levels the sink servers should give you a lot of space for an on-call engineer to be notified that something is off and adjust / rate limit / drop accordingly. Edit: udated with more info
fluential 6y ago
RELP is very much standard syslog wire protocol(which is just plain tcp anyway), but with some extra stuff to avoid losing data. Just fixing tcp issue here, where data could be ACK-ed into OS receive buffer and the app could be killed before offloading into a persisten storage. Not losing data is very important in log shipping. The json that syslog wraps data in is actually using the same syslog protocol - nothing is chaging here. It's encapsulated within the same protocol. The only reason to unwrap json is when you need to start structuring your data for BI views, your milage may vary. The key is that these two are de-coupled. Because you work on a basic level networking / storage layers its very easy to reason and build reliable pipelines according to your needs. As you say — you can't beat that. PS: The pain starts when you need to deal with multiline logs like java stack traces but then you just work with devs to log everything into json. Edit: split comment, more info