3 ms·
For new projects that aren't already tied to syslog(), Apache Kafka is also an interesting library to look at. I've always felt that using UDP is an uneasy tra
by phunge 13y ago
For new projects that aren't already tied to syslog(), Apache Kafka is also an interesting library to look at.
I've always felt that using UDP is an uneasy tradeoff -- you're solving the problem of blocking, but at the cost of getting no batching whatsoever. When you're logging to disk, TCP, or memory, the fast path call to log just plops your data into a buffer and returns to you. More messages, the bigger your buffer gets. With UDP, each message instead goes straight to the NIC and out.
Kafka makes a different tradeoff -- buffer/batch to disk. Once you have enough data, compress it and ship it off reliably. If your consumer goes away, buffer as much as you can before losing messages; disk is cheap! The cost comes as small additional latency. In the fast path, it's quick (data stays RAM resident, and it's much cheaper to compress & batch send 1000 messages than 1000 UDPs).
- nbm 13y agoThe other alternatives specifically for logging are Scribe (which we use here at Facebook) and Apache Flume. As I understand the use case for this software, it would be talking UDP to a local syslog server, not over the network. At Facebook, we use scribe, and generally use (in-memory, in-app) batched writes to local scribe, and then batched writes (memory+disk, scribe daemon) from that local scribe to an upstream scribe storage system.