4 ms·

Relatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too)

Relatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too) in its mode where it’s shipping logs to a central node? I’ve configured and used it with relatively small (high single digit nodes, bursts of activity to a million or two requests per minute or so) set-ups but have wondered if there’s a reason it’s not a more common solution for distributed logging and tracing (yes it doesn’t solve the UI problem for those, but it does solve collecting your logs) Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything.

nesarkvechnep 6mo ago
People don’t care about syslog. 98% of my colleagues haven’t heard of it.
- malux85 6mo ago
  You are drawing a global conclusion from a tiny sample!
  - nesarkvechnep 6mo ago
    I really hope that I am because I care about it and like to use it whenever I can.
- rrrix1 6mo ago
  Except every sysadmin and security engineer ever
  - nesarkvechnep 6mo ago
    Never met one who knew about syslog but I'm glad that there's a high chance I'm not right generally.
jbaiter 6mo ago
We're doing this with a few dozen GiBs of logs a day (rsylog -> central rsylog -> elasticsearch). It works reliably, but the config is an absolute nightmare, documentation is a mixed bag and troubleshooting often involves deep dives into the C code. We're planning to migrate to Alloy+Loki.
- baby_souffle 6mo ago
  Similar experience here as well. Syslog configs and plugins is a mess. Vector is not perfect but it’s got a decent amount of tooling and has native support for tests which is really useful
ehostunreach 6mo ago
Since this is an OTel-related submission, you could also use OTel collectors to collect and forward logs to a central OTel collector instance. > yes it doesn’t solve the UI problem for those, but it does solve collecting your logs I work for Netdata and over the last couple months, we've developed an external Netdata plugin that can ingest/index OTel logs [1]. The current implementation stores logs in systemd-compatible journal files and our visualization is effectively the same one someone would get when querying systemd journal logs [2]. i > Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything. I've not used rsyslogd specifically, but I don't see how you'd have any issues with the log volume you described. [1] https://github.com/netdata/netdata/tree/master/src/crates/netdata-otel/otel-plugin https://github.com/netdata/netdata/tree/master/src/crates/ne... [2] https://learn.netdata.cloud/docs/logs/systemd-journal-logs/systemd-journal-plugin-reference https://learn.netdata.cloud/docs/logs/systemd-journal-logs/s...

nesarkvechnep 6mo ago

People don’t care about syslog. 98% of my colleagues haven’t heard of it.

malux85 6mo ago
You are drawing a global conclusion from a tiny sample!
- nesarkvechnep 6mo ago
  I really hope that I am because I care about it and like to use it whenever I can.
rrrix1 6mo ago
Except every sysadmin and security engineer ever
- nesarkvechnep 6mo ago
  Never met one who knew about syslog but I'm glad that there's a high chance I'm not right generally.

jbaiter 6mo ago

We're doing this with a few dozen GiBs of logs a day (rsylog -> central rsylog -> elasticsearch). It works reliably, but the config is an absolute nightmare, documentation is a mixed bag and troubleshooting often involves deep dives into the C code. We're planning to migrate to Alloy+Loki.

baby_souffle 6mo ago
Similar experience here as well. Syslog configs and plugins is a mess. Vector is not perfect but it’s got a decent amount of tooling and has native support for tests which is really useful

ehostunreach 6mo ago

Since this is an OTel-related submission, you could also use OTel collectors to collect and forward logs to a central OTel collector instance. > yes it doesn’t solve the UI problem for those, but it does solve collecting your logs I work for Netdata and over the last couple months, we've developed an external Netdata plugin that can ingest/index OTel logs [1]. The current implementation stores logs in systemd-compatible journal files and our visualization is effectively the same one someone would get when querying systemd journal logs [2]. i > Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything. I've not used rsyslogd specifically, but I don't see how you'd have any issues with the log volume you described. [1] https://github.com/netdata/netdata/tree/master/src/crates/netdata-otel/otel-plugin https://github.com/netdata/netdata/tree/master/src/crates/ne... [2] https://learn.netdata.cloud/docs/logs/systemd-journal-logs/systemd-journal-plugin-reference https://learn.netdata.cloud/docs/logs/systemd-journal-logs/s...