2 ms·

Whenever I have to deal with logs, it's either: a) simplicity of rsyslog b) monstruosity of ELK | Grafana | etc. Somehow I like Prometheus (I think it's "sim

by danwee 3y ago

Whenever I have to deal with logs, it's either: a) simplicity of rsyslog b) monstruosity of ELK | Grafana | etc. Somehow I like Prometheus (I think it's "simple"), but it's not enough to display and search for logs. Somehow, none of the companies I have worked for, have used "simple tools" like rsyslog to handle logs. They all used cloud (Datadog, New relic) or self hosted (ELK, Prometheus + Grafana). I wonder why (I guess it's because "money buys you simplicity") I just want the following: - on each machine I want to get logs from: install the agent (a simple binary) + simple /etc/myagent.conf. The agent forwards logs to my "main log server" - on my "main log server": install the "log processor" (again, just a binary please!) + simple /etc/mylogprocessor.conf. The "log processor" shows me a nice localhost:9090/ web interface in which I can search for logs (indexed by any field I want). Easy, no? My use case is not thousands of machines nor Terabytes of data logs per second. I just have a few machines and I don't want to deal with multi-clustered solutions or anything like that. Just 2 binaries! Does that exist?

ilyt 3y ago
We just use rsyslog to send to ELK instance but it's less than perfect and it doesn't log everything we want to coz not every app have very good login. The problem I have encountered that even "simple" (just my home NAS + few devices) setups require some log mungling to get useful info into whatever system uses it. Many apps don't have "log in JSON" option in the first place, and near-always there is no real standard in fields of that message either. And also near-always I want to filter out or rate-limit some particularly spammy message or service just because I don't even want to look at it when browsing logs as it is just noise > Easy, no? My use case is not thousands of machines nor Terabytes of data logs per second. I just have a few machines and I don't want to deal with multi-clustered solutions or anything like that. Just 2 binaries! Does that exist? ...graylog I guess ? I looked at it and it is apparently pretty integrated, but price on higher volumes made us do ELK on "actual big stuff"

by danwee 3y ago

ilyt 3y ago

We just use rsyslog to send to ELK instance but it's less than perfect and it doesn't log everything we want to coz not every app have very good login. The problem I have encountered that even "simple" (just my home NAS + few devices) setups require some log mungling to get useful info into whatever system uses it. Many apps don't have "log in JSON" option in the first place, and near-always there is no real standard in fields of that message either. And also near-always I want to filter out or rate-limit some particularly spammy message or service just because I don't even want to look at it when browsing logs as it is just noise > Easy, no? My use case is not thousands of machines nor Terabytes of data logs per second. I just have a few machines and I don't want to deal with multi-clustered solutions or anything like that. Just 2 binaries! Does that exist? ...graylog I guess ? I looked at it and it is apparently pretty integrated, but price on higher volumes made us do ELK on "actual big stuff"