4 ms·

I'm a big believer in logging, and I find it hard to believe that collecting and monitoring logs needs to have a server with 32GB of RAM and massive Java and El

I'm a big believer in logging, and I find it hard to believe that collecting and monitoring logs needs to have a server with 32GB of RAM and massive Java and ElasticSearch back-ends like ELK, Splunk, Graylog, or similar monitoring software. I've been searching for something self-hosted that can run in my server's spare capacity and monitor the OS and application logs with a simple searching interface (just a web interface to grep would be handy) and come up short. If I can't host it on the cheapest DO plan, it's out of my budget. I really don't want to have to build it myself but I'm just about at that point.

mseebach 9y ago
Sounds like you want something like syslog-ng paired with Nagios. The "big" solutions require some hardware because they do a lot more than merely collecting your logs.
Artemis2 9y ago
oklog might be for you? https://github.com/oklog/oklog https://github.com/oklog/oklog It’s still new, but the architecture is sound and it worked well for me on a small single-node deployment (about 30 clients sending logs).
- Blindedwino 9y ago
  It looks promising. Now if it could integrate easily with syslog-ng/rsyslog, that would be perfect.
hobofan 9y ago
I used to do something similar (but a bit less fancy), with logs processed by Heka + bash scripts using jq. Heka has now been phased out in favor of hindsight[0], but I haven't personally tried that one out yet. [0]: https://github.com/mozilla-services/hindsight https://github.com/mozilla-services/hindsight
odonnellryan 9y ago
Nagios is not-so-easy to configure, but probably will get you most of the way there.
- Blindedwino 9y ago
  Nagios doesn't do log monitoring out of the box though. Unless I've missed that? It definitely doesn't aggregate logs.
  - odonnellryan 9y ago
    Nagios has check_log but that is not great for memory, I believe it loads the entire log file into memory to take a diff to only check on the entries missed. There is check_logfile which has a similar name but is a different library, I wrote about how to set this up a while ago: https://medium.com/luma-consulting/how-to-install-check-logfiles-on-ubuntu-linux-to-work-with-nagios-23e8b034a8a5 https://medium.com/luma-consulting/how-to-install-check-logf... It's not super obvious how this plugin works unfortunately. Not the craziest thing in the world to configure but certainly not easy starting from zero!

mseebach 9y ago

Sounds like you want something like syslog-ng paired with Nagios. The "big" solutions require some hardware because they do a lot more than merely collecting your logs.

Artemis2 9y ago

oklog might be for you? https://github.com/oklog/oklog https://github.com/oklog/oklog It’s still new, but the architecture is sound and it worked well for me on a small single-node deployment (about 30 clients sending logs).

Blindedwino 9y ago
It looks promising. Now if it could integrate easily with syslog-ng/rsyslog, that would be perfect.

hobofan 9y ago

I used to do something similar (but a bit less fancy), with logs processed by Heka + bash scripts using jq. Heka has now been phased out in favor of hindsight[0], but I haven't personally tried that one out yet. [0]: https://github.com/mozilla-services/hindsight https://github.com/mozilla-services/hindsight

odonnellryan 9y ago

Nagios is not-so-easy to configure, but probably will get you most of the way there.

Blindedwino 9y ago
Nagios doesn't do log monitoring out of the box though. Unless I've missed that? It definitely doesn't aggregate logs.
- odonnellryan 9y ago
  Nagios has check_log but that is not great for memory, I believe it loads the entire log file into memory to take a diff to only check on the entries missed. There is check_logfile which has a similar name but is a different library, I wrote about how to set this up a while ago: https://medium.com/luma-consulting/how-to-install-check-logfiles-on-ubuntu-linux-to-work-with-nagios-23e8b034a8a5 https://medium.com/luma-consulting/how-to-install-check-logf... It's not super obvious how this plugin works unfortunately. Not the craziest thing in the world to configure but certainly not easy starting from zero!