5 ms·

I share your viewpoint and am currently thinking through a similiar small-scale deployment. How do you handle logging? For single boxes I'd just use logwatch an

by bauerd 9y ago

I share your viewpoint and am currently thinking through a similiar small-scale deployment. How do you handle logging? For single boxes I'd just use logwatch and friends, but for aggregating log output (both system/application logs) from a small-ish Consul cluster I feel the options are complete overkill. I've had a look at ELK, time-series backed stuff (e.g. Prometheus), but all I really want is log aggregation with search capability and optionally Regexp-based alerting. It seems to me Logstash provides what I want, but I sure as hell won't run 3 JVMs and a Redis instance to aggregate logs. tl;dr How to handle log aggregation within a small-scale cluster without losing your sanity?

kaitnieks 9y ago
For simple, non-critical stuff this is the simplest solution I've come up with - if you don't need real-time aggregated logs (i.e. you only want to archive them) and you have log rotation configured, you can simply use cron with "aws s3 sync" or rsync for the logs folder.
greenleafjacob 9y ago
Check out papertrail.
hn_user2 9y ago
For small stuff I still like the simple syslog. It’s usually built in and supports aggregating to a single server. In the past have created a server just with syslog. Adjust all your servers syslog configs to point to that one. Then log in and use grep. Not fancy. But gets the job done with a single line config change in your syslog configs.
- arca_vorago 9y ago
  Senior sysadmin here, and I agree this is one of the most common methods. I have also had great success tying in the ossec or pam module notifications into syslog messages that all go to the central syslog (nsyslog/rsyslog etc) server. The problem with the elk and similar stacks imho is lack of security and speed. Things this old school setup doesn't have a problem with. The problem is that too many managers don't like not having gui dashboards... which is why splunk et al have really taken off. This re-enforces my idea that a purely terminal based business dashboard might be a cool product with a fairly large market. I've been eyeballing some of the go/ncurses work for this.
- devonkim 9y ago
  rsyslog or syslog-ng were preferred these days for better security and performance I thought? Unless you’re using something like stunnel to wrap the syslog messages that is.
ibotty 9y ago
Monitoring and logging are best separated. Prometheus is great even on a small scale. For log aggregation there is the relatively new oklog. I have been running oklog for a customer successfully. You ought to deploy it like a regular (non-cloud-native) service though: When a server crashes, restore the oklog instance from backups, not spinning a new one.
AzMoo_ 9y ago
I just have a server dedicated to syslog and have rsyslog running on everything else.
mrweasel 9y ago
We're kinda big on Splunk, everything that remotely looks like log data get shipped of to Splunk. I've worked with ELK before and I'm not a fan, the usability is very low compared to Splunk. It's much cheaper though. At a previous employer we switch from Splunk to ELK and the number of searches we did on a daily basis dropped to almost zero. Before that almost every support "ticket" would start with a Splunk search. You could also try out https://www.humio.com https://www.humio.com. I only seen it deploy by one customer, but it looks nice.
- jsmeaton 9y ago
  We use Sumologic and it’s really nice. Much better than ELK. I’ve only used splunk in a evaluation context a long time ago, so I’m not sure how they’d compare. Splunk is just so expensive.
  - kondor6c 9y ago
    I found Sumologic to lack features that ELK has such as packetbeats. But you still have to manage the Sumologic forwarding agent and keep under your ingest limit. Additionally my mind never exactly flowed with their query syntax. They refreshed the UI recently and I liked it (aside from compounding my overutlization of tabs haha), you now more easily/graphically select time series from the graph. I can see how it is nice to not have to worry about re-indexing and what type of device the underlying data rests on. I have not tried the ELK hosted solution so some of my criticisms could apply to it as well.
- Carasso 9y ago
  We use Humio alongside Splunk and they have an office in SF. Nice people, price is nice too :)
devonkim 9y ago
If you’re not violently opposed to MongoDB Graylog is very popular. I haven’t heard bad things about it so far besides the MongoDB portion but log aggregation is one of those systems that may be ok with some hiccups from time to time as opposed to an actual primary business data store.

by bauerd 9y ago

kaitnieks 9y ago

For simple, non-critical stuff this is the simplest solution I've come up with - if you don't need real-time aggregated logs (i.e. you only want to archive them) and you have log rotation configured, you can simply use cron with "aws s3 sync" or rsync for the logs folder.

greenleafjacob 9y ago

Check out papertrail.

hn_user2 9y ago

For small stuff I still like the simple syslog. It’s usually built in and supports aggregating to a single server. In the past have created a server just with syslog. Adjust all your servers syslog configs to point to that one. Then log in and use grep. Not fancy. But gets the job done with a single line config change in your syslog configs.

arca_vorago 9y ago
Senior sysadmin here, and I agree this is one of the most common methods. I have also had great success tying in the ossec or pam module notifications into syslog messages that all go to the central syslog (nsyslog/rsyslog etc) server. The problem with the elk and similar stacks imho is lack of security and speed. Things this old school setup doesn't have a problem with. The problem is that too many managers don't like not having gui dashboards... which is why splunk et al have really taken off. This re-enforces my idea that a purely terminal based business dashboard might be a cool product with a fairly large market. I've been eyeballing some of the go/ncurses work for this.
devonkim 9y ago
rsyslog or syslog-ng were preferred these days for better security and performance I thought? Unless you’re using something like stunnel to wrap the syslog messages that is.

ibotty 9y ago

Monitoring and logging are best separated. Prometheus is great even on a small scale. For log aggregation there is the relatively new oklog. I have been running oklog for a customer successfully. You ought to deploy it like a regular (non-cloud-native) service though: When a server crashes, restore the oklog instance from backups, not spinning a new one.

AzMoo_ 9y ago

I just have a server dedicated to syslog and have rsyslog running on everything else.

mrweasel 9y ago

We're kinda big on Splunk, everything that remotely looks like log data get shipped of to Splunk. I've worked with ELK before and I'm not a fan, the usability is very low compared to Splunk. It's much cheaper though. At a previous employer we switch from Splunk to ELK and the number of searches we did on a daily basis dropped to almost zero. Before that almost every support "ticket" would start with a Splunk search. You could also try out https://www.humio.com https://www.humio.com. I only seen it deploy by one customer, but it looks nice.

jsmeaton 9y ago
We use Sumologic and it’s really nice. Much better than ELK. I’ve only used splunk in a evaluation context a long time ago, so I’m not sure how they’d compare. Splunk is just so expensive.
- kondor6c 9y ago
  I found Sumologic to lack features that ELK has such as packetbeats. But you still have to manage the Sumologic forwarding agent and keep under your ingest limit. Additionally my mind never exactly flowed with their query syntax. They refreshed the UI recently and I liked it (aside from compounding my overutlization of tabs haha), you now more easily/graphically select time series from the graph. I can see how it is nice to not have to worry about re-indexing and what type of device the underlying data rests on. I have not tried the ELK hosted solution so some of my criticisms could apply to it as well.
Carasso 9y ago
We use Humio alongside Splunk and they have an office in SF. Nice people, price is nice too :)

devonkim 9y ago

If you’re not violently opposed to MongoDB Graylog is very popular. I haven’t heard bad things about it so far besides the MongoDB portion but log aggregation is one of those systems that may be ok with some hiccups from time to time as opposed to an actual primary business data store.