4 ms·
I do that daily. Most of the services we run consist of 1 - 8 application servers ( often closer to 2 than 8), so things like Docker doesn't make much sense. Ev
by mrweasel 9y ago
I do that daily. Most of the services we run consist of 1 - 8 application servers ( often closer to 2 than 8), so things like Docker doesn't make much sense. Even though we of cause try to automate as much as possible, using things like Ansible, we often log in to servers directly to verify changes. Database servers are normally manually managed, to some extend, so we'll always login directly.
When I'm on-call and get an alarm on a server, checking that colleagues aren't logged in is normally the first thing you do. If a service fails it's more often than not someone doing maintenance, and they just forgot to tell you, or disable monitoring. And as someone else pointed out, also check disk usage.
Hacker News can be a little blind to the fact that most software projects are rather small and modern servers are really powerful. It's actually pretty rare that someone manages enough infrastructure for a single service that logging isn't a viable option.
- bauerd 9y agoI share your viewpoint and am currently thinking through a similiar small-scale deployment. How do you handle logging? For single boxes I'd just use logwatch and friends, but for aggregating log output (both system/application logs) from a small-ish Consul cluster I feel the options are complete overkill. I've had a look at ELK, time-series backed stuff (e.g. Prometheus), but all I really want is log aggregation with search capability and optionally Regexp-based alerting. It seems to me Logstash provides what I want, but I sure as hell won't run 3 JVMs and a Redis instance to aggregate logs. tl;dr How to handle log aggregation within a small-scale cluster without losing your sanity?
- kaitnieks 9y agoFor simple, non-critical stuff this is the simplest solution I've come up with - if you don't need real-time aggregated logs (i.e. you only want to archive them) and you have log rotation configured, you can simply use cron with "aws s3 sync" or rsync for the logs folder.
- greenleafjacob 9y agoCheck out papertrail.
- hn_user2 9y agoFor small stuff I still like the simple syslog. It’s usually built in and supports aggregating to a single server. In the past have created a server just with syslog. Adjust all your servers syslog configs to point to that one. Then log in and use grep. Not fancy. But gets the job done with a single line config change in your syslog configs.
- arca_vorago 9y agoSenior sysadmin here, and I agree this is one of the most common methods. I have also had great success tying in the ossec or pam module notifications into syslog messages that all go to the central syslog (nsyslog/rsyslog etc) server. The problem with the elk and similar stacks imho is lack of security and speed. Things this old school setup doesn't have a problem with. The problem is that too many managers don't like not having gui dashboards... which is why splunk et al have really taken off. This re-enforces my idea that a purely terminal based business dashboard might be a cool product with a fairly large market. I've been eyeballing some of the go/ncurses work for this.
- devonkim 9y agorsyslog or syslog-ng were preferred these days for better security and performance I thought? Unless you’re using something like stunnel to wrap the syslog messages that is.
- ibotty 9y agoMonitoring and logging are best separated. Prometheus is great even on a small scale. For log aggregation there is the relatively new oklog. I have been running oklog for a customer successfully. You ought to deploy it like a regular (non-cloud-native) service though: When a server crashes, restore the oklog instance from backups, not spinning a new one.
- AzMoo_ 9y agoI just have a server dedicated to syslog and have rsyslog running on everything else.
- mrweasel 9y agoWe're kinda big on Splunk, everything that remotely looks like log data get shipped of to Splunk. I've worked with ELK before and I'm not a fan, the usability is very low compared to Splunk. It's much cheaper though. At a previous employer we switch from Splunk to ELK and the number of searches we did on a daily basis dropped to almost zero. Before that almost every support "ticket" would start with a Splunk search. You could also try out https://www.humio.com https://www.humio.com. I only seen it deploy by one customer, but it looks nice.
- jsmeaton 9y agoWe use Sumologic and it’s really nice. Much better than ELK. I’ve only used splunk in a evaluation context a long time ago, so I’m not sure how they’d compare. Splunk is just so expensive.
- kondor6c 9y agoI found Sumologic to lack features that ELK has such as packetbeats. But you still have to manage the Sumologic forwarding agent and keep under your ingest limit. Additionally my mind never exactly flowed with their query syntax. They refreshed the UI recently and I liked it (aside from compounding my overutlization of tabs haha), you now more easily/graphically select time series from the graph. I can see how it is nice to not have to worry about re-indexing and what type of device the underlying data rests on. I have not tried the ELK hosted solution so some of my criticisms could apply to it as well.
- Carasso 9y agoWe use Humio alongside Splunk and they have an office in SF. Nice people, price is nice too :)
- devonkim 9y agoIf you’re not violently opposed to MongoDB Graylog is very popular. I haven’t heard bad things about it so far besides the MongoDB portion but log aggregation is one of those systems that may be ok with some hiccups from time to time as opposed to an actual primary business data store.
- technion 9y agoEven the word "small" is quite relative. One of my clients is big enough that you've probably heard of them, and their website/app (which is the company's only thing) is one LAMP server. That was only last year upgraded from a Windows 2003 WAMP server on a desktop. I tell friends what I did there and they say "wow you must be really good at Puppet at scale" based on assumptions.
- thunfischbrot 9y agoMost people have yet to discover what a tiny little machine can do if you spend some time addressing bottlenecks. I have fond memories of running websites on the cheapest VPSs I could find, and getting very respectable performance from them with little more than cutting down on resource-hungry services, going static html whenever possible and being vigilant about any resources loaded by the browser. See also https://lowendbox.com/ https://lowendbox.com/ and https://hackernoon.com/10-things-i-learned-making-the-fastest-site-in-the-world-18a0e1cdf4a7 https://hackernoon.com/10-things-i-learned-making-the-fastes...
- ryandrake 9y agoI run a couple of very small scale (mostly static content) web sites and E-mail serving and a few other services on a single "Cheapest VPS" and have never gotten interested in Puppet or Ansible or any of the other things mentioned in this thread. It just seems like adding abstraction and automation on top of things that are not worth abstracting or automating. To me, these things are Yet Another Software That Could Fail. When I need to change a configuration file, I ssh in, sudo vi /etc/blahblah.conf and get on with my life.
- theonealtair 9y agoI also was never interested in Puppet, however I was only interested in Ansible, and only use it as sort of an automated check list when setting up my server, in case I need to rebuild the server, the tedious steps can be automated. But I've never used it for multiple server management.
- 9y ago