5 ms·

Is there a lightweight solution for centralized logging with full text search and a nice UI? ELK seems like it requires some serious RAM and cores even if you

by nissimk 12y ago

Is there a lightweight solution for centralized logging with full text search and a nice UI? ELK seems like it requires some serious RAM and cores even if you are just talking about elasticsearch. Is there a similar solution that works well in just a couple of gigs of RAM for a slower message flow rate? Is anybody here successfully using ELK in a low ram environment with low message flow?

simonw 12y ago
Have you tried it yourself on your dataset? I've been experimenting with ElasticSearch on small datasets with a single node on my laptop and it doesn't seem to be hugely resource demanding.
ihsw 12y ago
Please define low RAM. I've got a t2.small server on AWS that's taking in a low/moderate logging from three servers (syslog x3, nginx x3, zero-low traffic redis) and it's usually in the 750MB of RAM with absolutely no notable CPU load. Granted Kibana/ElasticSearch gets pretty much no traffic other than checking it once or twice a day, just to get a glance at 4xx/5xx errors. So yeah, you just defined my current use-case down to the letter -- centralized logging, fancy interface for filtering/searching/visualizing said log data.
mbell 12y ago
I haven't found ELK to be terribly resource intensive. We're currently loading ~20,000 events / min into an ELK stack that runs on a single m1.xlarge box, we do use redis as a logstash broker to deal with unusually bursty load, but it doesn't seem to have much issue keeping up with normal load.
- robbles 12y ago
  Did you need a customized server configuration of Elasticsearch to enable this kind of throughput? And do you split your indexes up by day or some other criteria? We're running a similar configuration and I'd like to know the limitations before we'd need to start using a clustered setup.
  - mbell 12y ago
    Nothing special on the ES config. Indexes are split up by day and we avoid indexing fields with very high cardinality.
    - robbles 12y ago
      Thanks for the info. Does avoiding indexing high cardinality fields keep down the memory usage of an open index? I assume it would reduce the disk usage as well.
      - mbell 12y ago
        Memory usage and CPU time to maintain the index.
snewman 12y ago
If you're open to commercial solutions, there are several SaaS choices that are quite cheap at low log volume. For instance, Scalyr (500MB/day for $19/month), LogEntries (free up to 167MB/day), or Papertrail (33MB/day for $7/month). All will provide centralized logging and full text search, and some form of dashboards and alerting. Scalyr provides especially powerful features for log parsing and analysis, as well as integrating system and application metrics, and it's wicked fast -- most searches run in well under 1 second. (Disclosure: I am the founder of Scalyr.) If you're interested, the respective web sites are easy to find, or drop me a line (my email address is in my profile).
- 23david 12y ago
  For most teams, I definitely recommend getting started with a hosted elasticsearch option initially, and after validating a POC, only then consider a move to self-managed. Check out https://bonsai.io https://bonsai.io as a hosted option... I've helped a bunch of financial and security companies implement and deploy elk as a critical infra component. Generally I'm seeing it used to complement Splunk, Alienvault and other log/network/security monitoring solutions. Happy to help with any questions about deploying and managing ELK in production.
Iaks 12y ago
I am running a monolithic installation in a container (1 core, 4gb ram) that is being hosted by an intel NUC. I'm taking in about 100k records a day from 10-12 syslog sources and have no trouble with slow downs at all. (I do almost 0 processing of the incoming data, however.) I think I could lower the containers ram allocation to 2gb... maybe. I haven't tried aggregating a years worth of data into a single search, but for reasonable log troubleshooting everything is going well.
kiyoto 12y ago
Some people like using Grafana with InfluxDB backend and Fluentd for log collection.

by nissimk 12y ago

simonw 12y ago

Have you tried it yourself on your dataset? I've been experimenting with ElasticSearch on small datasets with a single node on my laptop and it doesn't seem to be hugely resource demanding.

ihsw 12y ago

Please define low RAM. I've got a t2.small server on AWS that's taking in a low/moderate logging from three servers (syslog x3, nginx x3, zero-low traffic redis) and it's usually in the 750MB of RAM with absolutely no notable CPU load. Granted Kibana/ElasticSearch gets pretty much no traffic other than checking it once or twice a day, just to get a glance at 4xx/5xx errors. So yeah, you just defined my current use-case down to the letter -- centralized logging, fancy interface for filtering/searching/visualizing said log data.

mbell 12y ago

I haven't found ELK to be terribly resource intensive. We're currently loading ~20,000 events / min into an ELK stack that runs on a single m1.xlarge box, we do use redis as a logstash broker to deal with unusually bursty load, but it doesn't seem to have much issue keeping up with normal load.

robbles 12y ago
Did you need a customized server configuration of Elasticsearch to enable this kind of throughput? And do you split your indexes up by day or some other criteria? We're running a similar configuration and I'd like to know the limitations before we'd need to start using a clustered setup.
- mbell 12y ago
  Nothing special on the ES config. Indexes are split up by day and we avoid indexing fields with very high cardinality.
  - robbles 12y ago
    Thanks for the info. Does avoiding indexing high cardinality fields keep down the memory usage of an open index? I assume it would reduce the disk usage as well.
    - mbell 12y ago
      Memory usage and CPU time to maintain the index.

snewman 12y ago

If you're open to commercial solutions, there are several SaaS choices that are quite cheap at low log volume. For instance, Scalyr (500MB/day for $19/month), LogEntries (free up to 167MB/day), or Papertrail (33MB/day for $7/month). All will provide centralized logging and full text search, and some form of dashboards and alerting. Scalyr provides especially powerful features for log parsing and analysis, as well as integrating system and application metrics, and it's wicked fast -- most searches run in well under 1 second. (Disclosure: I am the founder of Scalyr.) If you're interested, the respective web sites are easy to find, or drop me a line (my email address is in my profile).

23david 12y ago
For most teams, I definitely recommend getting started with a hosted elasticsearch option initially, and after validating a POC, only then consider a move to self-managed. Check out https://bonsai.io https://bonsai.io as a hosted option... I've helped a bunch of financial and security companies implement and deploy elk as a critical infra component. Generally I'm seeing it used to complement Splunk, Alienvault and other log/network/security monitoring solutions. Happy to help with any questions about deploying and managing ELK in production.

Iaks 12y ago

I am running a monolithic installation in a container (1 core, 4gb ram) that is being hosted by an intel NUC. I'm taking in about 100k records a day from 10-12 syslog sources and have no trouble with slow downs at all. (I do almost 0 processing of the incoming data, however.) I think I could lower the containers ram allocation to 2gb... maybe. I haven't tried aggregating a years worth of data into a single search, but for reasonable log troubleshooting everything is going well.

kiyoto 12y ago

Some people like using Grafana with InfluxDB backend and Fluentd for log collection.