5 ms·

Graylog2: Java, Ruby, MongoDB-powered log management, monitoring, and alerting

30 points by mattyb 16y ago

viraptor 16y ago
It was fun to try out this software. It lacks some features in the interface though... With some more automatic processing and UI improvements, it could definitely be a low-end rival to splunk, but right now it can only do a standard search on custom attributes or whole text (not with a proper full-text index though) or show stuff "around that message". That means basics only. But with the development happening all the time, it's definitely a project worth keeping in bookmarks. If it can get some of the features available in octopussy without looking just as fugly, I'm in :)
devinfoley 16y ago
This looks great! Is anybody using it in production though? I can't find any case studies on their site.
- _lennart 16y ago
  yes. there are some startups and grown companies using it.
- aedocw 16y ago
  We are using it and loving it. More important than just shipping syslog to graylog2, we're using it to monitor 30+ servers in a cluster. Each server has a "health agent" that sends all the important metrics that happened in the last minute, packed into a GELF message. This could easily scale to 1000+ servers. On the monitoring side, one server dips into MongoDB to pull out the records and maintain records and graphs of everything that's going on (including sending alerts when a machine fails to check in as frequently as it should.) Graylog2 is pretty solid, and the people working on it are extremely responsive. If you suggest features that make sense, they'll probably be implemented within a few weeks (if not a few days!)
  - Corrado 16y ago
    Any chance you would open source that "health agent"? Or is it a part of Graylog2 already? I haven't checked it out yet, but I am pretty excited about using Graylog2 in our environment to replace some other monitoring and seeing things like CPU idle times and HD space remaining would be very helpful indeed. Bonus. I just watched a Railscast highlighting the Notification system in Rail3 and I can see where it would compliment Graylog2 very well. :)
  - kordless 16y ago
    Let me just caution those with highly scalable expectations. A large MongoDB instance is going to weight in at around 1-10TB. Here's an updated list of production MongoDB instances for reference: http://www.mongodb.org/display/DOCS/Production+Deployments http://www.mongodb.org/display/DOCS/Production+Deployments At both Splunk and Loggly I've seen customers sending in multi-tens-of-millions events a day from a handful of boxes without even breaking a sweat. It wouldn't take long to blow through several TB of storage in MongoDB if you were storing all that log data with a retention of a few months. Log volumes vary from use-case to use-case though, and I absolutely love the GrayLog2 guys and the way they listen to their users. It's definitely a great tool for a job that is usually a real pain in the ass!
- Moocar 16y ago
  We're using it. I was honestly convinced by their ridiculous tag line: "Manage your logs in the dark and have lasers going and make it look like you're from space". Awesome! Also, when you login it gives you status message like "mounting party hats", and "enraging gorilla". It's nice to have some comedy in what can potentially be such a dull arena.
deleted 16y ago
[deleted]
kordless 16y ago
Be sure to check out LogStash as well. Uses a similar approach with MongoDB, plus ElasticSearch for...search: http://code.google.com/p/logstash/ http://code.google.com/p/logstash/

30 points by mattyb 16y ago

It was fun to try out this software. It lacks some features in the interface though... With some more automatic processing and UI improvements, it could definitely be a low-end rival to splunk, but right now it can only do a standard search on custom attributes or whole text (not with a proper full-text index though) or show stuff "around that message". That means basics only. But with the development happening all the time, it's definitely a project worth keeping in bookmarks. If it can get some of the features available in octopussy without looking just as fugly, I'm in :)

devinfoley 16y ago

This looks great! Is anybody using it in production though? I can't find any case studies on their site.

_lennart 16y ago
yes. there are some startups and grown companies using it.
aedocw 16y ago
We are using it and loving it. More important than just shipping syslog to graylog2, we're using it to monitor 30+ servers in a cluster. Each server has a "health agent" that sends all the important metrics that happened in the last minute, packed into a GELF message. This could easily scale to 1000+ servers. On the monitoring side, one server dips into MongoDB to pull out the records and maintain records and graphs of everything that's going on (including sending alerts when a machine fails to check in as frequently as it should.) Graylog2 is pretty solid, and the people working on it are extremely responsive. If you suggest features that make sense, they'll probably be implemented within a few weeks (if not a few days!)
- Corrado 16y ago
  Any chance you would open source that "health agent"? Or is it a part of Graylog2 already? I haven't checked it out yet, but I am pretty excited about using Graylog2 in our environment to replace some other monitoring and seeing things like CPU idle times and HD space remaining would be very helpful indeed. Bonus. I just watched a Railscast highlighting the Notification system in Rail3 and I can see where it would compliment Graylog2 very well. :)
- kordless 16y ago
  Let me just caution those with highly scalable expectations. A large MongoDB instance is going to weight in at around 1-10TB. Here's an updated list of production MongoDB instances for reference: http://www.mongodb.org/display/DOCS/Production+Deployments http://www.mongodb.org/display/DOCS/Production+Deployments At both Splunk and Loggly I've seen customers sending in multi-tens-of-millions events a day from a handful of boxes without even breaking a sweat. It wouldn't take long to blow through several TB of storage in MongoDB if you were storing all that log data with a retention of a few months. Log volumes vary from use-case to use-case though, and I absolutely love the GrayLog2 guys and the way they listen to their users. It's definitely a great tool for a job that is usually a real pain in the ass!
Moocar 16y ago
We're using it. I was honestly convinced by their ridiculous tag line: "Manage your logs in the dark and have lasers going and make it look like you're from space". Awesome! Also, when you login it gives you status message like "mounting party hats", and "enraging gorilla". It's nice to have some comedy in what can potentially be such a dull arena.

deleted 16y ago

[deleted]

kordless 16y ago

Be sure to check out LogStash as well. Uses a similar approach with MongoDB, plus ElasticSearch for...search: http://code.google.com/p/logstash/ http://code.google.com/p/logstash/