4 ms·

We are using it and loving it. More important than just shipping syslog to graylog2, we're using it to monitor 30+ servers in a cluster. Each server has a "he

by aedocw 16y ago

We are using it and loving it. More important than just shipping syslog to graylog2, we're using it to monitor 30+ servers in a cluster. Each server has a "health agent" that sends all the important metrics that happened in the last minute, packed into a GELF message. This could easily scale to 1000+ servers. On the monitoring side, one server dips into MongoDB to pull out the records and maintain records and graphs of everything that's going on (including sending alerts when a machine fails to check in as frequently as it should.) Graylog2 is pretty solid, and the people working on it are extremely responsive. If you suggest features that make sense, they'll probably be implemented within a few weeks (if not a few days!)

Corrado 16y ago
Any chance you would open source that "health agent"? Or is it a part of Graylog2 already? I haven't checked it out yet, but I am pretty excited about using Graylog2 in our environment to replace some other monitoring and seeing things like CPU idle times and HD space remaining would be very helpful indeed. Bonus. I just watched a Railscast highlighting the Notification system in Rail3 and I can see where it would compliment Graylog2 very well. :)
kordless 16y ago
Let me just caution those with highly scalable expectations. A large MongoDB instance is going to weight in at around 1-10TB. Here's an updated list of production MongoDB instances for reference: http://www.mongodb.org/display/DOCS/Production+Deployments http://www.mongodb.org/display/DOCS/Production+Deployments At both Splunk and Loggly I've seen customers sending in multi-tens-of-millions events a day from a handful of boxes without even breaking a sweat. It wouldn't take long to blow through several TB of storage in MongoDB if you were storing all that log data with a retention of a few months. Log volumes vary from use-case to use-case though, and I absolutely love the GrayLog2 guys and the way they listen to their users. It's definitely a great tool for a job that is usually a real pain in the ass!

by aedocw 16y ago

Corrado 16y ago

Any chance you would open source that "health agent"? Or is it a part of Graylog2 already? I haven't checked it out yet, but I am pretty excited about using Graylog2 in our environment to replace some other monitoring and seeing things like CPU idle times and HD space remaining would be very helpful indeed. Bonus. I just watched a Railscast highlighting the Notification system in Rail3 and I can see where it would compliment Graylog2 very well. :)

kordless 16y ago

Let me just caution those with highly scalable expectations. A large MongoDB instance is going to weight in at around 1-10TB. Here's an updated list of production MongoDB instances for reference: http://www.mongodb.org/display/DOCS/Production+Deployments http://www.mongodb.org/display/DOCS/Production+Deployments At both Splunk and Loggly I've seen customers sending in multi-tens-of-millions events a day from a handful of boxes without even breaking a sweat. It wouldn't take long to blow through several TB of storage in MongoDB if you were storing all that log data with a retention of a few months. Log volumes vary from use-case to use-case though, and I absolutely love the GrayLog2 guys and the way they listen to their users. It's definitely a great tool for a job that is usually a real pain in the ass!