5 ms·

logstash + elasticsearch + kibana That's for personal projects, the CloudFlare setup is a little more complex and perhaps one of the data team would be best an

by buro9 12y ago

logstash + elasticsearch + kibana That's for personal projects, the CloudFlare setup is a little more complex and perhaps one of the data team would be best answering that... if you're interested then I can ping them to see if there's a volunteer for a blog post describing how we do logs at scale.

hckrt_ 12y ago
That would be very helpful indeed...thanks in advance!
johnbellone 12y ago
An ELK stack is definitely a lot easier and straight forward to setup. Since we need to support a dozen or so teams with varying performance considerations we found that logstash left a lot to be desired. In the end fluentd performed well and gave us a lot of flexibility.
- avtar 12y ago
  > we found that logstash left a lot to be desired. In the end fluentd performed well and gave us a lot of flexibility. Would you mind providing more details about why you felt that Logstash was lacking?
- kiyoto 12y ago
  Glad to hear that! I wrote https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization https://www.digitalocean.com/community/tutorials/elasticsear... a while back.
- provost 12y ago
  I'd like your view from the trenches regarding Logstash vs Fluentd, specifically why it's better for 12+ teams? I'm having to make a similar decision myself and would enjoy your insight! Thanks
  - kiyoto 12y ago
    (Disclaimer: I am a Fluentd maintainer) Here are some comparisons out in the wild (Note: neither of them is a Logstash or Fluentd maintainer afaik) - http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/ http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/ - https://blog.deimos.fr/2014/05/13/logstash-vs-fluentd/ https://blog.deimos.fr/2014/05/13/logstash-vs-fluentd/ Also, if you have any question or doubt about Fluentd, please feel free to email me at kiyoto@treasure-data.com
frakkingcylons 12y ago
I'd be very interested to read that kind of post.
mdekkers 12y ago
A post like that would be awesome

by buro9 12y ago

hckrt_ 12y ago

That would be very helpful indeed...thanks in advance!

johnbellone 12y ago

An ELK stack is definitely a lot easier and straight forward to setup. Since we need to support a dozen or so teams with varying performance considerations we found that logstash left a lot to be desired. In the end fluentd performed well and gave us a lot of flexibility.

avtar 12y ago
> we found that logstash left a lot to be desired. In the end fluentd performed well and gave us a lot of flexibility. Would you mind providing more details about why you felt that Logstash was lacking?
kiyoto 12y ago
Glad to hear that! I wrote https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization https://www.digitalocean.com/community/tutorials/elasticsear... a while back.
provost 12y ago
I'd like your view from the trenches regarding Logstash vs Fluentd, specifically why it's better for 12+ teams? I'm having to make a similar decision myself and would enjoy your insight! Thanks
- kiyoto 12y ago
  (Disclaimer: I am a Fluentd maintainer) Here are some comparisons out in the wild (Note: neither of them is a Logstash or Fluentd maintainer afaik) - http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/ http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/ - https://blog.deimos.fr/2014/05/13/logstash-vs-fluentd/ https://blog.deimos.fr/2014/05/13/logstash-vs-fluentd/ Also, if you have any question or doubt about Fluentd, please feel free to email me at kiyoto@treasure-data.com

frakkingcylons 12y ago

I'd be very interested to read that kind of post.

mdekkers 12y ago

A post like that would be awesome