4 ms·

I was excited and hopeful to try the ELK stack a while back, but, like all the other comments here, have decided it was too unreliable and brittle. I have an o

by clebio 11y ago

I was excited and hopeful to try the ELK stack a while back, but, like all the other comments here, have decided it was too unreliable and brittle. I have an open issue on Logstash, spent a fair amount of time detailing it, but have gotten no feedback. And then I realized there's 600+ open issues! https://github.com/elastic/logstash/issues/4389 https://github.com/elastic/logstash/issues/4389 I'm considering using Syslog-ng and would love to hear if anyone has comments on that. Based on other comments here, will be checking out Riemann and Fluentd as well. https://syslog-ng.org/ https://syslog-ng.org/

dozzie 11y ago
Don't get too excited about Riemann. It's a stream processing engine, not data transport. It has its uses, but not for shipping logs. syslog-ng and its competitor, rsyslog, are fine, but they are concentrated around logs specifically. Fluentd (and logstash) can be used to transport other data, too, like monitoring or inventory.
- clebio 11y ago
  Good info (from jedisct1 as well) -- the distinctions are indeed important!
jedisct1 11y ago
Fluentd is great, and works very well to inject logs into Elasticsearch. We were using it at OpenDNS. We didn't even try Logstash, as Fluentd did the job. At OVH, both turned out to be way too slow, so we wrote Flowgger which is heavily used by all our services: https://github.com/jedisct1/flowgger https://github.com/jedisct1/flowgger
Xylakant 11y ago
I'm not an elastic person, but I can shed some light on this: You're holding it wrong. It's not a bug. You have multiple config files in one directory - if you do that, all those files are combined to one, that means that each event gets handed to each of your individual outputs - multiplying the message. See https://www.elastic.co/guide/en/logstash/current/command-line-flags.html https://www.elastic.co/guide/en/logstash/current/command-lin... Feel free to hop on the IRC if you have further questions, there's usually somebody qualified to answer.
- clebio 11y ago
  I appreciate the help, though isn't the point of `/etc//conf.d` directories generally that you have multiple config files? This is a common idiom that other packages handle correctly (differently?). I have hopped on the logstash IRC at times to ask about some of this, though I guess not this exact item. In fact, there's a different (well-known issue) that the init script for logstash has the config path hard-coded: https://botbot.me/freenode/logstash/2015-11-17/?msg=54338903&page=7 https://botbot.me/freenode/logstash/2015-11-17/?msg=54338903... There's also the problem that logstash (and forwarder) doesn't seem to let me do anything useful with the file names. I could work around that, sure, but it would be nice to have meaningful file names (not the "ls." thing that LS uses). Syslog-ng, for comparison, gives you a lot of control of that.
  - clebio 11y ago
    (I've never gotten used to HN's non-Markdown markup.)
  - Xylakant 11y ago
    > isn't the point of `/etc//conf.d` directories generally that you have multiple config files? Yes certainly. It's totally fine to place multiple config files there, I do as well. I split up my configs in the various outputs, inputs etc. It's just that logstash combines them to a single pipeline and does not run a pipeline per config file. Nginx doesn't run a webserver per config file either :). It's certainly something that's unexpected and could be much better documented, but alas, I'm just a user :) (and I do agree, your issue could have been handled much better, especially since it's not actually a bug)
    - clebio 11y ago
      Awesome, thanks for the clarification. This might help (if I ever go back to using logstash at this point!). Yeah, my point was more that they accepted my issue, but there's no action and there are more than 600+ other open issues. Seems Elastic is too busy branding and pushing breaking changes to their APIs. I do sincerely appreciate your clarifications and comments on this one, though.

by clebio 11y ago

dozzie 11y ago

Don't get too excited about Riemann. It's a stream processing engine, not data transport. It has its uses, but not for shipping logs. syslog-ng and its competitor, rsyslog, are fine, but they are concentrated around logs specifically. Fluentd (and logstash) can be used to transport other data, too, like monitoring or inventory.

clebio 11y ago
Good info (from jedisct1 as well) -- the distinctions are indeed important!

jedisct1 11y ago

Fluentd is great, and works very well to inject logs into Elasticsearch. We were using it at OpenDNS. We didn't even try Logstash, as Fluentd did the job. At OVH, both turned out to be way too slow, so we wrote Flowgger which is heavily used by all our services: https://github.com/jedisct1/flowgger https://github.com/jedisct1/flowgger

Xylakant 11y ago

I'm not an elastic person, but I can shed some light on this: You're holding it wrong. It's not a bug. You have multiple config files in one directory - if you do that, all those files are combined to one, that means that each event gets handed to each of your individual outputs - multiplying the message. See https://www.elastic.co/guide/en/logstash/current/command-line-flags.html https://www.elastic.co/guide/en/logstash/current/command-lin... Feel free to hop on the IRC if you have further questions, there's usually somebody qualified to answer.

clebio 11y ago
I appreciate the help, though isn't the point of `/etc//conf.d` directories generally that you have multiple config files? This is a common idiom that other packages handle correctly (differently?). I have hopped on the logstash IRC at times to ask about some of this, though I guess not this exact item. In fact, there's a different (well-known issue) that the init script for logstash has the config path hard-coded: https://botbot.me/freenode/logstash/2015-11-17/?msg=54338903&page=7 https://botbot.me/freenode/logstash/2015-11-17/?msg=54338903... There's also the problem that logstash (and forwarder) doesn't seem to let me do anything useful with the file names. I could work around that, sure, but it would be nice to have meaningful file names (not the "ls." thing that LS uses). Syslog-ng, for comparison, gives you a lot of control of that.
- clebio 11y ago
  (I've never gotten used to HN's non-Markdown markup.)
- Xylakant 11y ago
  > isn't the point of `/etc//conf.d` directories generally that you have multiple config files? Yes certainly. It's totally fine to place multiple config files there, I do as well. I split up my configs in the various outputs, inputs etc. It's just that logstash combines them to a single pipeline and does not run a pipeline per config file. Nginx doesn't run a webserver per config file either :). It's certainly something that's unexpected and could be much better documented, but alas, I'm just a user :) (and I do agree, your issue could have been handled much better, especially since it's not actually a bug)
  - clebio 11y ago
    Awesome, thanks for the clarification. This might help (if I ever go back to using logstash at this point!). Yeah, my point was more that they accepted my issue, but there's no action and there are more than 600+ other open issues. Seems Elastic is too busy branding and pushing breaking changes to their APIs. I do sincerely appreciate your clarifications and comments on this one, though.