4 ms·
>In particular, the ability to live-stream ("tail") logs seems to be a feature generally missing from logging aggregators. If you are talking about tailing log
by ktamura 10y ago
>In particular, the ability to live-stream ("tail") logs seems to be a feature generally missing from logging aggregators.
If you are talking about tailing log files live, Fluentd has supported it from Day 1: http://docs.fluentd.org/articles/in_tail http://docs.fluentd.org/articles/in_tail
Also, as other sibling comments mention, there are tools, both SaaS and open source, that you can use as a destination of the logs Fluentd tails/listens/collects.
* Elasticsearch: https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization https://www.digitalocean.com/community/tutorials/elasticsear...
* Graylog: http://www.fluentd.org/guides/recipes/graylog2 http://www.fluentd.org/guides/recipes/graylog2
* Scalyr: https://github.com/scalyr/scalyr-fluentd https://github.com/scalyr/scalyr-fluentd
* Loggly: https://www.loggly.com/blog/stream-filtering-loggly-fluentd/ https://www.loggly.com/blog/stream-filtering-loggly-fluentd/
* SumoLogic: https://gist.github.com/d-smith/8d3e7d53db772c6a7845 https://gist.github.com/d-smith/8d3e7d53db772c6a7845
* Papertrail: https://github.com/docebo/fluent-plugin-remote-syslog https://github.com/docebo/fluent-plugin-remote-syslog
(and literally hundreds of others)
>though I rather wished Heka had taken off; it's much more flexible and in theory leaner and faster since it's Go
Heka was a great project, and a drop-in binary (as opposed to requiring a VM like Ruby) approach was interesting if not compelling in certain situations. That said, I never saw any benchmark that showed Heka was materially faster than Fluentd (or Logstash, for that matter). A lot of speed in this type of complex software comes from data structures/algorithms, an appropriate use of low-level language bindings, etc.
While language plays a role in the speed of software, it's hardly the only factor. As you said, it's only in theory, not in practice =)
- lobster_johnson 10y agoI meant tailing the aggregated logs themselves. I have looked at the CLI tools that exist for some of those SaaS services, and I have not found anything half decent. For example, you will definitely want to filter on labels (including regexps) while tailing, and such filtrering should support adjustable context (both # of lines and time interval) and should support an optional time range to scroll back into history ("tail from 2pm"). And of course, grepping of historical entries. Another thing I am not impressed with is pricing. Loggly seems the most reasonable in terms of price per volume, except it limits the number of "team members" to a ridiculous degree (5 users or something like that). I set up Graylog once and wasn't impressed with it. Its reliance on Elasticsearch means it is quite static when it comes to the schema/input format. You can't change the settings; there is no reindexing support (or at least this was the case when I tried it, a year ago or so). Also, don't think it has any CLI tools?