4 ms·

Fluentd is decent (though I rather wished Heka had taken off; it's much more flexible and in theory leaner and faster since it's Go), but I'm still looking for

Fluentd is decent (though I rather wished Heka had taken off; it's much more flexible and in theory leaner and faster since it's Go), but I'm still looking for a good solution to aggregate, inspect and search logs. In particular, the ability to live-stream ("tail") logs seems to be a feature generally missing from logging aggregators. There was one (Loggly?) which provided a CLI tool for tailing, but it wasn't very impressive. (All the SaaS apps I have looked at do quite poorly when it comes to rendering live logs in a browser, too.) Any recommendations?

yacn 10y ago
We use Sumo Logic which has a Live-Tail feature [0], albeit I haven't used it myself yet. [0] https://www.sumologic.com/press/2016-01-21/sumo-logic-announces-availability-of-sumo-logic-live-tail/ https://www.sumologic.com/press/2016-01-21/sumo-logic-announ...
nlh 10y ago
Scalyr supports live tail (in both classic and new UI modes): http://www.scalyr.com http://www.scalyr.com (*disclaimer: I'm one of the co-founders)
- lobster_johnson 10y ago
  Are the screenshots from the old or new UI? Looks a bit dated on first glance (sorry). However, the pricing looks very silly. 10GB/day is toy volumes, and only 30 days' retention, and this for $400/mo? That's not going to fly (sorry again).
  - nlh 10y ago
    No need to apologize :) Yes - screens are from old UI. New UI is currently in public beta and, as you'll see, dramatically improved: https://www.scalyr.com/product/new-ui/opt-in https://www.scalyr.com/product/new-ui/opt-in Re: pricing. A much bigger topic than there's room for here on HN, but in short - for a service that can ingest 1TB/day+ of your logs and gives you search times measured in milliseconds, you'll actually find our pricing is not only in line, but below a lot of other commercial providers (Splunk, Sumo Logic, etc.) And when you compare TCO with open source solutions (ELK, etc.) that require a significant amount of effort to scale, it's similarly competitive.
    - lobster_johnson 10y ago
      I think all the current logging providers are hugely overpriced. I understand why people run ELK or Graylog themselves. I suspect you're going to feel some competition from Google StackDriver/BigQuery. You can load 300GB/mo (your biggest plan) into BigQuery for about $156/mo including storage. As someone who is basing our stack on Kubernetes, "effort to scale" is pretty damn small these days!
toomuchtodo 10y ago
https://www.graylog.org/ https://www.graylog.org/ It's self-hosted, not a service, but it is rock solid, and it can easily sustain tens of thousands of messages being sent to it per second. Docker can send Graylog logging messages natively. Use it. I promise you won't be disappointed.
- tokenizerrr 10y ago
  I started using this recently. Initial page loads are kinda slow, searches are fast and so is clicking links after the initial page load (seems to be a SPA). Did you encounter this at all? If so, know any fixes?
  - toomuchtodo 10y ago
    How much RAM/heap do you have assigned to the Graylog web frontend node? All on the same instance/physical server? Keep in mind, Graylog is using Elasticsearch under the hood extensively. If you're not allocating enough ram, you're gonna have a bad time. If you'd like, I can send an initial email to you (from your HN profile) and you can bounce questions off me if you'd like.
    - tokenizerrr 10y ago
      16GB for the OS, with only 10GB used most of the time currently. I'm also using pretty much the default graylog configuration, not sure what needs tweaking since it's really just the initial pageload being slow. Anything that's done after that without reloading the page is nice and speedy. And yes, that would be very nice of you!
kvz 10y ago
Paper trail isn't perfect but does an okayish job at tailing in the browser
pas 10y ago
Well, it's a tall order, but I think your best bet is to build something with elastic stuff: beats, logstash, elasticsearch, kibana. We store logs for long term in Ceph (we contributed a RADOS plugin so logstash can output directly into Ceph). So, new stuff goes to elastic, gets indexed, you can look at it via Kibana, or build custom dashboards, even directly from the logstash firehose. As logs get older, you can delete whole daily indexes from ES, and if you want to investigate/datamine/aggregate something, you can still grep the archived logs. The bottleneck will be probably Kibana (or the admin/operator looking at the end result), as all the other components can be scaled (beats are already per-node, logstash is stateless, so just run more of them behind a round-robin DNS name and beats will pick one up - or of course you can use HAproxy to load-balance, and the elasticsearch cluster can be rather large too).
ktamura 10y ago
>In particular, the ability to live-stream ("tail") logs seems to be a feature generally missing from logging aggregators. If you are talking about tailing log files live, Fluentd has supported it from Day 1: http://docs.fluentd.org/articles/in_tail http://docs.fluentd.org/articles/in_tail Also, as other sibling comments mention, there are tools, both SaaS and open source, that you can use as a destination of the logs Fluentd tails/listens/collects. * Elasticsearch: https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization https://www.digitalocean.com/community/tutorials/elasticsear... * Graylog: http://www.fluentd.org/guides/recipes/graylog2 http://www.fluentd.org/guides/recipes/graylog2 * Scalyr: https://github.com/scalyr/scalyr-fluentd https://github.com/scalyr/scalyr-fluentd * Loggly: https://www.loggly.com/blog/stream-filtering-loggly-fluentd/ https://www.loggly.com/blog/stream-filtering-loggly-fluentd/ * SumoLogic: https://gist.github.com/d-smith/8d3e7d53db772c6a7845 https://gist.github.com/d-smith/8d3e7d53db772c6a7845 * Papertrail: https://github.com/docebo/fluent-plugin-remote-syslog https://github.com/docebo/fluent-plugin-remote-syslog (and literally hundreds of others) >though I rather wished Heka had taken off; it's much more flexible and in theory leaner and faster since it's Go Heka was a great project, and a drop-in binary (as opposed to requiring a VM like Ruby) approach was interesting if not compelling in certain situations. That said, I never saw any benchmark that showed Heka was materially faster than Fluentd (or Logstash, for that matter). A lot of speed in this type of complex software comes from data structures/algorithms, an appropriate use of low-level language bindings, etc. While language plays a role in the speed of software, it's hardly the only factor. As you said, it's only in theory, not in practice =)
- lobster_johnson 10y ago
  I meant tailing the aggregated logs themselves. I have looked at the CLI tools that exist for some of those SaaS services, and I have not found anything half decent. For example, you will definitely want to filter on labels (including regexps) while tailing, and such filtrering should support adjustable context (both # of lines and time interval) and should support an optional time range to scroll back into history ("tail from 2pm"). And of course, grepping of historical entries. Another thing I am not impressed with is pricing. Loggly seems the most reasonable in terms of price per volume, except it limits the number of "team members" to a ridiculous degree (5 users or something like that). I set up Graylog once and wasn't impressed with it. Its reliance on Elasticsearch means it is quite static when it comes to the schema/input format. You can't change the settings; there is no reindexing support (or at least this was the case when I tried it, a year ago or so). Also, don't think it has any CLI tools?

yacn 10y ago

We use Sumo Logic which has a Live-Tail feature [0], albeit I haven't used it myself yet. [0] https://www.sumologic.com/press/2016-01-21/sumo-logic-announces-availability-of-sumo-logic-live-tail/ https://www.sumologic.com/press/2016-01-21/sumo-logic-announ...

nlh 10y ago

Scalyr supports live tail (in both classic and new UI modes): http://www.scalyr.com http://www.scalyr.com (*disclaimer: I'm one of the co-founders)

lobster_johnson 10y ago
Are the screenshots from the old or new UI? Looks a bit dated on first glance (sorry). However, the pricing looks very silly. 10GB/day is toy volumes, and only 30 days' retention, and this for $400/mo? That's not going to fly (sorry again).
- nlh 10y ago
  No need to apologize :) Yes - screens are from old UI. New UI is currently in public beta and, as you'll see, dramatically improved: https://www.scalyr.com/product/new-ui/opt-in https://www.scalyr.com/product/new-ui/opt-in Re: pricing. A much bigger topic than there's room for here on HN, but in short - for a service that can ingest 1TB/day+ of your logs and gives you search times measured in milliseconds, you'll actually find our pricing is not only in line, but below a lot of other commercial providers (Splunk, Sumo Logic, etc.) And when you compare TCO with open source solutions (ELK, etc.) that require a significant amount of effort to scale, it's similarly competitive.
  - lobster_johnson 10y ago
    I think all the current logging providers are hugely overpriced. I understand why people run ELK or Graylog themselves. I suspect you're going to feel some competition from Google StackDriver/BigQuery. You can load 300GB/mo (your biggest plan) into BigQuery for about $156/mo including storage. As someone who is basing our stack on Kubernetes, "effort to scale" is pretty damn small these days!

toomuchtodo 10y ago

https://www.graylog.org/ https://www.graylog.org/ It's self-hosted, not a service, but it is rock solid, and it can easily sustain tens of thousands of messages being sent to it per second. Docker can send Graylog logging messages natively. Use it. I promise you won't be disappointed.

tokenizerrr 10y ago
I started using this recently. Initial page loads are kinda slow, searches are fast and so is clicking links after the initial page load (seems to be a SPA). Did you encounter this at all? If so, know any fixes?
- toomuchtodo 10y ago
  How much RAM/heap do you have assigned to the Graylog web frontend node? All on the same instance/physical server? Keep in mind, Graylog is using Elasticsearch under the hood extensively. If you're not allocating enough ram, you're gonna have a bad time. If you'd like, I can send an initial email to you (from your HN profile) and you can bounce questions off me if you'd like.
  - tokenizerrr 10y ago
    16GB for the OS, with only 10GB used most of the time currently. I'm also using pretty much the default graylog configuration, not sure what needs tweaking since it's really just the initial pageload being slow. Anything that's done after that without reloading the page is nice and speedy. And yes, that would be very nice of you!

kvz 10y ago

Paper trail isn't perfect but does an okayish job at tailing in the browser

pas 10y ago

Well, it's a tall order, but I think your best bet is to build something with elastic stuff: beats, logstash, elasticsearch, kibana. We store logs for long term in Ceph (we contributed a RADOS plugin so logstash can output directly into Ceph). So, new stuff goes to elastic, gets indexed, you can look at it via Kibana, or build custom dashboards, even directly from the logstash firehose. As logs get older, you can delete whole daily indexes from ES, and if you want to investigate/datamine/aggregate something, you can still grep the archived logs. The bottleneck will be probably Kibana (or the admin/operator looking at the end result), as all the other components can be scaled (beats are already per-node, logstash is stateless, so just run more of them behind a round-robin DNS name and beats will pick one up - or of course you can use HAproxy to load-balance, and the elasticsearch cluster can be rather large too).

ktamura 10y ago

>In particular, the ability to live-stream ("tail") logs seems to be a feature generally missing from logging aggregators. If you are talking about tailing log files live, Fluentd has supported it from Day 1: http://docs.fluentd.org/articles/in_tail http://docs.fluentd.org/articles/in_tail Also, as other sibling comments mention, there are tools, both SaaS and open source, that you can use as a destination of the logs Fluentd tails/listens/collects. * Elasticsearch: https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization https://www.digitalocean.com/community/tutorials/elasticsear... * Graylog: http://www.fluentd.org/guides/recipes/graylog2 http://www.fluentd.org/guides/recipes/graylog2 * Scalyr: https://github.com/scalyr/scalyr-fluentd https://github.com/scalyr/scalyr-fluentd * Loggly: https://www.loggly.com/blog/stream-filtering-loggly-fluentd/ https://www.loggly.com/blog/stream-filtering-loggly-fluentd/ * SumoLogic: https://gist.github.com/d-smith/8d3e7d53db772c6a7845 https://gist.github.com/d-smith/8d3e7d53db772c6a7845 * Papertrail: https://github.com/docebo/fluent-plugin-remote-syslog https://github.com/docebo/fluent-plugin-remote-syslog (and literally hundreds of others) >though I rather wished Heka had taken off; it's much more flexible and in theory leaner and faster since it's Go Heka was a great project, and a drop-in binary (as opposed to requiring a VM like Ruby) approach was interesting if not compelling in certain situations. That said, I never saw any benchmark that showed Heka was materially faster than Fluentd (or Logstash, for that matter). A lot of speed in this type of complex software comes from data structures/algorithms, an appropriate use of low-level language bindings, etc. While language plays a role in the speed of software, it's hardly the only factor. As you said, it's only in theory, not in practice =)

lobster_johnson 10y ago
I meant tailing the aggregated logs themselves. I have looked at the CLI tools that exist for some of those SaaS services, and I have not found anything half decent. For example, you will definitely want to filter on labels (including regexps) while tailing, and such filtrering should support adjustable context (both # of lines and time interval) and should support an optional time range to scroll back into history ("tail from 2pm"). And of course, grepping of historical entries. Another thing I am not impressed with is pricing. Loggly seems the most reasonable in terms of price per volume, except it limits the number of "team members" to a ridiculous degree (5 users or something like that). I set up Graylog once and wasn't impressed with it. Its reliance on Elasticsearch means it is quite static when it comes to the schema/input format. You can't change the settings; there is no reindexing support (or at least this was the case when I tried it, a year ago or so). Also, don't think it has any CLI tools?