4 ms·

I still remember when Grafana first come out because of InfluxDB. It's amazing at that time. I try to setup Graphite and eventually found out Grafana+InfluxDB a

I still remember when Grafana first come out because of InfluxDB. It's amazing at that time. I try to setup Graphite and eventually found out Grafana+InfluxDB and go w/ it from there. Grafana move really fast. Very practical. It first if I remember use ElasticSearch/InfluxDB to store its config itself. Then move to its own SQLite. Then MySQL/Postgres. Eventually add alerting. Add multiple data source. Nowsday my team even use its with SQL for a cheap version of Periscope/Mode. Then now they figure out this log thing. ElasticSearch is great but try to run it yourself. On a 50 nodes k8s, I bet your first ES config will down after 10mins the first time FluentD come up and send all the log under the sun since the cluster come up Given that Grafan guys know how to deal with visualization, I trust them to deliver a great experience again for log. Look at their history, I'm going to bet hard on this.

outworlder 8y ago
> Then now they figure out this log thing. ElasticSearch is great but try to run it yourself. On a 50 nodes k8s, I bet your first ES config will down after 10mins the first time FluentD come up and send all the log under the sun since the cluster come up Tell me about it. I have not one, but multiple k8s clusters (some over 50 nodes), sending data to a single Elasticsearch cluster. A lot of tuning was done, and it's not yet perfect. Elasticsearch is amazing, but it requires external support, and the tools available to do that are not on par. Curator specifically – it's dumb as a rock and is very limited on what it can do. Even a hot/warm architecture is a challenge to do on curator alone, you should ideally have custom scripts to manage it. Which is a shame as this is one of the "reference" architectures by Elastic. Whatever Curator does should really be part of ES itself. The Kibana + ELK combo won't give you things like log tailing (logtrail is hackish and hard on the servers). The log forwarders are horrible to work with (be it logstash, filebeat – or worse: fluentd). Some days it just works and you are happy. Some other days, you wonder why we moved away from syslog senders and text files...
- kureikain 8y ago
  I actually have same conclusion as you. ES isn't easy to tune and run well. A thing many overlook are restart it. You cannot just go and `systemctl restart` it, once that occurs, constant rebalance happenning because it puts the load to the rest of system and eventually bring them down, all around :(. That make it harder to operator in K8S when it needs time to detach/attach volume, plus the overhead of overlay network. So I myself didn't know about a perfect tunning. But the log search on Kibana is super helpful though. That's why I kind of sold on this Loki thing. I have a good feeling that software in Go tend to require less config/tunning compare with Java. Eg, InfluxDB. > Some other days, you wonder why we moved away from syslog senders and text files... I wonder the same. I actually like to grep log with tail more than a crappy web ui like Kibana, Sumologic etc. Tailing log in a webui is clunky.
  - styfle 8y ago
    > once [a reboot] occurs, constant rebalance happening because it puts the load to the rest of system and eventually bring them down, all around I have this same problem with a 3 node cluster, no Kubernetes. This was using Elasticsearch 2.3 and it worked for a year or two and then all of a sudden, any network interruptions would cause the thing to get split brain and all indices would go red. It would take 8+ hours to go back to green and usually required a reboot of all servers or else it would never recover. It was happening often enough that the decision was just run a single ES node since the total data was only 25GB :(
- bogomipz 8y ago
  >"The log forwarders are horrible to work with (be it logstash, filebeat – or worse: fluentd)." Why are they horrible to work with? Why is fluentd worse than the other two?
- polyfractal 8y ago
  > Whatever Curator does should really be part of ES itself. Just as an FYI, Index Lifecycle Management (ILM) is landing in 6.6 and provides some of the features that Curator supports: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/index-lifecycle-management.html https://www.elastic.co/guide/en/elasticsearch/reference/6.x/... Basically, it allows you to define things like hot/warm/cold architecture, rollover, retention, etc at the index-level, inside Elasticsearch. Will be landing as beta
- aravindputrevu 8y ago
  > Whatever Curator does should really be part of ES itself. It is becoming part of ES. Index Life Cycle management UI is coming up in Kibana. > The Kibana + ELK combo won't give you things like log tailing (logtrail is hackish and hard on the servers) Recently, in 6.5.0 log tailing was added. Now one can look at streaming logs live. > The log forwarders are horrible to work with (be it logstash, filebeat – or worse: fluentd) I'd like to know the problem you faced.
cassianoleal 8y ago
> Then now they figure out this log thing. ElasticSearch is great but try to run it yourself. On a 50 nodes k8s, I bet your first ES config will down after 10mins the first time FluentD come up and send all the log under the sun since the cluster come up Put a buffer in front of it, pace the initial ingestion and Bob's your uncle. If that was the main problem operating an Elasticsearch cluster it would be all roses. I like the E?K stack a lot and I admit that operating ES is not exactly easy, but the initial ingestion issue the easy to solve.
- kureikain 8y ago
  That's what I endup doing though. No longer directly write to ES but go into Kafka. But taking a step back, thinking about how many components you have in the system just to make it work. Compare with Telegraf/InfluxDB/Grafana stack, which is for metrics, operate it is seamless. That's my high hope for Loki. I hope eventually Loki can just as easy to operate as InfluxDB.

outworlder 8y ago

> Then now they figure out this log thing. ElasticSearch is great but try to run it yourself. On a 50 nodes k8s, I bet your first ES config will down after 10mins the first time FluentD come up and send all the log under the sun since the cluster come up Tell me about it. I have not one, but multiple k8s clusters (some over 50 nodes), sending data to a single Elasticsearch cluster. A lot of tuning was done, and it's not yet perfect. Elasticsearch is amazing, but it requires external support, and the tools available to do that are not on par. Curator specifically – it's dumb as a rock and is very limited on what it can do. Even a hot/warm architecture is a challenge to do on curator alone, you should ideally have custom scripts to manage it. Which is a shame as this is one of the "reference" architectures by Elastic. Whatever Curator does should really be part of ES itself. The Kibana + ELK combo won't give you things like log tailing (logtrail is hackish and hard on the servers). The log forwarders are horrible to work with (be it logstash, filebeat – or worse: fluentd). Some days it just works and you are happy. Some other days, you wonder why we moved away from syslog senders and text files...

kureikain 8y ago
I actually have same conclusion as you. ES isn't easy to tune and run well. A thing many overlook are restart it. You cannot just go and `systemctl restart` it, once that occurs, constant rebalance happenning because it puts the load to the rest of system and eventually bring them down, all around :(. That make it harder to operator in K8S when it needs time to detach/attach volume, plus the overhead of overlay network. So I myself didn't know about a perfect tunning. But the log search on Kibana is super helpful though. That's why I kind of sold on this Loki thing. I have a good feeling that software in Go tend to require less config/tunning compare with Java. Eg, InfluxDB. > Some other days, you wonder why we moved away from syslog senders and text files... I wonder the same. I actually like to grep log with tail more than a crappy web ui like Kibana, Sumologic etc. Tailing log in a webui is clunky.
- styfle 8y ago
  > once [a reboot] occurs, constant rebalance happening because it puts the load to the rest of system and eventually bring them down, all around I have this same problem with a 3 node cluster, no Kubernetes. This was using Elasticsearch 2.3 and it worked for a year or two and then all of a sudden, any network interruptions would cause the thing to get split brain and all indices would go red. It would take 8+ hours to go back to green and usually required a reboot of all servers or else it would never recover. It was happening often enough that the decision was just run a single ES node since the total data was only 25GB :(
bogomipz 8y ago
>"The log forwarders are horrible to work with (be it logstash, filebeat – or worse: fluentd)." Why are they horrible to work with? Why is fluentd worse than the other two?
polyfractal 8y ago
> Whatever Curator does should really be part of ES itself. Just as an FYI, Index Lifecycle Management (ILM) is landing in 6.6 and provides some of the features that Curator supports: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/index-lifecycle-management.html https://www.elastic.co/guide/en/elasticsearch/reference/6.x/... Basically, it allows you to define things like hot/warm/cold architecture, rollover, retention, etc at the index-level, inside Elasticsearch. Will be landing as beta
aravindputrevu 8y ago
> Whatever Curator does should really be part of ES itself. It is becoming part of ES. Index Life Cycle management UI is coming up in Kibana. > The Kibana + ELK combo won't give you things like log tailing (logtrail is hackish and hard on the servers) Recently, in 6.5.0 log tailing was added. Now one can look at streaming logs live. > The log forwarders are horrible to work with (be it logstash, filebeat – or worse: fluentd) I'd like to know the problem you faced.

cassianoleal 8y ago

> Then now they figure out this log thing. ElasticSearch is great but try to run it yourself. On a 50 nodes k8s, I bet your first ES config will down after 10mins the first time FluentD come up and send all the log under the sun since the cluster come up Put a buffer in front of it, pace the initial ingestion and Bob's your uncle. If that was the main problem operating an Elasticsearch cluster it would be all roses. I like the E?K stack a lot and I admit that operating ES is not exactly easy, but the initial ingestion issue the easy to solve.

kureikain 8y ago
That's what I endup doing though. No longer directly write to ES but go into Kafka. But taking a step back, thinking about how many components you have in the system just to make it work. Compare with Telegraf/InfluxDB/Grafana stack, which is for metrics, operate it is seamless. That's my high hope for Loki. I hope eventually Loki can just as easy to operate as InfluxDB.