3 ms·

I am currently stuggeling setting up centralized logging; just can´t grok it. Logstash? Beats? Elastic Agent? Is ELK-Stack right in 2022? How do I get Logs out

I am currently stuggeling setting up centralized logging; just can´t grok it. Logstash? Beats? Elastic Agent? Is ELK-Stack right in 2022? How do I get Logs out of Docker to Logstash? Do I? Seems I need to get my head around this...

patrakov 4y ago
Please take a look at Loki. It's more lightweight than the ELK stack.
zie 4y ago
Generally you will find lots of different solutions to make it all work right. None of them are "wrong" or "right", there are 2 pieces to logs: 1) Capturing them 2) Indexing/acting on them Most tools focus on one or the other, some focus on both. > Logstash? Beats? Elastic Agent? Is ELK-Stack right in 2022? Loki and Graylog solve both issues in an integrated way. logstash, beats and syslog tend to focus only on the 1st part, moving the lots to a central place. > How do I get Logs out of Docker to Logstash? Do I? Yes you want the logs from Docker jobs. You can use a sidecar, or you can have the Docker image ship the logs or if you are running k8s, Nomad, etc they might be able to do it for you. The more important thing is the 1st, getting them all to one host, in pretty much any format. Once you have them there, then you can worry about trying to make sense of them.
- theblazehen 4y ago
  I'm pretty happy with Loki + promtail so far
  - number6 4y ago
    I will set it up next Day at work, thanks for the advice
    - alexjplant 4y ago
      If you do I strongly recommend storing the indices with the chunks[0] and configuring a hassle-free, scalable storage solution [1] up front. Migrating chunk storage after the fact is a royal pain. Do also note that the official Helm chart will only get you so far before you have to start horizontally scaling individual components [2]; depending upon your query and log volume you might want to scale these separately and the chart [3] doesn't support this. This GitHub issue [4] has some Kubernetes YAML examples if you need them. [0] https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/ https://grafana.com/docs/loki/latest/operations/storage/bolt... [1] https://grafana.com/docs/loki/latest/operations/storage/#supported-stores https://grafana.com/docs/loki/latest/operations/storage/#sup... [2] https://grafana.com/docs/loki/latest/fundamentals/architecture/components/ https://grafana.com/docs/loki/latest/fundamentals/architectu... [3] https://github.com/grafana/helm-charts/blob/main/charts/loki/templates/statefulset.yaml https://github.com/grafana/helm-charts/blob/main/charts/loki... [4] https://github.com/grafana/loki/issues/643 https://github.com/grafana/loki/issues/643
theptip 4y ago
If you just want to pay someone and move on, Datadog has great connectors, but is expensive. I hear Honeycomb is great, and they have a free tier. (I started with Grafana and Prometheus FWIW)

patrakov 4y ago

Please take a look at Loki. It's more lightweight than the ELK stack.

zie 4y ago

Generally you will find lots of different solutions to make it all work right. None of them are "wrong" or "right", there are 2 pieces to logs: 1) Capturing them 2) Indexing/acting on them Most tools focus on one or the other, some focus on both. > Logstash? Beats? Elastic Agent? Is ELK-Stack right in 2022? Loki and Graylog solve both issues in an integrated way. logstash, beats and syslog tend to focus only on the 1st part, moving the lots to a central place. > How do I get Logs out of Docker to Logstash? Do I? Yes you want the logs from Docker jobs. You can use a sidecar, or you can have the Docker image ship the logs or if you are running k8s, Nomad, etc they might be able to do it for you. The more important thing is the 1st, getting them all to one host, in pretty much any format. Once you have them there, then you can worry about trying to make sense of them.

theblazehen 4y ago
I'm pretty happy with Loki + promtail so far
- number6 4y ago
  I will set it up next Day at work, thanks for the advice
  - alexjplant 4y ago
    If you do I strongly recommend storing the indices with the chunks[0] and configuring a hassle-free, scalable storage solution [1] up front. Migrating chunk storage after the fact is a royal pain. Do also note that the official Helm chart will only get you so far before you have to start horizontally scaling individual components [2]; depending upon your query and log volume you might want to scale these separately and the chart [3] doesn't support this. This GitHub issue [4] has some Kubernetes YAML examples if you need them. [0] https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/ https://grafana.com/docs/loki/latest/operations/storage/bolt... [1] https://grafana.com/docs/loki/latest/operations/storage/#supported-stores https://grafana.com/docs/loki/latest/operations/storage/#sup... [2] https://grafana.com/docs/loki/latest/fundamentals/architecture/components/ https://grafana.com/docs/loki/latest/fundamentals/architectu... [3] https://github.com/grafana/helm-charts/blob/main/charts/loki/templates/statefulset.yaml https://github.com/grafana/helm-charts/blob/main/charts/loki... [4] https://github.com/grafana/loki/issues/643 https://github.com/grafana/loki/issues/643

theptip 4y ago

If you just want to pay someone and move on, Datadog has great connectors, but is expensive. I hear Honeycomb is great, and they have a free tier. (I started with Grafana and Prometheus FWIW)