3 ms·

I'm wondering if Telegraf is a legit replacement for logstash or fluentd for shipping logs. I couldn't find this info from docs, so do you know if: - it suppor

I'm wondering if Telegraf is a legit replacement for logstash or fluentd for shipping logs. I couldn't find this info from docs, so do you know if: - it supports multi-line logs (e.g. java stack traces) - it can output to elasticsearch (didn't see an output plugin) - there's any solution for reading docker logs (looks like docker metrics are supported) - any other critical logstash functionality missing? If it doesn't support the classic Elasticsearch output, where are telegraf log users typically outputting logs to?

pauldix 10y ago
Telegraf users are generally parsing the logs into structured metrics and events that go into InfluxDB. Not sure that it supports multi-line logs or docker logs. I'll have to look into it, but both should probably be done if they're not already there. For ES output, we're happy to take PRs for other output plugins. There are people using Telegraf that aren't using InfluxDB. That's fine and we're happy to have an open source collector that others use and contribute to.
sparrc 10y ago
> - it supports multi-line logs (e.g. java stack traces) unfortunately it doesn't, there hasn't been a request for it yet but please feel free to open an issue on the repo with any details you can bring: https://github.com/influxdata/telegraf/issues https://github.com/influxdata/telegraf/issues > - it can output to elasticsearch (didn't see an output plugin) Nope, not yet: https://github.com/influxdata/telegraf/issues/782 https://github.com/influxdata/telegraf/issues/782 > - there's any solution for reading docker logs (looks like docker metrics are supported) If there is a logstash "grok" pattern for parsing docker logs, then telegraf supports it. Though it's probably worth formulating a grok pattern specifically for telegraf that properly takes advantage of tags, fields, measurements, etc. > - any other critical logstash functionality missing? Need more user feedback to tackle this one, but feel free to open an issue with anything that you find is missing :) https://github.com/influxdata/telegraf/issues https://github.com/influxdata/telegraf/issues
- joshpadnick 10y ago
  Thanks for this great response. To give more color on my use case, my goal is to ship logs off the server and to some central log aggregation platform. That could be self-managed (e.g. Elasticsearch) or hosted (e.g. SumoLogic, Loggly). In many cases, I'd want to get those logs from Docker containers and include metadata on each container so I know, e.g. what app is running, the container Id, on what host, etc. Traditionally, tools like logstash, fluentd, and heka meet these needs. It doesn't sounds like Telegraf is quite ready to support this use case in full, but could certainly head in that direction.

pauldix 10y ago

Telegraf users are generally parsing the logs into structured metrics and events that go into InfluxDB. Not sure that it supports multi-line logs or docker logs. I'll have to look into it, but both should probably be done if they're not already there. For ES output, we're happy to take PRs for other output plugins. There are people using Telegraf that aren't using InfluxDB. That's fine and we're happy to have an open source collector that others use and contribute to.

sparrc 10y ago

> - it supports multi-line logs (e.g. java stack traces) unfortunately it doesn't, there hasn't been a request for it yet but please feel free to open an issue on the repo with any details you can bring: https://github.com/influxdata/telegraf/issues https://github.com/influxdata/telegraf/issues > - it can output to elasticsearch (didn't see an output plugin) Nope, not yet: https://github.com/influxdata/telegraf/issues/782 https://github.com/influxdata/telegraf/issues/782 > - there's any solution for reading docker logs (looks like docker metrics are supported) If there is a logstash "grok" pattern for parsing docker logs, then telegraf supports it. Though it's probably worth formulating a grok pattern specifically for telegraf that properly takes advantage of tags, fields, measurements, etc. > - any other critical logstash functionality missing? Need more user feedback to tackle this one, but feel free to open an issue with anything that you find is missing :) https://github.com/influxdata/telegraf/issues https://github.com/influxdata/telegraf/issues

joshpadnick 10y ago
Thanks for this great response. To give more color on my use case, my goal is to ship logs off the server and to some central log aggregation platform. That could be self-managed (e.g. Elasticsearch) or hosted (e.g. SumoLogic, Loggly). In many cases, I'd want to get those logs from Docker containers and include metadata on each container so I know, e.g. what app is running, the container Id, on what host, etc. Traditionally, tools like logstash, fluentd, and heka meet these needs. It doesn't sounds like Telegraf is quite ready to support this use case in full, but could certainly head in that direction.