4 ms·

I'd be interested in replacing an in-house logs filtering/aggregation solution with Loki, but there's a lot that is not clear to me from the initial announcemen

by akavel 8y ago

I'd be interested in replacing an in-house logs filtering/aggregation solution with Loki, but there's a lot that is not clear to me from the initial announcement & materials. Could you please help me understand what are the answers to the following questions? - Can I use Loki without Prometheus? I'd like to feed raw logs to it, with custom-generated metadata. I don't want to have to use Prometheus, nor InfluxDB. - Can I edit (modify) metadata for some old log line after it was already inserted? Specifically, I need to be able to rebuild metadata later, if I add some new "filters" to my logs (I want to be able to apply them retroactively). - Can I run aggregate queries on the metadata (sum, avg, min/max)? If not, what can I do with the metadata? Can I graph the metadata on normal Grafana graphs? - Is the text of the logs compressed? If yes, what compression algorithm is used? If not, why? - Where can I find the API docs (or at least the API source code) for Loki?

gouthamve 8y ago
1. Yes, you can. We initially targeted Kubernetes because that's what we use and packaging for it is simple. We'll soon release packages for all distros, support syslog/journald etc. 2. Hmm, please open an issue regarding this with the use-case that prompted this? This is not a use-case we have but if this is important for more people, we'd be happy to support it! 3. You can only select based on the metadata, the metadata is just a set of kv pairs. Like {app=cassandra, namespace=prod, instance=cassandra-minion-0} 4. Yes, we used gzip. Please see the sheet referenced at the end of the design doc [0] to see what we compared with. 5. It's mainly protobufs right now [1] over HTTP, but we'll be adding more docs soon. Mind opening an issue for this so that we don't forget? [0] https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3-q6vBAsZYIQ5ZeYBkyM/edit https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3... [1] https://github.com/grafana/loki/tree/master/pkg/logproto https://github.com/grafana/loki/tree/master/pkg/logproto
- akx 8y ago
  Hm, why is zstd marked as "BROKEN"? zstd would've been my first choice for something like this.

by akavel 8y ago

gouthamve 8y ago

1. Yes, you can. We initially targeted Kubernetes because that's what we use and packaging for it is simple. We'll soon release packages for all distros, support syslog/journald etc. 2. Hmm, please open an issue regarding this with the use-case that prompted this? This is not a use-case we have but if this is important for more people, we'd be happy to support it! 3. You can only select based on the metadata, the metadata is just a set of kv pairs. Like {app=cassandra, namespace=prod, instance=cassandra-minion-0} 4. Yes, we used gzip. Please see the sheet referenced at the end of the design doc [0] to see what we compared with. 5. It's mainly protobufs right now [1] over HTTP, but we'll be adding more docs soon. Mind opening an issue for this so that we don't forget? [0] https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3-q6vBAsZYIQ5ZeYBkyM/edit https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3... [1] https://github.com/grafana/loki/tree/master/pkg/logproto https://github.com/grafana/loki/tree/master/pkg/logproto

akx 8y ago
Hm, why is zstd marked as "BROKEN"? zstd would've been my first choice for something like this.