5 ms·

Interesting to see a new approach! You wrote that you don't like Loki's LogQL, but it looks quite similar (Victoria's LogQL first): log.level:error _stream:

Interesting to see a new approach! You wrote that you don't like Loki's LogQL, but it looks quite similar (Victoria's LogQL first): log.level:error _stream:{app!~"buggy_app|foobar"} {app!~"buggy_app|foobar"} | "log.level:error" The pipes are arguably a bit noisy in Loki queries (compared to spaces in Victoria's), but I find they do make the queries a bit more readable, and it's easier to understand under the hood how the queriers will handle the query, and coming from PromQL I found Loki's approach quite intuitive: https://grafana.com/docs/loki/latest/query/ https://grafana.com/docs/loki/latest/query/. Maybe I missed something fundamental though, interested to hear more about the differences, since I only read the couple links you shared!

BillFranklin 2y ago
As an addendum, I think the main flaws with Loki is log presentation in the Grafana UI, rather than necessarily querying syntax (extracting data from log lines with regex etc/pattern matching is gross, and if you're doing that you have a bigger problem). With the Grafana Loki UI the main issue is if you log perfect JSON / logfmt and Loki parses it, the Grafana UI can struggle a bit with rendering views of logs in the way you'd want. Kibana's columnar views with nice filtering UX is much easier to use. I think Elastic still has this over Loki (that, and the capacity to build big expensive indexes if you have the stomach to manage them). I wrote a custom CLI for Loki to work around this, because the Loki CLI has a similar problem. All that said, I'd still recommend Loki, it's really good IME.
deepsun 2y ago
I believe pipes for logs were invented by SumoLogic 10+ years ago. Or maybe someone before that.
- valyala 2y ago
  Canonical pipes were invented ~50 years ago by Unix creators. They are successfully used for logs' analysis to this day. Unix pipes have a simple idea - connecting the output of one program to the input of another program. This allows building arbitrary complex data processing pipelines by combining simple programs like grep, cat, awk, cut, head, etc., via pipes. These pipelines have nice properties: - The data is processed in a streaming manner. This allows processing unlimited data streams without excess resource usage. - The data processing speed is limited by the slowest program in the pipeline. This allows pausing and resuming data processing at any time. For example, by putting 'less' to the end of the pipeline. - The data processing pipeline is terminated instantly as soon as some program in the pipeline terminates. For example, if you put 'tail -100' to the end of pipeline, then only the first 100 lines of data will be processed. VictoriaLogs adapts these properties into logs processing with LogsQL pipes. The difference is that the data, which is flown between pipes, represents structured logs [1]. [1] https://docs.victoriametrics.com/victorialogs/querying/#command-line https://docs.victoriametrics.com/victorialogs/querying/#comm...
valyala 2y ago
- Loki doesn't allow queries without stream filters. This may be very inconvenient. For example, try selecting all the logs with the 'error' word in Loki. In LogsQL you just type 'error' and that's it! - As I know, Loki doesn't allow selecting all the logs on the given time range. For example, try selecting all the logs for the last 5 minutes in Loki query language. In LogsQL this is just '_time:5m'. - Loki has unreadable syntax for calculating analytics over the selected logs. For example, counting the number of logs with the 'error' word over the last 5 minutes in Loki looks like: count_over_time({required="stream_filter"} | "error" [5m]) Compare this to LogsQL: _time:5m error | stats count() as errors - Loki doesn't allow calculating multiple stats in a single query. For example, try calculating the number of logs with the 'error' word, plus the total number of logs, over the last 5 minutes. In LogsQL this is easy: _time:5m | stats count() if(error) as errors, count() as total - As I know, Loki doesn't provide functionality for sorting of the returned logs. - Loki can return only up to 5000 logs from a single query by default. VictoriaLogs allows returning billions of logs from a single query, without any performance and resource usage issues [1]. [1] https://docs.victoriametrics.com/victorialogs/querying/#command-line https://docs.victoriametrics.com/victorialogs/querying/#comm...
- BillFranklin 2y ago
  I think we are conflating Loki/VictoriaLogs (the UI, API, and log databases) and LogsQL vs LogQL (the syntax). > Loki doesn't allow calculating multiple stats in a single query Victoria's support for projecting multiple stats in one query is nice, with Loki's LogQL you need to write two separate queries. > Loki doesn't allow queries without stream filters True! I don't mind filtering for e.g. `{env: "production"}` but each to their own :-) --- RE Loki vs Victoria, which I don't really want to get into since I am not a vendor and have not tried Victoria... > Loki doesn't allow selecting all the logs on the given time range True! TBH I don't mind selecting a date from a date picker. One issue is that I guess it's going to be a bit slower to write precise date/time ranges if they're baked into the query language? > Loki doesn't provide functionality for sorting of the returned logs > Loki can return only up to 5000 logs from a single query by default I think these are Grafana UI restrictions -- as I wrote above there are indeed limitations with the Grafana UI, but I don't think they're necessarily syntax related.

BillFranklin 2y ago

As an addendum, I think the main flaws with Loki is log presentation in the Grafana UI, rather than necessarily querying syntax (extracting data from log lines with regex etc/pattern matching is gross, and if you're doing that you have a bigger problem). With the Grafana Loki UI the main issue is if you log perfect JSON / logfmt and Loki parses it, the Grafana UI can struggle a bit with rendering views of logs in the way you'd want. Kibana's columnar views with nice filtering UX is much easier to use. I think Elastic still has this over Loki (that, and the capacity to build big expensive indexes if you have the stomach to manage them). I wrote a custom CLI for Loki to work around this, because the Loki CLI has a similar problem. All that said, I'd still recommend Loki, it's really good IME.

deepsun 2y ago

I believe pipes for logs were invented by SumoLogic 10+ years ago. Or maybe someone before that.

valyala 2y ago
Canonical pipes were invented ~50 years ago by Unix creators. They are successfully used for logs' analysis to this day. Unix pipes have a simple idea - connecting the output of one program to the input of another program. This allows building arbitrary complex data processing pipelines by combining simple programs like grep, cat, awk, cut, head, etc., via pipes. These pipelines have nice properties: - The data is processed in a streaming manner. This allows processing unlimited data streams without excess resource usage. - The data processing speed is limited by the slowest program in the pipeline. This allows pausing and resuming data processing at any time. For example, by putting 'less' to the end of the pipeline. - The data processing pipeline is terminated instantly as soon as some program in the pipeline terminates. For example, if you put 'tail -100' to the end of pipeline, then only the first 100 lines of data will be processed. VictoriaLogs adapts these properties into logs processing with LogsQL pipes. The difference is that the data, which is flown between pipes, represents structured logs [1]. [1] https://docs.victoriametrics.com/victorialogs/querying/#command-line https://docs.victoriametrics.com/victorialogs/querying/#comm...

valyala 2y ago

- Loki doesn't allow queries without stream filters. This may be very inconvenient. For example, try selecting all the logs with the 'error' word in Loki. In LogsQL you just type 'error' and that's it! - As I know, Loki doesn't allow selecting all the logs on the given time range. For example, try selecting all the logs for the last 5 minutes in Loki query language. In LogsQL this is just '_time:5m'. - Loki has unreadable syntax for calculating analytics over the selected logs. For example, counting the number of logs with the 'error' word over the last 5 minutes in Loki looks like: count_over_time({required="stream_filter"} | "error" [5m]) Compare this to LogsQL: _time:5m error | stats count() as errors - Loki doesn't allow calculating multiple stats in a single query. For example, try calculating the number of logs with the 'error' word, plus the total number of logs, over the last 5 minutes. In LogsQL this is easy: _time:5m | stats count() if(error) as errors, count() as total - As I know, Loki doesn't provide functionality for sorting of the returned logs. - Loki can return only up to 5000 logs from a single query by default. VictoriaLogs allows returning billions of logs from a single query, without any performance and resource usage issues [1]. [1] https://docs.victoriametrics.com/victorialogs/querying/#command-line https://docs.victoriametrics.com/victorialogs/querying/#comm...

BillFranklin 2y ago
I think we are conflating Loki/VictoriaLogs (the UI, API, and log databases) and LogsQL vs LogQL (the syntax). > Loki doesn't allow calculating multiple stats in a single query Victoria's support for projecting multiple stats in one query is nice, with Loki's LogQL you need to write two separate queries. > Loki doesn't allow queries without stream filters True! I don't mind filtering for e.g. `{env: "production"}` but each to their own :-) --- RE Loki vs Victoria, which I don't really want to get into since I am not a vendor and have not tried Victoria... > Loki doesn't allow selecting all the logs on the given time range True! TBH I don't mind selecting a date from a date picker. One issue is that I guess it's going to be a bit slower to write precise date/time ranges if they're baked into the query language? > Loki doesn't provide functionality for sorting of the returned logs > Loki can return only up to 5000 logs from a single query by default I think these are Grafana UI restrictions -- as I wrote above there are indeed limitations with the Grafana UI, but I don't think they're necessarily syntax related.