4 ms·

Show HN: LogsQL – opinionated query language for logs

61 points by valyala 2y ago

I don't like the existing query languages for Elasticsearch and Grafana Loki, because they are too awkward to use for typical logs' investigation cases. So I designed new query language - LogsQL - and wrote reference implementation for it as a part of VictoriaLogs - an open source database for logs. LogsQL is based on the following principles: - Simplicity. It is easy to write typical queries over logs in it. For example, a single word `error` is a valid LogsQL query, which returns all the logs with the `error` word. Another example is `_time:5m error`, which returns all the logs with the 'error' word over the last 5 minutes. - Composable building blocks similar to Unix pipes, which allow powerful filtering, transforming and calculating stats over the selected logs. For example, `_time:5m error | stats count() as rows` returns the number of logs with the `error` word over the last 5 minutes. - Readability. Typical LogsQL queries must be easy to read and understand even for persons unfamiliar with it. Take a look at LogsQL docs [1] and try using VictoriaLogs [2] in production. If you like Unix way and KISS design principle, then you'll enjoy LogsQL :) [1] https://docs.victoriametrics.com/victorialogs/logsql/ https://docs.victoriametrics.com/victorialogs/logsql/ [2] https://docs.victoriametrics.com/victorialogs/ https://docs.victoriametrics.com/victorialogs/

bdcravens 2y ago
You have to feed your logs into the VictoriaLogs database in order to use LogsQL, right? "LogsQL is a simple yet powerful query language for VictoriaLogs."
- valyala 2y ago
  Correct for now. I'm sure LogsQL support will be added into other systems for logs' management over time.
BillFranklin 2y ago
Interesting to see a new approach! You wrote that you don't like Loki's LogQL, but it looks quite similar (Victoria's LogQL first): log.level:error _stream:{app!~"buggy_app|foobar"} {app!~"buggy_app|foobar"} | "log.level:error" The pipes are arguably a bit noisy in Loki queries (compared to spaces in Victoria's), but I find they do make the queries a bit more readable, and it's easier to understand under the hood how the queriers will handle the query, and coming from PromQL I found Loki's approach quite intuitive: https://grafana.com/docs/loki/latest/query/ https://grafana.com/docs/loki/latest/query/. Maybe I missed something fundamental though, interested to hear more about the differences, since I only read the couple links you shared!
- BillFranklin 2y ago
  As an addendum, I think the main flaws with Loki is log presentation in the Grafana UI, rather than necessarily querying syntax (extracting data from log lines with regex etc/pattern matching is gross, and if you're doing that you have a bigger problem). With the Grafana Loki UI the main issue is if you log perfect JSON / logfmt and Loki parses it, the Grafana UI can struggle a bit with rendering views of logs in the way you'd want. Kibana's columnar views with nice filtering UX is much easier to use. I think Elastic still has this over Loki (that, and the capacity to build big expensive indexes if you have the stomach to manage them). I wrote a custom CLI for Loki to work around this, because the Loki CLI has a similar problem. All that said, I'd still recommend Loki, it's really good IME.
- deepsun 2y ago
  I believe pipes for logs were invented by SumoLogic 10+ years ago. Or maybe someone before that.
  - valyala 2y ago
    Canonical pipes were invented ~50 years ago by Unix creators. They are successfully used for logs' analysis to this day. Unix pipes have a simple idea - connecting the output of one program to the input of another program. This allows building arbitrary complex data processing pipelines by combining simple programs like grep, cat, awk, cut, head, etc., via pipes. These pipelines have nice properties: - The data is processed in a streaming manner. This allows processing unlimited data streams without excess resource usage. - The data processing speed is limited by the slowest program in the pipeline. This allows pausing and resuming data processing at any time. For example, by putting 'less' to the end of the pipeline. - The data processing pipeline is terminated instantly as soon as some program in the pipeline terminates. For example, if you put 'tail -100' to the end of pipeline, then only the first 100 lines of data will be processed. VictoriaLogs adapts these properties into logs processing with LogsQL pipes. The difference is that the data, which is flown between pipes, represents structured logs [1]. [1] https://docs.victoriametrics.com/victorialogs/querying/#command-line https://docs.victoriametrics.com/victorialogs/querying/#comm...
- 2y ago
dangoodmanUT 2y ago
IMO SQL is great for logs, ClickHouse and DuckDB SQL in particular are awesome
- lmeyerov 2y ago
  I find SQL nice for a happy path - highly structured, non-nested, and simple log scenarios, such as some heavily curated app log subset like an app-level authentication audit log When we have arbitrary logs from all over, and especially system.. I appreciate the sloppy and dynamic nature of SPL and its derivatives We have ported our app tier to OTel metrics/logs/traces, and I've been curious what will be more 'right' in this new era. Likewise, with louie.ai, we have genAI do the text2query for us by default, which shifts things a bit as well.
ople 2y ago
Recently there seems to be an bunch of SPL (Splunk) -like query languages popping up: PQL, PRQL, Grafana Explore Logs syntax, Kusto.. Probably others as well. Does yet another similar but slightly different language make sense? Why not leverage an existing one?
- valyala 2y ago
  Diversity is great! Let's see which query language for logs will survive.
  - ople 2y ago
    True. My bet is that arbitrary text queries converted to SQL via LLM will become an increasingly popular alternative. Not sure if it will completely replace the custom query languages though..
rixrax 2y ago
Am I the only one that feels that EVERYTHING is wrong in this ELK, Splunk, etc. Grafana world? The user interfaces that these monstrosities present us with are barely useable, everyone has their own query language, they force us to install their own agents own our hosts and servers, when I upload logs, many can't even take random JSON logs and input them in a structured way without defining pipeline rules or what now. And did I say that the Logstashes and Promtails and Vectors and what not pipeline tools with their Grok etc. filters feel like somebody wanted to really make busywork cool. I am happy that in my day to day work I can dump my mostly Linux logs to rsyslog, and eventually forward them to S3 glacier for a few years. So I am guessing the question I am asking is that what in the world are you doing with these observability or SIEM platforms and is anyone actually deriving some REAL value from using them?
- fragmede 2y ago
  Yes there's real value there. That everyone's got their own flavor is a bunch of extra work because we haven't solved the coordination problem yet is annoying, but that's solved by choosing one and sticking with it. Learn that query language really really well, and don't touch anything else. Splunk is useful as all hell once you get over the hump of learning their proprietary query language. it's really friggin useful. it's useful to the tune of Cisco buying them for $28 billion. people are deriving real value from them, the question is what are your problems that this can solve for you, but do you even have those problems in the first place? If you've not found it useful then why are you stuffing logs into S3? just send them to /dev/null instead
  - rixrax 2y ago
    >> just send them to /dev/null instead I wish. But 'regulatory compliance'. And 'we might need them' - just not sure for what - but we'll try another data analyst next quarter. Thankfully because of the GDPR (and maybe other reasons) there is a healthy pressure to also cleanse us from the logs we've collected. That said, I agree, based on my trials (and mostly errors), Splunk seems one of the better ones. Not considering the cost anyway. My trouble is that I am not a data analyst, but I get asked more than I would like about these things.
piterrro 2y ago
I'm a https://logdy.dev https://logdy.dev (logs to UI interface) author and been recently thinking about how to enable users use a query language to search throught logs beyond usual filter. I was looking at LogsQL but then I felt that is just another QL a user will need to learn. My next though was on SQL, but it was not designed for this purpose. Any ideas? I would appreciate any recommendation (peter at logd.dev) [1] https://github.com/logdyhq/logdy-core https://github.com/logdyhq/logdy-core
- valyala 2y ago
  How about allowing users creating pipelines with good old grep, awk, cut, etc., directly in the web UI?
deleted 2y ago
[deleted]

61 points by valyala 2y ago

bdcravens 2y ago

You have to feed your logs into the VictoriaLogs database in order to use LogsQL, right? "LogsQL is a simple yet powerful query language for VictoriaLogs."

valyala 2y ago
Correct for now. I'm sure LogsQL support will be added into other systems for logs' management over time.

BillFranklin 2y ago

Interesting to see a new approach! You wrote that you don't like Loki's LogQL, but it looks quite similar (Victoria's LogQL first): log.level:error _stream:{app!~"buggy_app|foobar"} {app!~"buggy_app|foobar"} | "log.level:error" The pipes are arguably a bit noisy in Loki queries (compared to spaces in Victoria's), but I find they do make the queries a bit more readable, and it's easier to understand under the hood how the queriers will handle the query, and coming from PromQL I found Loki's approach quite intuitive: https://grafana.com/docs/loki/latest/query/ https://grafana.com/docs/loki/latest/query/. Maybe I missed something fundamental though, interested to hear more about the differences, since I only read the couple links you shared!

BillFranklin 2y ago
As an addendum, I think the main flaws with Loki is log presentation in the Grafana UI, rather than necessarily querying syntax (extracting data from log lines with regex etc/pattern matching is gross, and if you're doing that you have a bigger problem). With the Grafana Loki UI the main issue is if you log perfect JSON / logfmt and Loki parses it, the Grafana UI can struggle a bit with rendering views of logs in the way you'd want. Kibana's columnar views with nice filtering UX is much easier to use. I think Elastic still has this over Loki (that, and the capacity to build big expensive indexes if you have the stomach to manage them). I wrote a custom CLI for Loki to work around this, because the Loki CLI has a similar problem. All that said, I'd still recommend Loki, it's really good IME.
deepsun 2y ago
I believe pipes for logs were invented by SumoLogic 10+ years ago. Or maybe someone before that.
- valyala 2y ago
  Canonical pipes were invented ~50 years ago by Unix creators. They are successfully used for logs' analysis to this day. Unix pipes have a simple idea - connecting the output of one program to the input of another program. This allows building arbitrary complex data processing pipelines by combining simple programs like grep, cat, awk, cut, head, etc., via pipes. These pipelines have nice properties: - The data is processed in a streaming manner. This allows processing unlimited data streams without excess resource usage. - The data processing speed is limited by the slowest program in the pipeline. This allows pausing and resuming data processing at any time. For example, by putting 'less' to the end of the pipeline. - The data processing pipeline is terminated instantly as soon as some program in the pipeline terminates. For example, if you put 'tail -100' to the end of pipeline, then only the first 100 lines of data will be processed. VictoriaLogs adapts these properties into logs processing with LogsQL pipes. The difference is that the data, which is flown between pipes, represents structured logs [1]. [1] https://docs.victoriametrics.com/victorialogs/querying/#command-line https://docs.victoriametrics.com/victorialogs/querying/#comm...
2y ago

dangoodmanUT 2y ago

IMO SQL is great for logs, ClickHouse and DuckDB SQL in particular are awesome

lmeyerov 2y ago
I find SQL nice for a happy path - highly structured, non-nested, and simple log scenarios, such as some heavily curated app log subset like an app-level authentication audit log When we have arbitrary logs from all over, and especially system.. I appreciate the sloppy and dynamic nature of SPL and its derivatives We have ported our app tier to OTel metrics/logs/traces, and I've been curious what will be more 'right' in this new era. Likewise, with louie.ai, we have genAI do the text2query for us by default, which shifts things a bit as well.

ople 2y ago

Recently there seems to be an bunch of SPL (Splunk) -like query languages popping up: PQL, PRQL, Grafana Explore Logs syntax, Kusto.. Probably others as well. Does yet another similar but slightly different language make sense? Why not leverage an existing one?

valyala 2y ago
Diversity is great! Let's see which query language for logs will survive.
- ople 2y ago
  True. My bet is that arbitrary text queries converted to SQL via LLM will become an increasingly popular alternative. Not sure if it will completely replace the custom query languages though..

rixrax 2y ago

Am I the only one that feels that EVERYTHING is wrong in this ELK, Splunk, etc. Grafana world? The user interfaces that these monstrosities present us with are barely useable, everyone has their own query language, they force us to install their own agents own our hosts and servers, when I upload logs, many can't even take random JSON logs and input them in a structured way without defining pipeline rules or what now. And did I say that the Logstashes and Promtails and Vectors and what not pipeline tools with their Grok etc. filters feel like somebody wanted to really make busywork cool. I am happy that in my day to day work I can dump my mostly Linux logs to rsyslog, and eventually forward them to S3 glacier for a few years. So I am guessing the question I am asking is that what in the world are you doing with these observability or SIEM platforms and is anyone actually deriving some REAL value from using them?

fragmede 2y ago
Yes there's real value there. That everyone's got their own flavor is a bunch of extra work because we haven't solved the coordination problem yet is annoying, but that's solved by choosing one and sticking with it. Learn that query language really really well, and don't touch anything else. Splunk is useful as all hell once you get over the hump of learning their proprietary query language. it's really friggin useful. it's useful to the tune of Cisco buying them for $28 billion. people are deriving real value from them, the question is what are your problems that this can solve for you, but do you even have those problems in the first place? If you've not found it useful then why are you stuffing logs into S3? just send them to /dev/null instead
- rixrax 2y ago
  >> just send them to /dev/null instead I wish. But 'regulatory compliance'. And 'we might need them' - just not sure for what - but we'll try another data analyst next quarter. Thankfully because of the GDPR (and maybe other reasons) there is a healthy pressure to also cleanse us from the logs we've collected. That said, I agree, based on my trials (and mostly errors), Splunk seems one of the better ones. Not considering the cost anyway. My trouble is that I am not a data analyst, but I get asked more than I would like about these things.

piterrro 2y ago

I'm a https://logdy.dev https://logdy.dev (logs to UI interface) author and been recently thinking about how to enable users use a query language to search throught logs beyond usual filter. I was looking at LogsQL but then I felt that is just another QL a user will need to learn. My next though was on SQL, but it was not designed for this purpose. Any ideas? I would appreciate any recommendation (peter at logd.dev) [1] https://github.com/logdyhq/logdy-core https://github.com/logdyhq/logdy-core

valyala 2y ago
How about allowing users creating pipelines with good old grep, awk, cut, etc., directly in the web UI?

deleted 2y ago

[deleted]