4 ms·

VictoriaLogs 0.1.0:log management and log analytics system from VictoriaMetrics

28 points by yla92 3y ago

mdaniel 3y ago
damn shame about using "!" in place of the (IMHO) far more common "-" syntax: https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#logical-filter https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#lo...
- valyala 3y ago
  Initially LogsQL used the '-' char as 'NOT' shortcut. But lately we decided replacing it with '!' char, since the '-' is frequently used in full-text search phrases and/or field names. For example: 'some-field:some-value'. This could bring more confusion to LogsQL. So we decided to use '!' char as 'NOT' shortcut, since it occurs less frequently than '-' in typical qieries.
  - mdaniel 3y ago
    I mean, it's your QL so you can do as you see fit but surely `_msg:alpha-beta` didn't previously parse as `{"_msg":["alpha", "NOT", "beta"]}` and nor would I expect `alpha-beta:charlie` to split in such a non-whitespace way I guess it boils down to whether LogsQL seeks to separate itself from similar solutions, in which case doing abnormal syntax is great, because there's no "uncanny search syntax valley" to be found, or whether it seeks to ease transition from existing search syntaxes in which case every deviation is one more "oh, right" to keep in one's head while transitioning between the systems - https://lucene.apache.org/core/9_6_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#NOT https://lucene.apache.org/core/9_6_0/queryparser/org/apache/... - https://support.google.com/websearch/answer/2466433 https://support.google.com/websearch/answer/2466433 - https://slack.com/help/articles/202528808-Search-in-Slack#narrow-your-search-with-modifiers-pre-ia https://slack.com/help/articles/202528808-Search-in-Slack#na... likely a bunch more. OTOH, maybe this is sampling bias since I'm sure there are syntaxes out there which use bang negation, too, I just haven't used them as much
    - valyala 3y ago
      Fair reasoning. We need to think more about using the '-' instead of '!' in LogsQL. On the other hand, the '!' char is used in almost every programming language as 'NOT' operator. Probably, this is my professional bias.
valyala 3y ago
VictoriaLogs core developer here. The main motivation for creating VictoriaLogs was not so good experience with the existing open source solutions for logs - mainly ElasticSearch (or Opensearch) and Grafana Loki. These systems have the following annoying issues, which make users unhappy: - Non-trivial setup and operation - Hard to use query language - Too high resource usage (CPU, RAM, disk space, disk IO) - Bad integration with the traditional command-line tools for logs analysis - grep, head, less, jq, awk, etc. So we at VictoriaMetrics decided creating user-friendly open source database for logs - VictoriaLogs. We believe it solves pain points of Elasticsearch and Grafana Loki mentioned above. Try it on your production workloads and compare how it feels to your current solution for logs. https://docs.victoriametrics.com/VictoriaLogs/ https://docs.victoriametrics.com/VictoriaLogs/
- mdaniel 3y ago
  Since you're here, I would really enjoy reading about the operational concerns about running V.L. for real (and my apologies for not digging into the repo for "code is the truth" type docs): - Does it use its own flatfile system, or some novel storage mechanism? - Any plans on offering "S3-ish" store for practically unlimited storage (at, of course, the risk of latency for fetches)? - Does it tolerate running on NFS?
  - valyala 3y ago
    - VictoriaLogs uses its own storage format, which is inspired by ClickHouse storage format [1]. This is basically a column storage format optimized for structured logs with arbitrary number of fields. - Yes, VictoriaLogs is going to support object storage. See our roadmap [2] - VictoriaLogs should work good on NFS. It writes data to disk in big sequential chunks, which should work optimally on NFS. It reads data in parallel from multiple threads during queries. This should help increasing query performance on high-latency storage such as NFS. [1] https://clickhouse.com/docs/en/development/architecture https://clickhouse.com/docs/en/development/architecture [2] https://docs.victoriametrics.com/VictoriaLogs/Roadmap.html https://docs.victoriametrics.com/VictoriaLogs/Roadmap.html

28 points by yla92 3y ago

damn shame about using "!" in place of the (IMHO) far more common "-" syntax: https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#logical-filter https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html#lo...

valyala 3y ago
Initially LogsQL used the '-' char as 'NOT' shortcut. But lately we decided replacing it with '!' char, since the '-' is frequently used in full-text search phrases and/or field names. For example: 'some-field:some-value'. This could bring more confusion to LogsQL. So we decided to use '!' char as 'NOT' shortcut, since it occurs less frequently than '-' in typical qieries.
- mdaniel 3y ago
  I mean, it's your QL so you can do as you see fit but surely `_msg:alpha-beta` didn't previously parse as `{"_msg":["alpha", "NOT", "beta"]}` and nor would I expect `alpha-beta:charlie` to split in such a non-whitespace way I guess it boils down to whether LogsQL seeks to separate itself from similar solutions, in which case doing abnormal syntax is great, because there's no "uncanny search syntax valley" to be found, or whether it seeks to ease transition from existing search syntaxes in which case every deviation is one more "oh, right" to keep in one's head while transitioning between the systems - https://lucene.apache.org/core/9_6_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#NOT https://lucene.apache.org/core/9_6_0/queryparser/org/apache/... - https://support.google.com/websearch/answer/2466433 https://support.google.com/websearch/answer/2466433 - https://slack.com/help/articles/202528808-Search-in-Slack#narrow-your-search-with-modifiers-pre-ia https://slack.com/help/articles/202528808-Search-in-Slack#na... likely a bunch more. OTOH, maybe this is sampling bias since I'm sure there are syntaxes out there which use bang negation, too, I just haven't used them as much
  - valyala 3y ago
    Fair reasoning. We need to think more about using the '-' instead of '!' in LogsQL. On the other hand, the '!' char is used in almost every programming language as 'NOT' operator. Probably, this is my professional bias.

valyala 3y ago

VictoriaLogs core developer here. The main motivation for creating VictoriaLogs was not so good experience with the existing open source solutions for logs - mainly ElasticSearch (or Opensearch) and Grafana Loki. These systems have the following annoying issues, which make users unhappy: - Non-trivial setup and operation - Hard to use query language - Too high resource usage (CPU, RAM, disk space, disk IO) - Bad integration with the traditional command-line tools for logs analysis - grep, head, less, jq, awk, etc. So we at VictoriaMetrics decided creating user-friendly open source database for logs - VictoriaLogs. We believe it solves pain points of Elasticsearch and Grafana Loki mentioned above. Try it on your production workloads and compare how it feels to your current solution for logs. https://docs.victoriametrics.com/VictoriaLogs/ https://docs.victoriametrics.com/VictoriaLogs/

mdaniel 3y ago
Since you're here, I would really enjoy reading about the operational concerns about running V.L. for real (and my apologies for not digging into the repo for "code is the truth" type docs): - Does it use its own flatfile system, or some novel storage mechanism? - Any plans on offering "S3-ish" store for practically unlimited storage (at, of course, the risk of latency for fetches)? - Does it tolerate running on NFS?
- valyala 3y ago
  - VictoriaLogs uses its own storage format, which is inspired by ClickHouse storage format [1]. This is basically a column storage format optimized for structured logs with arbitrary number of fields. - Yes, VictoriaLogs is going to support object storage. See our roadmap [2] - VictoriaLogs should work good on NFS. It writes data to disk in big sequential chunks, which should work optimally on NFS. It reads data in parallel from multiple threads during queries. This should help increasing query performance on high-latency storage such as NFS. [1] https://clickhouse.com/docs/en/development/architecture https://clickhouse.com/docs/en/development/architecture [2] https://docs.victoriametrics.com/VictoriaLogs/Roadmap.html https://docs.victoriametrics.com/VictoriaLogs/Roadmap.html