6 ms·

Some notes on Grafana Loki's new "structured metadata"

117 points by valyala 2y ago

kbouck 2y ago
It's also not ideal to have a different query language for different Grafana datastores (LogQL, PromQL, TraceQL). Are there any plans on making a unified Grafana query language?
- pjd7 2y ago
  Unifying things slows engineers down, so probably not (for some time).
  - Wheaties466 2y ago
    well, not having a unified language slows down things on the other end.
- jakozaur 2y ago
  There is an effort in OpenTelemetry to create a standard query language for observability. There were a lot of discussions with a lot of opinions; there were even several talks during KubeConEU about that: https://sched.co/1tcyx https://sched.co/1tcyx https://sched.co/1txI1 https://sched.co/1txI1 We are still waiting for a compelling implementation that will show the way.
  - ople 2y ago
    Why not just use SQL? With LLMs evolving to do sophisticated text-to-SQL, the case for a custom language for the sake of simplicity is diminishing. I think that expressiveness, performance and level of fluency by base language models (i.e. the amount of examples in training set) are the key differentiators for query languages in the future. SQL ticks all those boxes.
    - jakozaur 2y ago
      You are right. SQL is the best language, but it likely needs some extensions. See SQL with pipe syntax. Read Google paper or try it out in Big Query. There are a lot of fundamentals in observability, but there are very verbose in SQL: - rate operator, which translates absolute value to rate, possible with SQL and window functions, but takes many lines of code - pivot, where you like to see the top 5 counts of errors of most hit-by-error microservices plus others over time - sampling is frequent in observability and will be useful for LLMs, it is a one-liner in SQL with pipe syntax, even customizing specific I actually believe LLM gen AI plays extremely well with pipe syntax. It allows us to feed partial results to LLM, sampling as well as show how LLM is evolving queries over time. SQL troubleshooting is not a single query but a series of them. Still, SQL with pipe syntax is just syntactical sugar on SQL. It let's you use all SQL features as well as compiles to SQL.
- pbh101 2y ago
  I think I’m probably not interested in this. PromQL is already relatively dense to learn, but reasonably well fit to the domain model and internally consistent, unlike most other metric querying tools I’ve tried over the years. Maybe that would work as well with traces and logs but IMO the problem space is quite different and not sure how much value we’d get from a unified language where some subsets only apply to parts, ie traces and logs and metric, as opposed to spiritually similar but distinct languages.
DeathArrow 2y ago
Did someone use both Grafana Loki and Kibana? Does it have any advantages over Kibana? I am mostly interested in resource usage and versatility of filtering. In Kibana, if something is there I will find it with ease and it doesn't take a lot of time to investigate issues in a microservice based application. It is also quite fast.
- weitzj 2y ago
  From the Enterprise Perspektive at least for my use cases(fine grained permissions using extra id) , elasticsearch with kibana always had a solution available. For grafana cloud and Loki you can close to a good usability with LBAC (label based access control) but you still need have many data sources to map onto each “team view” to make it user friendly. What is missing for me is like in elastic a single datasource for all logs which every team member across all teams can see and you scope out the visibility level with LBAC
- feydaykyn 2y ago
  Compared over Kibana, we experience: - 3x reduced costs - no more index corruption because a key changed type - slower performance for queries over 1 day, especially when non optimized without any filtering - non intuitive ui/ux So good but not perfect! When we have the time we'll look for alternatives
  - kbouck 2y ago
    Re: storage, Kibana (Elastic) has a new (as of v8.17) "logsdb" index mode which claims to be ~2.5x more storage efficient than previous options.
  - valyala 2y ago
    Did you try VictoriaLogs? It is easier to configure than Loki and Elasticsearch, and it uses less CPU and RAM. See https://itnext.io/how-do-open-source-solutions-for-logs-work-elasticsearch-loki-and-victorialogs-9f7097ecbc2f https://itnext.io/how-do-open-source-solutions-for-logs-work...
    - 2y ago
jiveturkey 2y ago
(2024) important because the title includes _new_
slekker 2y ago
A disclaimer is that OP is CEO of another company in the same sector
- NewJazz 2y ago
  Chris Siebenmann? What company is he the CEO of?
  - fsiefken 2y ago
    No, the OP of the HN thread is from VictoriaMetrics (open source), he’s not Chris Siebenmann, unix systems administrator at the University of Toronto’s Computer Science Labs.
    - duckydude20 2y ago
      i find him everywhere theres is mention of loki. trying so hard to push down loki. sure loki have issues but i don't feel this is a good approach to promote your product.
      - valyala 2y ago
        I'm trying to push Loki developers to fix these issues to make Loki users happy. What's the point to hide these issues? The won't disappear on their own over time. I'm very glad to know about VictoriaLogs issues, so they could be addressed quicker. If you hit such issues, then please file them at https://github.com/VictoriaMetrics/VictoriaMetrics/issues https://github.com/VictoriaMetrics/VictoriaMetrics/issues or publish an article highlighting these issues.
        klysm 2y ago
        The alternative and obvious explanation is that you are make Loki look worse so your product looks better in comparison.
        2y ago
ohgr 2y ago
Not much I agree with in this article. Seems to be based on little operational experience with the product, particular indicated by a couple of major mistakes and assumptions in the article (compacting does happen, didn't read the manual about deployment configurations clearly). Loki has its idiosyncrasies but they are there for a good reason. Anyone who has sat there waiting hours for a Kibana or Splunk query to run to get some information out will know what I'd referring to. You don't dragnet your entire log stream unless your logs are terrible, which needs to be fixed, or you don't know when something happened, which needs fixing. I watch many people run queries that scan terabytes of data with gay abandon on a regular basis on older platforms and still never get what they need out. The structured metadata distinction is important because when you do a query against that you are not using an index, just parsed out data. That means explicitly you're not filtering, you're scanning and that is expensive. If you have a problem with finding things, then it's not the logging engine, it's the logs!
xyz-x 2y ago
Does anyone have experience using Apache Iceberg, Hudi or DeltaLake for storing logs instead of Loki?
- pbh101 2y ago
  danluu.com mentioned this approach (or just, ‘big data’ systems) for traces and metrics iirc. Not sure if for logs too. Aren’t all of those relatively tabular? What would you be looking for from those tools to help with logs?
ptman 2y ago
@valyala , as others have noted, you are CEO of VictoriaMetrics and have written (most of?) VictoriaLogs. How is VictoriaLogs coming along? This is an older blog post.
- Dusseldorf 2y ago
  I switched our team over to VictoriaLogs from ELK when VL1.0 was released a few months back and we've been very happy with it. Nowhere near as much finicky performance tuning, no more logs failing to ingest because a string looked a bit too numeric, and the query language has fewer weird gotchas. At the end of the day ELK was throwing us a bunch of roadblocks in order to solve problems we didn't need solved. Maybe if we were trying to build some big analysis layer on top of our logs that would've been nice. VL has worked great for our use case of needing to centralize and view logs.
- valyala 2y ago
  VictoriaLogs is free from issues mentioned in the referred article. It supports log fields with big number of unique values (such as user_id, trace_id, ip, etc.) from the beginning, and it doesn't need any configuration for working with such fields. It automatically indexes all the log fields and provides fast full-text search over all the ingested log fields. This is explained in more details at https://itnext.io/why-victorialogs-is-a-better-alternative-to-grafana-loki-7e941567c4d5 https://itnext.io/why-victorialogs-is-a-better-alternative-t...

117 points by valyala 2y ago

It's also not ideal to have a different query language for different Grafana datastores (LogQL, PromQL, TraceQL). Are there any plans on making a unified Grafana query language?

pjd7 2y ago
Unifying things slows engineers down, so probably not (for some time).
- Wheaties466 2y ago
  well, not having a unified language slows down things on the other end.
jakozaur 2y ago
There is an effort in OpenTelemetry to create a standard query language for observability. There were a lot of discussions with a lot of opinions; there were even several talks during KubeConEU about that: https://sched.co/1tcyx https://sched.co/1tcyx https://sched.co/1txI1 https://sched.co/1txI1 We are still waiting for a compelling implementation that will show the way.
- ople 2y ago
  Why not just use SQL? With LLMs evolving to do sophisticated text-to-SQL, the case for a custom language for the sake of simplicity is diminishing. I think that expressiveness, performance and level of fluency by base language models (i.e. the amount of examples in training set) are the key differentiators for query languages in the future. SQL ticks all those boxes.
  - jakozaur 2y ago
    You are right. SQL is the best language, but it likely needs some extensions. See SQL with pipe syntax. Read Google paper or try it out in Big Query. There are a lot of fundamentals in observability, but there are very verbose in SQL: - rate operator, which translates absolute value to rate, possible with SQL and window functions, but takes many lines of code - pivot, where you like to see the top 5 counts of errors of most hit-by-error microservices plus others over time - sampling is frequent in observability and will be useful for LLMs, it is a one-liner in SQL with pipe syntax, even customizing specific I actually believe LLM gen AI plays extremely well with pipe syntax. It allows us to feed partial results to LLM, sampling as well as show how LLM is evolving queries over time. SQL troubleshooting is not a single query but a series of them. Still, SQL with pipe syntax is just syntactical sugar on SQL. It let's you use all SQL features as well as compiles to SQL.
pbh101 2y ago
I think I’m probably not interested in this. PromQL is already relatively dense to learn, but reasonably well fit to the domain model and internally consistent, unlike most other metric querying tools I’ve tried over the years. Maybe that would work as well with traces and logs but IMO the problem space is quite different and not sure how much value we’d get from a unified language where some subsets only apply to parts, ie traces and logs and metric, as opposed to spiritually similar but distinct languages.

DeathArrow 2y ago

Did someone use both Grafana Loki and Kibana? Does it have any advantages over Kibana? I am mostly interested in resource usage and versatility of filtering. In Kibana, if something is there I will find it with ease and it doesn't take a lot of time to investigate issues in a microservice based application. It is also quite fast.

weitzj 2y ago
From the Enterprise Perspektive at least for my use cases(fine grained permissions using extra id) , elasticsearch with kibana always had a solution available. For grafana cloud and Loki you can close to a good usability with LBAC (label based access control) but you still need have many data sources to map onto each “team view” to make it user friendly. What is missing for me is like in elastic a single datasource for all logs which every team member across all teams can see and you scope out the visibility level with LBAC
feydaykyn 2y ago
Compared over Kibana, we experience: - 3x reduced costs - no more index corruption because a key changed type - slower performance for queries over 1 day, especially when non optimized without any filtering - non intuitive ui/ux So good but not perfect! When we have the time we'll look for alternatives
- kbouck 2y ago
  Re: storage, Kibana (Elastic) has a new (as of v8.17) "logsdb" index mode which claims to be ~2.5x more storage efficient than previous options.
- valyala 2y ago
  Did you try VictoriaLogs? It is easier to configure than Loki and Elasticsearch, and it uses less CPU and RAM. See https://itnext.io/how-do-open-source-solutions-for-logs-work-elasticsearch-loki-and-victorialogs-9f7097ecbc2f https://itnext.io/how-do-open-source-solutions-for-logs-work...
  - 2y ago

jiveturkey 2y ago

(2024) important because the title includes _new_

slekker 2y ago

A disclaimer is that OP is CEO of another company in the same sector

NewJazz 2y ago
Chris Siebenmann? What company is he the CEO of?
- fsiefken 2y ago
  No, the OP of the HN thread is from VictoriaMetrics (open source), he’s not Chris Siebenmann, unix systems administrator at the University of Toronto’s Computer Science Labs.
  - duckydude20 2y ago
    i find him everywhere theres is mention of loki. trying so hard to push down loki. sure loki have issues but i don't feel this is a good approach to promote your product.
    - valyala 2y ago
      I'm trying to push Loki developers to fix these issues to make Loki users happy. What's the point to hide these issues? The won't disappear on their own over time. I'm very glad to know about VictoriaLogs issues, so they could be addressed quicker. If you hit such issues, then please file them at https://github.com/VictoriaMetrics/VictoriaMetrics/issues https://github.com/VictoriaMetrics/VictoriaMetrics/issues or publish an article highlighting these issues.
      - klysm 2y ago
        The alternative and obvious explanation is that you are make Loki look worse so your product looks better in comparison.
        2y ago

ohgr 2y ago

Not much I agree with in this article. Seems to be based on little operational experience with the product, particular indicated by a couple of major mistakes and assumptions in the article (compacting does happen, didn't read the manual about deployment configurations clearly). Loki has its idiosyncrasies but they are there for a good reason. Anyone who has sat there waiting hours for a Kibana or Splunk query to run to get some information out will know what I'd referring to. You don't dragnet your entire log stream unless your logs are terrible, which needs to be fixed, or you don't know when something happened, which needs fixing. I watch many people run queries that scan terabytes of data with gay abandon on a regular basis on older platforms and still never get what they need out. The structured metadata distinction is important because when you do a query against that you are not using an index, just parsed out data. That means explicitly you're not filtering, you're scanning and that is expensive. If you have a problem with finding things, then it's not the logging engine, it's the logs!

xyz-x 2y ago

Does anyone have experience using Apache Iceberg, Hudi or DeltaLake for storing logs instead of Loki?

pbh101 2y ago
danluu.com mentioned this approach (or just, ‘big data’ systems) for traces and metrics iirc. Not sure if for logs too. Aren’t all of those relatively tabular? What would you be looking for from those tools to help with logs?

ptman 2y ago

@valyala , as others have noted, you are CEO of VictoriaMetrics and have written (most of?) VictoriaLogs. How is VictoriaLogs coming along? This is an older blog post.

Dusseldorf 2y ago
I switched our team over to VictoriaLogs from ELK when VL1.0 was released a few months back and we've been very happy with it. Nowhere near as much finicky performance tuning, no more logs failing to ingest because a string looked a bit too numeric, and the query language has fewer weird gotchas. At the end of the day ELK was throwing us a bunch of roadblocks in order to solve problems we didn't need solved. Maybe if we were trying to build some big analysis layer on top of our logs that would've been nice. VL has worked great for our use case of needing to centralize and view logs.
valyala 2y ago
VictoriaLogs is free from issues mentioned in the referred article. It supports log fields with big number of unique values (such as user_id, trace_id, ip, etc.) from the beginning, and it doesn't need any configuration for working with such fields. It automatically indexes all the log fields and provides fast full-text search over all the ingested log fields. This is explained in more details at https://itnext.io/why-victorialogs-is-a-better-alternative-to-grafana-loki-7e941567c4d5 https://itnext.io/why-victorialogs-is-a-better-alternative-t...