9 ms·

Preview of Explore Logs, a new way to browse your logs without writing LogQL

193 points by matryer 2y ago

candiddevmike 2y ago
Why have explore logs as a separate app instead of bundled with Loki? It would be nice if Loki had the same kind of barebones querying/debugging functionality as Prometheus...
- skrtskrt 2y ago
  Loki is just the backend just like Prometheus is just the backend
  - pluies 2y ago
    Yeah but Prometheus has a web ui where you can run PromQL queries and it'll give you basic graphs back, which is handy for throwing a quick query at it before putting it into something more long-term like a Grafana dashboard or an alerting rule.
    - skrtskrt 2y ago
      wow I always thought that Prom UI was just a tacked-on part of alertmanager or something because it’s so rudimentary. In my experience, everyone just uses Grafana Explore since that’s what Grafana was originally purpose-built for and it’s crazy easy to set up. Just pull down a container or helm chart or whatever. Since Grafana built Loki, it doesn't make any sense why Grafana would create a separate querying UI app for Loki when they already have Grafana Explore. Prometheus (and I assume its UI) was created by Google [edit: sorry, created by SoundCloud, inspired by a Google Borg tool] before Grafana became the de facto Prometheus query UI, so it’s not really analogous.
      - richardwhiuk 2y ago
        Prometheus was created by SoundCloud, not Google, but was inspired by the Google Borgmon tool.
jdoss 2y ago
I recently setup Victoria Metrics + https://github.com/prometheus/snmp_exporter https://github.com/prometheus/snmp_exporter + Grafana to get start tracking bandwidth on my top of rack switches in my Datacenter rack which has been a pretty awesome setup. The way you can auto generate a config for your SNMP MIBs with SNMP Exporter was unexpectedly not a terrible experience. My next task is to get centralized logging going with Victoria Logs + Vector, I'll have to check this out once I get everything setup. I believe I can use LogQL with Victoria Logs but I haven't tried it out yet. https://docs.victoriametrics.com/victorialogs/logsql/ https://docs.victoriametrics.com/victorialogs/logsql/
- NortySpock 2y ago
  I've been eyeing a VictoriaLogs setup for my docker container fleet, but I haven't quite spotted where docker's remote logging export options overlap with VictoriaLogs ingestion options. Wrinkle: two docker remote logging plugins I tried (e.g. loki, elastic) didn't seem to work on ARM processors out of the box.
  - jdoss 2y ago
    Check out Vector for shipping logs from Docker. It might work out for you https://vector.dev/docs/reference/configuration/sources/docker_logs/ https://vector.dev/docs/reference/configuration/sources/dock... I use Podman for all of my container stuff and there are issues with how Podman produces JSON logs https://github.com/vectordotdev/vector/issues/6807 https://github.com/vectordotdev/vector/issues/6807 https://github.com/containers/podman/issues/16317 https://github.com/containers/podman/issues/16317 which needs to get fixed before I can use it for my workloads.
- nklmilojevic 2y ago
  This is what I've been doing on my cluster: https://github.com/nklmilojevic/home/blob/main/kubernetes/apps/monitoring/victorialogs/app/helm-release.yaml https://github.com/nklmilojevic/home/blob/main/kubernetes/ap... https://github.com/nklmilojevic/home/tree/main/kubernetes/apps/monitoring/vector/app https://github.com/nklmilojevic/home/tree/main/kubernetes/ap... Here you have Vector in aggregator + agent mode and several sources. VictoriaLogs also recently added Grafana datasource so it is fairly easy to set it up: https://github.com/nklmilojevic/home/blob/main/kubernetes/apps/monitoring/grafana/app/helmrelease.yaml#L32 https://github.com/nklmilojevic/home/blob/main/kubernetes/ap... I'm a big fan of VictoriaMetrics as well and we use it extensively in my company at high scale.
deleted 2y ago
[deleted]
vbezhenar 2y ago
I thought it was a standalone web app, but it's integrated into Grafana. I'm confused a bit. There's already Explore functionality in Grafana for Loki. Seems like spreading the efforts for no reason.
School-Cotton 2y ago
I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago. It seems nobody else cares about this use case and wants you to use LogQL and the incredibly clunky Grafana web UI instead, because it makes it possible to aggregate across many different processes, slice and dice by various labels, etc., which as I said, I have never (or almost never) actually wanted to do. Hopefully this new UI is a step in the right direction as people won't need to futz around with LogQL anymore, but it seems like it still doesn't quite do what I want.
- nine_k 2y ago
  Could LogQL do.something like select * from stdout, stderr where session_id = 123456 ? If not, why?
  - Matthias247 2y ago
    yes it can, if you tag your log stream correctly - either by having the stream externally tagged via attributes, or internally by following certain conventions in the log line. You can also do something like select client_ip from requests where elapsed_ms > 10000 which is incredibly powerful
  - skrtskrt 2y ago
    yep, with the caveat that you probably don't want to have the backend of whatever log system you use (not exactly sure how Loki does it) to have an index on something as high-cardinality as session id so that query could get slow. But these log query systems can also optimize these queries for instance by by sampling, using distributed trace ids to ensure you get shown corresponding, allowing you to get only logs where at least one step in the trace errored, etc.
  - westurner 2y ago
    strace and gdb can trace and close and reopen process file handles 0,1,2. ldpreloadhook has an example of hooking write() with LD_PRELOAD=, which e.g. golang programs built without libc don't support. When systemd is /sbin/init, it owns all subprocess' file handles already, so there's no need to close(0), time, open(0) with gdb. Without having to logship (copy buffers that are flushed and/or have newline characters in the stream) to a network or local Arrow database files and or SQLite vtables, journalctl (journald) supports pattern matching with: -t syslogidentifier, -u unit; and -g grepexpr of the MESSAGE= field: journalctl -u <TAB> journalctl -u init.scope --reverse journalctl -u unit.scope -g "Reached target" # and then "/sleep" to search and highlight with less journalctl -u auditd.service # this is slow because it's a full table scan, because # journald does not index the logfiles; # and -g/--grep is case insensitive if the query is all lowercase: journalctl -g avc --reverse journalctl -g AVC --reverse # this is faster: journalctl -t audit -g AVC -r # this is still faster, # because it only searches the current boot: journalctl -b 0 -t audit -g AVC # these are equivalent: journalctl -b 0 --dmesg -t kernel journalctl -k # journalctl -b 0 --user | grep -i -C "xyz123" There is a GNOME Logs viewer that has 'All' and a few mutually exclusive filter/reports in a side pane, and a search expression field to narrow a filter/report like All or Important. There is a Grafana Loki Docker Driver that logships from all containers visible on that DOCKER_HOST docker socket to Grafana for querying with Loki: https://grafana.com/docs/loki/latest/send-data/docker-driver/ https://grafana.com/docs/loki/latest/send-data/docker-driver... Podman with Systemd doesn't need the Grafana Docker Driver (or other logshippers like logstash, loggly, or fluentd) because systemd spawns containers and optionally pipes their stdout/stderr logs to journald. Influx has Telegraf, InfluxDB, Chronograf, and Kapacitor. Chronograf is their WebUI which provides a query interface for configurable chart dashboards and InfluxQL. Grafana supports SQL, PromQL, InfluxQL, and LogQL. Graylog2 also indexes logfiles. But you can't query stdout and stderr you or /sbin/init haven't logged to a file.
nicoritschel 2y ago
While this is a step in the right direction, just let me write something closer to SQL. Influx did this correctly.
- deleted 2y ago
  [deleted]
- willseth 2y ago
  That’s what LogQL is, which is already in Loki. This is a new feature.
  - hamandcheese 2y ago
    LogQL is nothing like SQL. Try aggregating and quickly you'll be asking yourself wtf an instant query is and how is that different.
    - willseth 2y ago
      I mean, time series data is different from generic tabular data, so obviously there are impedance differences that are reflected in the query languages. I can see how some people might feel more at home using something even more like SQL, but there are a lot of common use cases where SQL is awkward and/or more verbose.
      - hamandcheese 2y ago
        Are logs time series data? That seems to be the thesis behind LogQL. But way more often than not I'm searching for a needle in a haystack, not charting trends over time.
- remram 2y ago
  Influx used SQL, then deprecated it and made everyone use Flux, then deprecated that and moved back to SQL. They are definitely not "doing it correctly" when it comes to query language.
wsatb 2y ago
I'll preface this with the fact I haven't look at Loki in a bit, so maybe this has changed. But I found the documentation needing a lot of work and the configuration for promtail to be obtuse and not very user friendly. I haven't used it for those reasons, not because of the query language.
- John23832 2y ago
  Having used Loki/promagent etc, it was sort of a pain/nonintuitive to set up.
- corytheboyd 2y ago
  They gotta sell their managed cloud service somehow, I have always assumed that this is part of the sales strategy
- liampulles 2y ago
  Our team uses loki and I have to say I think their collected helm charts are pretty easy to use - my problem is more that it seems to be quite slow to run on-prem. Very often my loki query times out and I have to do more work filtering down the log lines or selecting a narrower time range. I'm kind of amazed the UI doesn't select small time ranges iteratively to build up the response, especially since I believe this is what the CLI does. Perhaps this is also part of their cloud offering provides and it is part of their marketing strategy. Not a good one because if we came down to the decision I would start by looking for something else from being p'ed off by Loki. But I guess it still works pretty well considering it is free.
  - skrtskrt 2y ago
    Loki UI in Grafana Explore seems to only select 1000 lines by default for me? Also Loki on the backend splits/parallelizes requests if it can. The Grafana backend Mimir / Loki / Tempo products all appear to be architected pretty similar, and I'm more experienced operating Mimir, but the answer to read load often has to just do with right-sizing the deployment scale, and using caches aggressively.
- blue_cookeh 2y ago
deleted 2y ago
[deleted]
damm 2y ago
This is hard coded to searching for service_name in the query which doesn't return any data for me. I will stop wasting my time here and try the Metric Explore panel
dangoodmanUT 2y ago
LogQL is honestly fine, it's not SQL, but it's fine
pachico 2y ago
I am a happy customer of Grafana Cloud, yet, I can't use their backends for logs and metrics as they are terribly expensive and slow. Somehow, VictoriaMetrics manages to provide much better results.
hamandcheese 2y ago
LogQL so far just does not click for me. I get that it's trying to be like Prometheus, but logs are not the same as time series - we have each and every log! So why am I forced to query it like a time series data source? I want to query my logs like a SQL table, not a time series database.
saintfiends 2y ago
Loki OSS is just a sales pitch for their managed service. It doesn't work well without dedicating significant time tweaking and configuring it. Documentation is confusing at best if you want to do anything serious. You have to also be ready to handle support calls if you open it up for others to use, because it WILL have issues fairly regularly if you have a good volume of logs and query range is more than a day or two. Unless you have the bank to go with their managed service, don't bother.
Spiwux 2y ago
After having used Datadog for several years, going back to Grafana / Loki / Prometheus felt like regressing by two decades. As much as I appreciate free solutions, I feel like Grafana has really fallen behind when it comes to developer experience
- nthngtshr 2y ago
  Could you provide more details? Although I've never had the opportunity to use Datadog at any of my previous positions, I am quite familiar with Grafana and I'm generally pretty happy with it. What's the TL;DR for why Datadog is better?
- zikohh 2y ago
  Grafana cloud is better for querying logs. Grafana cloud is probably a bit better for querying metrics. Grafana cloud is terrible at finding traces or even loading them. Datadog is lightyears ahead. For alerting I feel datadog has better features but is overwhelming with all the different options. grafana is very quirky for searching for traces. And has a huge learning curve.
habitue 2y ago
After going all in on tracing and ignoring logs entirely... I gotta say I'm really glad I never have to deal with logs.

193 points by matryer 2y ago

Why have explore logs as a separate app instead of bundled with Loki? It would be nice if Loki had the same kind of barebones querying/debugging functionality as Prometheus...

skrtskrt 2y ago
Loki is just the backend just like Prometheus is just the backend
- pluies 2y ago
  Yeah but Prometheus has a web ui where you can run PromQL queries and it'll give you basic graphs back, which is handy for throwing a quick query at it before putting it into something more long-term like a Grafana dashboard or an alerting rule.
  - skrtskrt 2y ago
    wow I always thought that Prom UI was just a tacked-on part of alertmanager or something because it’s so rudimentary. In my experience, everyone just uses Grafana Explore since that’s what Grafana was originally purpose-built for and it’s crazy easy to set up. Just pull down a container or helm chart or whatever. Since Grafana built Loki, it doesn't make any sense why Grafana would create a separate querying UI app for Loki when they already have Grafana Explore. Prometheus (and I assume its UI) was created by Google [edit: sorry, created by SoundCloud, inspired by a Google Borg tool] before Grafana became the de facto Prometheus query UI, so it’s not really analogous.
    - richardwhiuk 2y ago
      Prometheus was created by SoundCloud, not Google, but was inspired by the Google Borgmon tool.

jdoss 2y ago

I recently setup Victoria Metrics + https://github.com/prometheus/snmp_exporter https://github.com/prometheus/snmp_exporter + Grafana to get start tracking bandwidth on my top of rack switches in my Datacenter rack which has been a pretty awesome setup. The way you can auto generate a config for your SNMP MIBs with SNMP Exporter was unexpectedly not a terrible experience. My next task is to get centralized logging going with Victoria Logs + Vector, I'll have to check this out once I get everything setup. I believe I can use LogQL with Victoria Logs but I haven't tried it out yet. https://docs.victoriametrics.com/victorialogs/logsql/ https://docs.victoriametrics.com/victorialogs/logsql/

NortySpock 2y ago
I've been eyeing a VictoriaLogs setup for my docker container fleet, but I haven't quite spotted where docker's remote logging export options overlap with VictoriaLogs ingestion options. Wrinkle: two docker remote logging plugins I tried (e.g. loki, elastic) didn't seem to work on ARM processors out of the box.
- jdoss 2y ago
  Check out Vector for shipping logs from Docker. It might work out for you https://vector.dev/docs/reference/configuration/sources/docker_logs/ https://vector.dev/docs/reference/configuration/sources/dock... I use Podman for all of my container stuff and there are issues with how Podman produces JSON logs https://github.com/vectordotdev/vector/issues/6807 https://github.com/vectordotdev/vector/issues/6807 https://github.com/containers/podman/issues/16317 https://github.com/containers/podman/issues/16317 which needs to get fixed before I can use it for my workloads.
nklmilojevic 2y ago
This is what I've been doing on my cluster: https://github.com/nklmilojevic/home/blob/main/kubernetes/apps/monitoring/victorialogs/app/helm-release.yaml https://github.com/nklmilojevic/home/blob/main/kubernetes/ap... https://github.com/nklmilojevic/home/tree/main/kubernetes/apps/monitoring/vector/app https://github.com/nklmilojevic/home/tree/main/kubernetes/ap... Here you have Vector in aggregator + agent mode and several sources. VictoriaLogs also recently added Grafana datasource so it is fairly easy to set it up: https://github.com/nklmilojevic/home/blob/main/kubernetes/apps/monitoring/grafana/app/helmrelease.yaml#L32 https://github.com/nklmilojevic/home/blob/main/kubernetes/ap... I'm a big fan of VictoriaMetrics as well and we use it extensively in my company at high scale.

deleted 2y ago

[deleted]

vbezhenar 2y ago

I thought it was a standalone web app, but it's integrated into Grafana. I'm confused a bit. There's already Explore functionality in Grafana for Loki. Seems like spreading the efforts for no reason.

School-Cotton 2y ago

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago. It seems nobody else cares about this use case and wants you to use LogQL and the incredibly clunky Grafana web UI instead, because it makes it possible to aggregate across many different processes, slice and dice by various labels, etc., which as I said, I have never (or almost never) actually wanted to do. Hopefully this new UI is a step in the right direction as people won't need to futz around with LogQL anymore, but it seems like it still doesn't quite do what I want.

nine_k 2y ago
Could LogQL do.something like select * from stdout, stderr where session_id = 123456 ? If not, why?
- Matthias247 2y ago
  yes it can, if you tag your log stream correctly - either by having the stream externally tagged via attributes, or internally by following certain conventions in the log line. You can also do something like select client_ip from requests where elapsed_ms > 10000 which is incredibly powerful
- skrtskrt 2y ago
  yep, with the caveat that you probably don't want to have the backend of whatever log system you use (not exactly sure how Loki does it) to have an index on something as high-cardinality as session id so that query could get slow. But these log query systems can also optimize these queries for instance by by sampling, using distributed trace ids to ensure you get shown corresponding, allowing you to get only logs where at least one step in the trace errored, etc.
- westurner 2y ago
  strace and gdb can trace and close and reopen process file handles 0,1,2. ldpreloadhook has an example of hooking write() with LD_PRELOAD=, which e.g. golang programs built without libc don't support. When systemd is /sbin/init, it owns all subprocess' file handles already, so there's no need to close(0), time, open(0) with gdb. Without having to logship (copy buffers that are flushed and/or have newline characters in the stream) to a network or local Arrow database files and or SQLite vtables, journalctl (journald) supports pattern matching with: -t syslogidentifier, -u unit; and -g grepexpr of the MESSAGE= field: journalctl -u <TAB> journalctl -u init.scope --reverse journalctl -u unit.scope -g "Reached target" # and then "/sleep" to search and highlight with less journalctl -u auditd.service # this is slow because it's a full table scan, because # journald does not index the logfiles; # and -g/--grep is case insensitive if the query is all lowercase: journalctl -g avc --reverse journalctl -g AVC --reverse # this is faster: journalctl -t audit -g AVC -r # this is still faster, # because it only searches the current boot: journalctl -b 0 -t audit -g AVC # these are equivalent: journalctl -b 0 --dmesg -t kernel journalctl -k # journalctl -b 0 --user | grep -i -C "xyz123" There is a GNOME Logs viewer that has 'All' and a few mutually exclusive filter/reports in a side pane, and a search expression field to narrow a filter/report like All or Important. There is a Grafana Loki Docker Driver that logships from all containers visible on that DOCKER_HOST docker socket to Grafana for querying with Loki: https://grafana.com/docs/loki/latest/send-data/docker-driver/ https://grafana.com/docs/loki/latest/send-data/docker-driver... Podman with Systemd doesn't need the Grafana Docker Driver (or other logshippers like logstash, loggly, or fluentd) because systemd spawns containers and optionally pipes their stdout/stderr logs to journald. Influx has Telegraf, InfluxDB, Chronograf, and Kapacitor. Chronograf is their WebUI which provides a query interface for configurable chart dashboards and InfluxQL. Grafana supports SQL, PromQL, InfluxQL, and LogQL. Graylog2 also indexes logfiles. But you can't query stdout and stderr you or /sbin/init haven't logged to a file.

nicoritschel 2y ago

While this is a step in the right direction, just let me write something closer to SQL. Influx did this correctly.

deleted 2y ago
[deleted]
willseth 2y ago
That’s what LogQL is, which is already in Loki. This is a new feature.
- hamandcheese 2y ago
  LogQL is nothing like SQL. Try aggregating and quickly you'll be asking yourself wtf an instant query is and how is that different.
  - willseth 2y ago
    I mean, time series data is different from generic tabular data, so obviously there are impedance differences that are reflected in the query languages. I can see how some people might feel more at home using something even more like SQL, but there are a lot of common use cases where SQL is awkward and/or more verbose.
    - hamandcheese 2y ago
      Are logs time series data? That seems to be the thesis behind LogQL. But way more often than not I'm searching for a needle in a haystack, not charting trends over time.
remram 2y ago
Influx used SQL, then deprecated it and made everyone use Flux, then deprecated that and moved back to SQL. They are definitely not "doing it correctly" when it comes to query language.

wsatb 2y ago

I'll preface this with the fact I haven't look at Loki in a bit, so maybe this has changed. But I found the documentation needing a lot of work and the configuration for promtail to be obtuse and not very user friendly. I haven't used it for those reasons, not because of the query language.

John23832 2y ago
Having used Loki/promagent etc, it was sort of a pain/nonintuitive to set up.
corytheboyd 2y ago
They gotta sell their managed cloud service somehow, I have always assumed that this is part of the sales strategy
liampulles 2y ago
Our team uses loki and I have to say I think their collected helm charts are pretty easy to use - my problem is more that it seems to be quite slow to run on-prem. Very often my loki query times out and I have to do more work filtering down the log lines or selecting a narrower time range. I'm kind of amazed the UI doesn't select small time ranges iteratively to build up the response, especially since I believe this is what the CLI does. Perhaps this is also part of their cloud offering provides and it is part of their marketing strategy. Not a good one because if we came down to the decision I would start by looking for something else from being p'ed off by Loki. But I guess it still works pretty well considering it is free.
- skrtskrt 2y ago
  Loki UI in Grafana Explore seems to only select 1000 lines by default for me? Also Loki on the backend splits/parallelizes requests if it can. The Grafana backend Mimir / Loki / Tempo products all appear to be architected pretty similar, and I'm more experienced operating Mimir, but the answer to read load often has to just do with right-sizing the deployment scale, and using caches aggressively.
blue_cookeh 2y ago

deleted 2y ago

[deleted]

damm 2y ago

This is hard coded to searching for service_name in the query which doesn't return any data for me. I will stop wasting my time here and try the Metric Explore panel

dangoodmanUT 2y ago

LogQL is honestly fine, it's not SQL, but it's fine

pachico 2y ago

I am a happy customer of Grafana Cloud, yet, I can't use their backends for logs and metrics as they are terribly expensive and slow. Somehow, VictoriaMetrics manages to provide much better results.

hamandcheese 2y ago

LogQL so far just does not click for me. I get that it's trying to be like Prometheus, but logs are not the same as time series - we have each and every log! So why am I forced to query it like a time series data source? I want to query my logs like a SQL table, not a time series database.

saintfiends 2y ago

Loki OSS is just a sales pitch for their managed service. It doesn't work well without dedicating significant time tweaking and configuring it. Documentation is confusing at best if you want to do anything serious. You have to also be ready to handle support calls if you open it up for others to use, because it WILL have issues fairly regularly if you have a good volume of logs and query range is more than a day or two. Unless you have the bank to go with their managed service, don't bother.

Spiwux 2y ago

After having used Datadog for several years, going back to Grafana / Loki / Prometheus felt like regressing by two decades. As much as I appreciate free solutions, I feel like Grafana has really fallen behind when it comes to developer experience

nthngtshr 2y ago
Could you provide more details? Although I've never had the opportunity to use Datadog at any of my previous positions, I am quite familiar with Grafana and I'm generally pretty happy with it. What's the TL;DR for why Datadog is better?
zikohh 2y ago
Grafana cloud is better for querying logs. Grafana cloud is probably a bit better for querying metrics. Grafana cloud is terrible at finding traces or even loading them. Datadog is lightyears ahead. For alerting I feel datadog has better features but is overwhelming with all the different options. grafana is very quirky for searching for traces. And has a huge learning curve.

habitue 2y ago

After going all in on tracing and ignoring logs entirely... I gotta say I'm really glad I never have to deal with logs.