4 ms·

Did someone use both Grafana Loki and Kibana? Does it have any advantages over Kibana? I am mostly interested in resource usage and versatility of filtering. I

Did someone use both Grafana Loki and Kibana? Does it have any advantages over Kibana? I am mostly interested in resource usage and versatility of filtering. In Kibana, if something is there I will find it with ease and it doesn't take a lot of time to investigate issues in a microservice based application. It is also quite fast.

weitzj 2y ago
From the Enterprise Perspektive at least for my use cases(fine grained permissions using extra id) , elasticsearch with kibana always had a solution available. For grafana cloud and Loki you can close to a good usability with LBAC (label based access control) but you still need have many data sources to map onto each “team view” to make it user friendly. What is missing for me is like in elastic a single datasource for all logs which every team member across all teams can see and you scope out the visibility level with LBAC
feydaykyn 2y ago
Compared over Kibana, we experience: - 3x reduced costs - no more index corruption because a key changed type - slower performance for queries over 1 day, especially when non optimized without any filtering - non intuitive ui/ux So good but not perfect! When we have the time we'll look for alternatives
- kbouck 2y ago
  Re: storage, Kibana (Elastic) has a new (as of v8.17) "logsdb" index mode which claims to be ~2.5x more storage efficient than previous options.
- valyala 2y ago
  Did you try VictoriaLogs? It is easier to configure than Loki and Elasticsearch, and it uses less CPU and RAM. See https://itnext.io/how-do-open-source-solutions-for-logs-work-elasticsearch-loki-and-victorialogs-9f7097ecbc2f https://itnext.io/how-do-open-source-solutions-for-logs-work...
  - feydaykyn 2y ago
    Stop spamming. You want your product known? Create a dedicated post.
Unroasted6154 2y ago
Loki was much cheaper to run in my experience, using S3 storage. And you could scale the parts you needed dynamically in K8s. Elastic was kind of a resource hog and much more expensive for the same amount of data. That might be dependent on your use case though.
suraci 2y ago
1. It is extremely resource-efficient. 2. It has a convenient and simple query language. 3. It works very well with traces and metrics. the pain part: 1. It struggles to query logs over a wide time range. 2. Its indexing (or labeling) capabilities are very limited, similar to Prometheus. 3. Due to 1 and 2, it is difficult to configure and use correctly to avoid errors related to usage limits (e.g., maximum series limits).
- arcanemachiner 2y ago
  Which its is it that your its are referring to?
  - suraci 2y ago
    loki
- deleted 2y ago
  [deleted]
- valyala 2y ago
  > 2. It has a convenient and simple query language IMHO, Loki query language is the most inconvenient language for logs I've seen: - It doesn't support calculating multiple stats in a single query. For example, it cannot calculate the number of logs and the average request duration in a single query. - Its' syntax for aggregate functions is very unintuitive and is hard to use, especially if you aren't familiar with PromQL. - It requires putting an annoying "|=" separator between words and phrases you are searching in logs. - You need to use a hack with JSON parsing when filtering or stats calculations on log fields is needed.
ohgr 2y ago
Kibana + ElasticSearch was a mess for us. Was glad to get rid of it. Cost a fortune to run and was time consuming. Loki conversely doesn’t even show up on our costs report (other than the S3 bucket) and requires very little if any maintenance! Also out of box configuration sinks 1TB/hr quite happily in microservices mode.
- valyala 2y ago
  How much CPU and RAM do all the Loki components use for your workload? Could you share Loki config, which can deal with 1TB/hr volume of logs?
  - ohgr 2y ago
    No idea. It’s their out of box microservices set up. I think the only thing we pumped up was the chunks cache to 30Gb.
jakozaur 2y ago
Kibana is great, but managing Elasticsearch is pain. Slow ingestion, query performance, takes a lot of space, harder to tune up. Modern columnar SQL such as ClickHouse are 10+ times more efficient in real-world use cases. I'm a CEO and founder of Quesma, which, let's use Kibana with ClickHouse: https://quesma.com/ https://quesma.com/ Forever free, source-available license.
remram 2y ago
ELK could never deal with my logs which are sometimes-JSON. Loki can ingest and query it just fine. Also the query/extraction language makes a lot more sense to me.
- cortesoft 2y ago
  Couldn’t you use the logstash part of ELK to process the JSON?
  - remram 2y ago
    JSON no problem, sometimes-JSON no. It would drop the not-JSON message entirely.
- valyala 2y ago
  Elasticsearch can store arbitrary text in log fields, including JSON-encoded string. Elasticsearch can also tokenize JSON-encoded string and provide fast full-text search over such string in the same way like it does for a regular plaintext string. why do you need storing JSON-encoded string inside log field? It is much better parsing the JSON into separate fields at log shipper and storing the parsed log fields into Elasticsearch. This gives better query performance and may also reduce disk space usage, since values for every parsed field are stored separately (this usually improves compression ratio and reduces disk read IO during queries if column-oriented storage is used for per-field data). I tried explaining this at https://itnext.io/why-victorialogs-is-a-better-alternative-to-grafana-loki-7e941567c4d5 https://itnext.io/why-victorialogs-is-a-better-alternative-t...
  - remram 2y ago
    The problem is that log messages are collected from many researcher-authored applications. Some of them are JSON, in which case I want to parse them and store the parsed JSON as you say. But some of them are not JSON, in which case I need to store the text. I was not able to do that with the log shipper. If I configured parsing, then the not-JSON messages would get dropped.
- 2y ago
parliament32 2y ago
Yes, we switched metrics and logs from an Elastic stack to Prometheus/Thanos/Loki/Grafana about two years ago. On the logs side specifically, resource usage is WAY lower (300eps is like 1.5 cores and 4gb of memory), not to mention going from persistent volumes (disks) to blob storage / S3 is far cheaper and doesn't require any maintenance. Queries are slower, however, because Elastic pre-indexes while Loki searches on-demand, so it really comes down to query volume and your need for query performance (does it matter if your search takes 300ms vs 3s?). I've also found running Elastic yourself requires constant maintenance, while Loki has been very hands-off. Strongly recommend.

weitzj 2y ago

From the Enterprise Perspektive at least for my use cases(fine grained permissions using extra id) , elasticsearch with kibana always had a solution available. For grafana cloud and Loki you can close to a good usability with LBAC (label based access control) but you still need have many data sources to map onto each “team view” to make it user friendly. What is missing for me is like in elastic a single datasource for all logs which every team member across all teams can see and you scope out the visibility level with LBAC

feydaykyn 2y ago

Compared over Kibana, we experience: - 3x reduced costs - no more index corruption because a key changed type - slower performance for queries over 1 day, especially when non optimized without any filtering - non intuitive ui/ux So good but not perfect! When we have the time we'll look for alternatives

kbouck 2y ago
Re: storage, Kibana (Elastic) has a new (as of v8.17) "logsdb" index mode which claims to be ~2.5x more storage efficient than previous options.
valyala 2y ago
Did you try VictoriaLogs? It is easier to configure than Loki and Elasticsearch, and it uses less CPU and RAM. See https://itnext.io/how-do-open-source-solutions-for-logs-work-elasticsearch-loki-and-victorialogs-9f7097ecbc2f https://itnext.io/how-do-open-source-solutions-for-logs-work...
- feydaykyn 2y ago
  Stop spamming. You want your product known? Create a dedicated post.

Unroasted6154 2y ago

Loki was much cheaper to run in my experience, using S3 storage. And you could scale the parts you needed dynamically in K8s. Elastic was kind of a resource hog and much more expensive for the same amount of data. That might be dependent on your use case though.

suraci 2y ago

1. It is extremely resource-efficient. 2. It has a convenient and simple query language. 3. It works very well with traces and metrics. the pain part: 1. It struggles to query logs over a wide time range. 2. Its indexing (or labeling) capabilities are very limited, similar to Prometheus. 3. Due to 1 and 2, it is difficult to configure and use correctly to avoid errors related to usage limits (e.g., maximum series limits).

arcanemachiner 2y ago
Which its is it that your its are referring to?
- suraci 2y ago
  loki
deleted 2y ago
[deleted]
valyala 2y ago
> 2. It has a convenient and simple query language IMHO, Loki query language is the most inconvenient language for logs I've seen: - It doesn't support calculating multiple stats in a single query. For example, it cannot calculate the number of logs and the average request duration in a single query. - Its' syntax for aggregate functions is very unintuitive and is hard to use, especially if you aren't familiar with PromQL. - It requires putting an annoying "|=" separator between words and phrases you are searching in logs. - You need to use a hack with JSON parsing when filtering or stats calculations on log fields is needed.

ohgr 2y ago

Kibana + ElasticSearch was a mess for us. Was glad to get rid of it. Cost a fortune to run and was time consuming. Loki conversely doesn’t even show up on our costs report (other than the S3 bucket) and requires very little if any maintenance! Also out of box configuration sinks 1TB/hr quite happily in microservices mode.

valyala 2y ago
How much CPU and RAM do all the Loki components use for your workload? Could you share Loki config, which can deal with 1TB/hr volume of logs?
- ohgr 2y ago
  No idea. It’s their out of box microservices set up. I think the only thing we pumped up was the chunks cache to 30Gb.

jakozaur 2y ago

Kibana is great, but managing Elasticsearch is pain. Slow ingestion, query performance, takes a lot of space, harder to tune up. Modern columnar SQL such as ClickHouse are 10+ times more efficient in real-world use cases. I'm a CEO and founder of Quesma, which, let's use Kibana with ClickHouse: https://quesma.com/ https://quesma.com/ Forever free, source-available license.

remram 2y ago

ELK could never deal with my logs which are sometimes-JSON. Loki can ingest and query it just fine. Also the query/extraction language makes a lot more sense to me.

cortesoft 2y ago
Couldn’t you use the logstash part of ELK to process the JSON?
- remram 2y ago
  JSON no problem, sometimes-JSON no. It would drop the not-JSON message entirely.
valyala 2y ago
Elasticsearch can store arbitrary text in log fields, including JSON-encoded string. Elasticsearch can also tokenize JSON-encoded string and provide fast full-text search over such string in the same way like it does for a regular plaintext string. why do you need storing JSON-encoded string inside log field? It is much better parsing the JSON into separate fields at log shipper and storing the parsed log fields into Elasticsearch. This gives better query performance and may also reduce disk space usage, since values for every parsed field are stored separately (this usually improves compression ratio and reduces disk read IO during queries if column-oriented storage is used for per-field data). I tried explaining this at https://itnext.io/why-victorialogs-is-a-better-alternative-to-grafana-loki-7e941567c4d5 https://itnext.io/why-victorialogs-is-a-better-alternative-t...
- remram 2y ago
  The problem is that log messages are collected from many researcher-authored applications. Some of them are JSON, in which case I want to parse them and store the parsed JSON as you say. But some of them are not JSON, in which case I need to store the text. I was not able to do that with the log shipper. If I configured parsing, then the not-JSON messages would get dropped.
2y ago

parliament32 2y ago

Yes, we switched metrics and logs from an Elastic stack to Prometheus/Thanos/Loki/Grafana about two years ago. On the logs side specifically, resource usage is WAY lower (300eps is like 1.5 cores and 4gb of memory), not to mention going from persistent volumes (disks) to blob storage / S3 is far cheaper and doesn't require any maintenance. Queries are slower, however, because Elastic pre-indexes while Loki searches on-demand, so it really comes down to query volume and your need for query performance (does it matter if your search takes 300ms vs 3s?). I've also found running Elastic yourself requires constant maintenance, while Loki has been very hands-off. Strongly recommend.