3 ms·

Grafana stack is cool. It solved the problem around observability for the first time in a neat and light way. Think about it however, for someone setting up the

Grafana stack is cool. It solved the problem around observability for the first time in a neat and light way. Think about it however, for someone setting up their observability stack they will need to setup 4 components: Grafana (dashboarding), Loki (logs), Mimir (Metrics) and Tempo (Traces). This is a lot of work and a lot of moving parts. It is also a lot of work to maintain and upgrade. Loki, Mimir and Tempo have their own querying languages and their own quirks. Good amount of stuff to learn. It is really hard for a small team to learn and maintain all of this. Junior engineers really struggle with this and seniors resent the time they have to spend on this. Loki also has issues around high cardinality, which is a problem for many teams. We need something new and better. We made sure to build a solution that is easy to setup and maintain. One single binary/container for single node setup and one stack to setup/upgrade/maintain for HA setup. We also made sure that it is easy to use and learn. We allow standard SQL to be used for querying logs, metrics and traces - nothing extra to learn here. Metrics querying is supported using PromQL too. Drag and drop for creating panels and dashboards is there too :-). Dashboarding is supported within the same stack. No need to setup Grafana or something else separately. OpenObserve also offer functions that grafana stack does not offer. Give it a shot, you are going to love it. Data is stored in open parquet format, allowing its usage even outside of OpenObserve if so desired. Hope this helps.

vlovich123 3y ago
One of the most annoying things of grafana is that panels can only be edited via the UI and the “changes” summary when you commit is near unusable with a bunch of auto generated garbage. Having a simple language/YAML to define those panels and an easy way to preview them (eg paste in your file to preview the page, paste in a panel source to try it) in addition to editing via GUI and being able to copy changes back to revision control would be great
- prabhatsharma 3y ago
  This is a good feedback. That would be true DaC (Dashboards as code). We already have plans to implement versioning of dashboards via git. Will see how to get DaC in place.
- smw 3y ago
  https://grafana.github.io/grafonnet-lib/ https://grafana.github.io/grafonnet-lib/
PlutoIsAPlanet 3y ago
Grafana log handling also isn't the best, especially when compared to New Relic etc.
- Too 3y ago
  Loki itself is rock solid. The Grafana integration isn’t the best yeah, it doesn’t even have tabular output. What’s the whole point of doing structural logging if you can’t do that. Many similar low hanging fruit are missing that could easily make it so much more useful. Now every service need its own bespoke query, slowing down exploration.
  - ChocolateGod 3y ago
    Exactly. They could take a tip or two from Greylog
Too 3y ago
How does OpenObserve handle logs with high cardinality? In Elasticsearch tuning these indices is one of the bigger operational burdens. Loki solves this by asking users to not put data in the indexed fields, instead brute force through it all. Does OpenObserve have any other approach to this?
- prabhatsharma 3y ago
  Loki solves the problem of high cardinality by saying - "don't have high cardinality". OpenObserve use parquet as the storage format which is much more resilient to high cardinality. OpenObserve has page indexes and partitions that allow it to handle this very well.
- valyala 3y ago
  Let's determine what is the "high cardinality" logs. Grafana Loki puts log messages with the same set of labels into a single "stream". For example, all the log messages with the labels {job="nginx", instance="host123"} go into a single stream. Loki stores every stream in a separate set of files and maintains in-memory index per each stream. So big number of streams in Loki lead to "high cardinality" issues such as performance degradation, high memory usage and increased storage load. That's why Loki developers recommend avoiding labels with big number of unique values such as ip, user_id, trace_id, etc. I'm unsure what does "high cardinality" mean for ElasticSearch case. As I know it happily accepts log messages with high-cardinality labels such as ip, user_id or trace_id. I don't know whether OpenObserve has any issues related to high cardinality. P.S. I'm, as VictoriaMetrics core developer, confident that the upcoming log storage and analytics solution from VictoriaMetrics - VictoriaLogs - will be free from Loki-like high cardinality issues.

vlovich123 3y ago

One of the most annoying things of grafana is that panels can only be edited via the UI and the “changes” summary when you commit is near unusable with a bunch of auto generated garbage. Having a simple language/YAML to define those panels and an easy way to preview them (eg paste in your file to preview the page, paste in a panel source to try it) in addition to editing via GUI and being able to copy changes back to revision control would be great

prabhatsharma 3y ago
This is a good feedback. That would be true DaC (Dashboards as code). We already have plans to implement versioning of dashboards via git. Will see how to get DaC in place.
smw 3y ago
https://grafana.github.io/grafonnet-lib/ https://grafana.github.io/grafonnet-lib/

PlutoIsAPlanet 3y ago

Grafana log handling also isn't the best, especially when compared to New Relic etc.

Too 3y ago
Loki itself is rock solid. The Grafana integration isn’t the best yeah, it doesn’t even have tabular output. What’s the whole point of doing structural logging if you can’t do that. Many similar low hanging fruit are missing that could easily make it so much more useful. Now every service need its own bespoke query, slowing down exploration.
- ChocolateGod 3y ago
  Exactly. They could take a tip or two from Greylog

Too 3y ago

How does OpenObserve handle logs with high cardinality? In Elasticsearch tuning these indices is one of the bigger operational burdens. Loki solves this by asking users to not put data in the indexed fields, instead brute force through it all. Does OpenObserve have any other approach to this?

prabhatsharma 3y ago
Loki solves the problem of high cardinality by saying - "don't have high cardinality". OpenObserve use parquet as the storage format which is much more resilient to high cardinality. OpenObserve has page indexes and partitions that allow it to handle this very well.
valyala 3y ago
Let's determine what is the "high cardinality" logs. Grafana Loki puts log messages with the same set of labels into a single "stream". For example, all the log messages with the labels {job="nginx", instance="host123"} go into a single stream. Loki stores every stream in a separate set of files and maintains in-memory index per each stream. So big number of streams in Loki lead to "high cardinality" issues such as performance degradation, high memory usage and increased storage load. That's why Loki developers recommend avoiding labels with big number of unique values such as ip, user_id, trace_id, etc. I'm unsure what does "high cardinality" mean for ElasticSearch case. As I know it happily accepts log messages with high-cardinality labels such as ip, user_id or trace_id. I don't know whether OpenObserve has any issues related to high cardinality. P.S. I'm, as VictoriaMetrics core developer, confident that the upcoming log storage and analytics solution from VictoriaMetrics - VictoriaLogs - will be free from Loki-like high cardinality issues.