7 ms·

I can't recommend Grafana anymore

245 points by gpi 11mo ago

eduction 11mo ago
I find the color of the text so light as to be unreadable.
- StackTopherFlow 11mo ago
  It’s challenging to read for me too.
- pseidemann 11mo ago
  The font doesn't render correctly on my device. It seems like as if some strokes are double, making lines inconsistent.
- incorrect-horse 11mo ago
  the "dark" theme having a *blaring* white banner at the top is an ... interesting design choice
BarryMilo 11mo ago
This is pretty interesting to me, as I do use Grafana in my current role. But none of their other products, and not their helm chart (we're on the Bitnami chart if that's a thing). So far it's pretty good. We're at least one major version behind, but hey everything still works. I cannot imagine other products support as many data sources (though I'm starting to think they all suck, I just dump what I can in InfluxDB).
- pahae 11mo ago
  I agree. I think OP has made the mistake of using more than just Grafana for dashboards and perhaps user queries. I operate a fairly large custom VictoriaMetrics-based Observability platform and have learned early on to only use Grafana as opposed to other Grafana products. Part of the stack used to use Mimir's frontend as caching layer but even that died with Mimir v3.0, now that it can't talk to generic Prometheus APIs anymore (vanilla Prom, VictoriaMetrics, promxy etc.). I went back to Cortex for caching. Such a custom stack is obviously not for everyone and takes much more time, knowledge and effort to deploy than some helm chart but overall I'd say it did save me some headache. At least when compared to the Google-like deprecation culture Grafana seems to have.
YZF 11mo ago
Not sure what's an alternative for Grafana in the open source world in terms of building dashboards for o11y? I'm not aware of one and Grafana is used very extensively in my company...
- throwaway173738 11mo ago
  We’re using Greylog+Elastic Search which would totally replace a Loki-only stack.
- edoceo 11mo ago
  I remember that alternative, free/FOSS products existed before Grafana (c2015) but many died, Grafana was everywhere. Now I also cannot find the old-alts. Vague memories of RRD and Nagios...
  - kbenson 11mo ago
    Munin was what we used for a while, along with a smattering of smokeping. We're using a combination of Zabbix (alerting) and local Grafana/Prometheus/Loki (observability) at this point, but I've been worried about when Grafana will rug-pull for a while now. Hopefully enough people using their cloud offering sates their appetite and they leave the people running locally alone.
  - linker3000 11mo ago
    I ran with Centreon for a while because you got Nagios + integrated dashboarding out of the box and a Community option. I'm out of that game now though so don't have the challenge. https://www.centreon.com/ https://www.centreon.com/
- o11c 11mo ago
  ... ugh, they actually made an `o11[a-z]` abbreviation? When I picked this nick, the only term I ever saw in the wild was `i18n`.
  - lelandfe 11mo ago
    K8s (Kubernetes), a11y (accessibility)... The kicker for me recently was hearing someone say "ally"
nacozarina 11mo ago
Sounds like grafana needed to fork
datadrivenangel 11mo ago
Building an elaborate pile of technical debt is a great way to have an elaborate pile of technical debt, but the lifespan of services being 2-3 years gets painful as you start composing a stack out of enough products that every quarter you need to replace something big.
- valyala 11mo ago
  There are products, which fight against the software bloat with bells and whistles, and against breaking backwards compatibility with every new release. They put user experience and stability as the top priority - https://docs.victoriametrics.com/victoriametrics/goals/ https://docs.victoriametrics.com/victoriametrics/goals/
  - slekker 11mo ago
    Please disclose you're the founder, you've been advertising your company throughout this thread
    - valyala 11mo ago
      I'm the core developer at VictoriaMetrics - you can find this information by clicking my account name here. Now it's your turn to disclose your employer name.
      - slekker 11mo ago
        I'm a freelancer. And I do use VictoriaMetrics + Grafana on projects for clients, have done so for years now. It what put bread on the table. I like your product, it's standard etiquette in the community to disclose these affiliations, I recognize your name :)
stym06 11mo ago
off topic, but prometheus pushgateway is such a bad implementation (once you push the metrics, it always stays there until it's restarted, like counter does not increase, it just pushes a new metric with the new value) that we had to write our own metrics collector endpoint.
- mickeyp 11mo ago
  That is literally how it is supposed to work. Prometheus grabs metrics --- that is how it works. If you for some reason find yourself unable to host an endpoint with metrics, you can use the fallback pushgateway to push metrics where yes they will stay until restarted. Ask yourself how it could ever work if they are subsequently deleted after read. How would multiple prometheus agents be able to read from the same source?
- deleted 11mo ago
  [deleted]
- jordanb 11mo ago
  The pushgateway is itself a horrible hack for the fact that prometheus is designed only for metrics scraping. Unfortunately the whole ecosystem around it is an utter mess.
  - pahae 11mo ago
    Remote Write is a viable alternative in Prometheus and its drop-in replacements. I'm not a massive fan of it myself as I feel the pull-based approach is superior overall but still make heavy use of it. The pushgateway's documentation itself calls out that there are only very limited cirumstances where it makes sense. I personally only used it in $old_job and only for batch jobs that could not use the node_exporter's textfile collector. I would not use it again and would even advise against it.
  - valyala 11mo ago
    You can use other monitoring systems, which support both pull-based and push-based models for metrics' collection. See, for example, https://docs.victoriametrics.com/victoriametrics/#how-to-import-time-series-data https://docs.victoriametrics.com/victoriametrics/#how-to-imp...
stym06 11mo ago
> "But I got it all working; now I can finally stop explaining to my boss why we need to re-structure the monitoring stack every year." Prometheus and Grafana have been progressing in their own ways and each of them is trying to have a fullstack solution and then the OTEL thingy came and ruined the party for everyone
- jamesblonde 11mo ago
  I still haven't got my head around how OTEL fits into a good open-source monitoring stack. Afaik, it is a protocol for metrics, traces, and logs. And we want our open-source monitoring services/dbs to support it, so they become pluggable. But, afaik, there's no one good DB for logs and metrics, so most of us use Prometheus for metrics and OpenSearch for logs. Does OTEL mean we just need to replace all our collectors (like logstash for logs and all the native metrics collectors and pushgateway crap) and then reconfigure Prometheus and OpenSearch?
  - rsanheim 11mo ago
    I think the answer is it doesn't fit in any definition of a _good_ monitoring stack, but we are stuck with it. It has largely become the blessed protocol, specification, and standard for OSS monitoring, along every axis (logging, tracing, collecting, instrumentation, etc)...its a bit like the efforts that resulted in J2EE and EJBs back in the day, only more diffuse and with more varied implementations. And we don't really have a simpler alternative in sight...at least in the java days there was the disgust and reaction via struts, spring, EJB3+, and of course other languages and communities. Not sure how we exactly we got into such an over-engineered mono-culture in terms of operations and monitoring and deployment for 80%+ of the industry (k8s + graf/loki/tempo + endless supporting tools or flavors), but it is really a sad state. Then you have endless implementations handling bits and pieces of various parts of the spec, and of course you have the tools to actually ingest and analyze and report on them.
  - pas 11mo ago
    logs, spans and metrics are stored as time-stamped stuff. sure simple fixed-width columnar storage is faster, and makes sense to special case for numbers (add downsampling and aggregations, and histogram maintenance and whatnot), but any write-optimized storage engine can handle this, it's not the hard part (basically LevelDB, and if there's need for scaling out it'll look like Cassandra, Aerospike, ScyllaDB, or ClickHouse ... see also https://docs.greptime.com/user-guide/concepts/data-model/ https://docs.greptime.com/user-guide/concepts/data-model/ and specialized storage engines https://docs.greptime.com/reference/about-greptimedb-engines/ https://docs.greptime.com/reference/about-greptimedb-engines... )
gtrealejandro 11mo ago
As someone who runs SaaS products, this post resonates painfully well. The author is 100% correct: Monitoring should be the most boring tool in the stack. Its one and only job is to be more reliable than the thing it's monitoring. The moment your monitoring stack requires a complex dependency like Kafka, or changes its entire agent flow every 18 months, it has failed its primary purpose. It has become the problem. This sounds less like a technical evolution and more like the classic VC-funded push to get everyone onto a high-margin cloud product, even at the cost of the open-source soul.
- valyala 11mo ago
  Boring monitoring software is rare, since it is hard to refuse adding yet another shiny feature (and breaking the core functionality along the way) instead of focusing on improving the usability for the core functionality. We at VictoriaMetrics try making boring monitoring solution which just works out of the box - https://docs.victoriametrics.com/victoriametrics/goals/ https://docs.victoriametrics.com/victoriametrics/goals/
seymon 11mo ago
But are there good alternatives to grafana in the foss space nowadays?
- pahae 11mo ago
  I only know of https://perses.dev/ https://perses.dev/ but haven't had a look at it for ~half a year. It was very barebones back then but I'm hopeful it can replace Grafana for at least basic dashboarding soon.
zer00eyz 11mo ago
FTA > "I know for a fact that that pace is partially driven by career-driven development." This isn't a Grafana problem, this is an industry wide problem. Resume driven product design, resume driven engineering, resume driven marketing. DO your 2-3 years, pump out something big to inflate your resume. Apply elsewhere to get the pay bump that almost no company is handing out. After the departures there is no one left who knows the system and the next people in want to replace the things they don't understand to pad their resume for the next job. Wash, rinse, repete. Loyalty, simply goes unrewarded in a lot of places in our industry (and at a many corporations). And the people who do stay... in many cases they turn into furniture that ends up holding potential good evolution back. They loose out to the technological magpies the bring shiny things to management because it will "move the needle". Sadly this is just one facet of the problems we are facing, from how we interview to how we run (or rent) our infrastructure things have gotten rather silly...
- mingus88 11mo ago
  without any stability, you really can’t blame the player for playing this game. The days where you could devote your career to a firm and retire with a pension are long gone The author of this article wants a boring tech stack that just works, and honestly after everything we’ve been through in the last five years, I kinda want a boring job I can keep until I retire, too
Szpadel 11mo ago
what are tested and fairly lightweight alternatives for Loki? elastic stack is so heavy it's out of question for smaller clusters, loki integration with grafana is nice to have but separate capable dashboard would be also fine
- valyala 11mo ago
  VictoriaLogs - https://www.truefoundry.com/blog/victorialogs-vs-loki https://www.truefoundry.com/blog/victorialogs-vs-loki
- prabhatsharma 11mo ago
  OpenObserve would be the simplest and most performant
morganherlocker 11mo ago
I frequently use a docker-compose template with prometheus pushgateway + grafana for deploying on single node servers, as described at the start of the article. It works well and is trivial to setup, but the complexity explodes once your metric volume or cardinality requires more scale like prometheus alternatives a la mimir. I think this would not need to be an issue as frequently if prometheus had a more efficient publish/scraping mechanism. iirc there was once a protobuf metric format that was dropped, and now there is just the text format. While it wouldn't handle billions of unique labels like mimir, a compact binary metric format could certainly allow for millions at reasonable resolution instead of wasting all that scale potential on repeated name strings. I should be able to push or expose a bulk blob all at once with ordered labels or at least raw int keys.
tamnd 11mo ago
Signoz is good, and active development https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz
- dabinat 11mo ago
  I second SigNoz. I was paying a fortune for a cloud observability platform that cost more and more every month. Then I switched to self-hosted SigNoz on a cheap Hetzner box and now my observability stack costs $10 a month.
mgedde 11mo ago
What's the most promising alternative to Prometheus/Grafana if you're developing a new solution around OTEL? If you could start today and pick tools, what would you go for?
- sohooo 11mo ago
  We also started with the typical kube-prometheus-stack, but we don’t like Prometheus/PromQL. Moreover, it only solves the „metrics“ part - to handle logs and traces, more quite heavy and complex components have to be added to the observability stack. This didn‘t feel right, so we looked around and found greptimedb https://github.com/GreptimeTeam/greptimedb https://github.com/GreptimeTeam/greptimedb, which simplifies the whole stack. It‘s designed to handle metrics, logs, and traces. We collect metrics and logs via OpenTelemetry, and visualize them with Grafana. It provides endpoints for Postgres, MySQL, PromQL; we‘re happy to be able to build dashboards using SQL as that’s where we have the most knowledge. The benchmarks look promising, but our k8s clusters aren’t huge anyway. As a platform engineer, we appreciate the simplicity of our observability stack. Any other happy greptimedb users around here? Together with OTel, we think we can handle all future obs needs.
  - killme2008 11mo ago
    Hi there, I’m from the GreptimeDB team. Thank you for giving GreptimeDB a shout-out—it means a lot to us. We created GreptimeDB to simplify the observability data stack with an all-in-one database, and we’re glad to hear it’s been helpful. OpenTelemetry-native is a requirement, not an option, for the new observability data stack. I believe otel-arrow (https://github.com/open-telemetry/otel-arrow https://github.com/open-telemetry/otel-arrow) has strong future potential, and we are committed to supporting and improving it. FYI: I think SQL is great for building everything—dashboards, alerting rules, and complex analytics—but PromQL still has unique value in the Prometheus ecosystem. To be transparent, GreptimeDB still has some performance issues with PromQL, which we’ll address before the 1.0 GA.
    - bminor13 11mo ago
      > I think SQL is great for building everything Are you saying that you prefer SQL over PromQL for metrics queries? I haven't tried querying metrics via SQL yet, but generally speaking have found PromQL to be one of the easier query languages to learn - more straightforward and concise IME. What advantages does SQL offer here?
solatic 11mo ago
Mimir is just architected for a totally different order of magnitude of metrics. At that scale, yeah, Kafka is actually necessary. There are no other open-source solutions offering the same scalability, period. That's besides the point that most customers will never need that level of scale. If you're not running Mimir on a dedicated Kubernetes cluster (or at least a dedicated-to-Grafana / observability cluster) then it's probably over-engineered for your use-case. Just use Prometheus.
- vulkoingim 11mo ago
  Have a look at Victoria Metrics - have run it at a relatively high scale with much more success than any other metric stores. It's one of those things that just work. It's extremely easy to run at in a single-instance mode and handles much more than you would expect. Scaling it is a breeze too. (I'm not affiliated, but a very happy user across multiple orgs and personal projects)
  - solatic 11mo ago
    The project where I looked at Mimir was a 500+ million timeseries project, with the desire to support scaling to the ten-figure level of timeseries (working for a BigCo supporting hundreds of product development teams). All of these systems that store metrics in object storage - you have to remember that object storage is not file storage. Generally speaking (stuff like S3 One Zone being a relatively recent exception) you cannot append to object files. Metrics queries are resolved by querying historical metrics in object storage plus a stateful service hosting the latest 2 hours of data before it can be compressed and uploaded to object storage as a single block. At a certain scale, you simply need to choose which is more important - being able to answer queries or being able to insert more timeseries. And if you don't prioritize insertion, it just results in the backlog getting bigger and bigger, which especially in the eventual case (Murphy's Law guarantees it) of a sudden flood of metrics to ingest will cause several hour ingestion delays during which you are blind. And if you do prioritize insertion, well the component simply won't respond to queries, which makes you blind anyway. Lose-lose. Mimir built in Kafka because it's quite literally necessary at scale. You need the stateful query component (with the latest 2 hours) to prioritize queries, then pull from the Kafka topic on a lower priority thread, when there's spare time to do so. Kafka soaks up the sudden ingestion floods so that they don't result in the stateful query component getting DoS'd. I took a quick look at VictoriaMetrics - no Kafka or Kafka-like component to soak up ingestion floods? DOA. Again, most companies are not BigCos. If you're a startup/scaleup with one VP supervising several development teams, you likely don't need that scale, probably VictoriaMetrics is just fine, you're not the first person I've heard recommend it. But I would say 80% of companies are small enough to be served with a simple Prometheus or Thanos Query over HA Prometheus setup, 17% of companies will get a lot of value out of Victoria Metrics, the last 3% really need Mimir's scalability.
didierbreedt 11mo ago
I have found Grafana to be a decent product, but Prom needs a better horizontally scalable solution. We use Vector and Clickhouse for logging and works really well.
- hagen1778 11mo ago
  There are plenty of ways to scale Prometheus: - Thanos - Mimir - VictoriaMetrics All of them provide a way to scale monitoring to insane numbers. The difference is in architecture, maintainability and performance. But make your own choices here. Before, I remember there was m3db from Uber. But the project seems pretty dead now. And there was a Cortex project, mostly maintaned by GrafanaLabs. But at some point they forked Cortex and named it Mimir. And Cortex is now maintained by Amazon and, as I undersand, is powering Amazon Managed Prometheus. However, I would avoid using Cortex ecaxctly because it is now maintained by Amazon.
arandomhuman 11mo ago
This article comes off sort of low effort and mentions a lot grievances without actual pinpointing precise issues. I think leveraging OTEL as a general processor with a generic output is a good idea, but discounting Grafana for implementing multi tenancy solutions and alloy which is pretty fucking good is kind of pointless.
8n4vidtmkvmk 11mo ago
This reads like a satire. There's so much jargon and so many products involved to just do a little bit of logging. It's ridiculous. That is to say I agree with the author.
logifail 11mo ago
> I want stability for my monitoring; I want it boring, and that’s something Grafana is not offering I used to be a fan of InfluxDB (back in the days of v1.x) then I went off it for exactly this reason.
bboreham 11mo ago
> Mimir in version 3.0 needs Apache Kafka to work. I’d like to adjust this understanding. Kafka is the big new thing, but it’s optional. The previous way using gRPC still works. I work on Mimir and other things at Grafana Labs.
- kodama-lens 11mo ago
  Author here. I know the old way still works and I respect that. Given the history I ask myself how long will it work, since ist not the default anymore.
- rickette 11mo ago
  Well the docs say the old way (no Kafka) is on its way out: "However, this architecture is set to be deprecated in a future release." So it doesn't stay optional unfortunately. It quite a heavy dependency to include...
  - skrtskrt 11mo ago
    Well it only needs to be Kafka-compatible, so personally I hope that things like RedPanda will turn out to be easier to run for this use case than actual Kafka. Having an S3-compatible store was already a fairly heavy dependency in terms of something to run correctly in production, it's just that most people don't even consider running their own object store at any real scale, they just go to cloud. Whereas running your own Kafka is something more platform teams are already attempting.
prymitive 11mo ago
IMHO in a way the constant churn of Grafana made it easier to live with it. When they break your dashboards every major release you just learn to let go. And by break it’s not just making it error and not work, it’s the constant moving of things around, refactoring the UI, replacing one component with another, all accompanied by a number of glitches every time they rewrite things. You just accept and ignore it eventually. What’s a bigger lock in for me is metrics and promql - you just can’t ever rename a poorly named metric or you face a world of pain. Or when Prometheus releases Native Histograms to replace the old ones, and suddenly everything from rules, alerts, ad-hoc queries and dashboards needs updating. And PromQL is so opaque, it just never give you an error unless there is a syntax issue. We need tools like https://github.com/cloudflare/pint https://github.com/cloudflare/pint just to know if my alert description isn’t trying to render a label that’s just not gonna be there, etc
- znpy 11mo ago
  > And PromQL is so opaque, it just never give you an error unless there is a syntax issue. We need tools like https://github.com/cloudflare/pint https://github.com/cloudflare/pint just to know if my alert description isn’t trying to render a label that’s just not gonna be there, etc PromQL should be blamed on prometheus though, not on grafana.
esseph 11mo ago
I feel this article in my bones. It's rough out there.
Too 11mo ago
It’s as though they deliberately make it complex and constantly moving to wear you out to give in to the convenient SaaS offer.
yomismoaqui 11mo ago
If your observability stack works and you are fine with it, do you need to update it? I understand updating some front facing service due to a vulnerability... But for a thing that it's internally accessible?
plqbfbv 11mo ago
I'm also seriously considering dropping Grafana for good for the same reasons stated in the post. Every year I need to rebuild a dashboard, reconfigure alerts, use the shiny new toy, etc etc. I'm tired. I just want the thing to alert me when something's down, and ideally if the check doesn't change and the datasource and metric don't change, the dashboard definition and the alert definition should be the same for the last and the next 10 years. The UI used to have the most 4-5 important links in the sidebar, now it's 10 menus with submenus of submenus, and I never know where to find the basics: Dashboards and Alerts. When something goes off I don't have time to re-learn the UI I look at maybe once a month.
- valyala 11mo ago
  Take a look at Perses - https://perses.dev/ https://perses.dev/
atoav 11mo ago
This is what keeps me from using things like these. Meanwhile I have uinx-oid scripts that keep chugging on up to date servers for more than a decade without any need to change them. I know there is always the temptation to make it really shiny and nice. But the more moving parts your system has the liklier it becomes that something will fail eventually. And as it happens these failures happen usually at the time when it is most inconvenient for all people involved. That doesn't mean that complex software cannot work reliably, but it takes more effort for the developer side to honor that unwritten contract with their users (if they are even aware of it). This is why sometimes doing it yourself, on your own servers can be benefitial because it gives you more control.
paxys 11mo ago
I don't know why software developers feel the urge to stay on the bleeding edge of every product and update their setup every week then turn around and compain "this stuff isn't stable!" I've had a grafana + prometheus setup on my servers since like 2017. It worked then and works today. I log in maybe once every year or two to update to a newer LTS version. Every dashboard is still pristine, and nothing has ever broken. I don't understand most of the words in the linked post and don't need to. The core package is the boring solution that 99% of people here need, and that works great.
- hoherd 11mo ago
  > Every dashboard is still pristine How did you handle the angular deprecation in grafana? Or are you just staying in an older version that still supports it?
  - tekla 11mo ago
    I've done it and it wasn't too bad? The auto migrate worked pretty well for 99% of stuff (hundreds of dashboards). Though I won't say I loved doing it.
  - paxys 11mo ago
    All of my dashboards were auto updated so I didn't really have to do anything.
piterrro 11mo ago
Who remembers Graphite and Carbon? This was 2010 era…
mt42or 11mo ago
Victoriametrics and never change. Avoid grafana ingestion tools. Grafana for ui, alerting.
blueybingo 11mo ago
Clickhouse, with their clickstack stack, for logging and traces - it’s not there for metrics yet but great for the other two.
- blueybingo 11mo ago
  Also mit and apache if you’re not fond of agpl
- akvadrako 11mo ago
  Victoria metrics which has similar database design to clickhouse is good for metrics. But it doesn't have a complete dashboard UI like Grafana.
  - hagen1778 11mo ago
    Just to add, VictoriaMetrics covers all 3 signals: - VictoriaMetrics for metrics. With Prometheus API support, so it integrates with Grafana using Prometheus datasource. It has its own Grafana datasource with extra functionality too. - VictoriaLogs for logs. Integrates natively with Grafana using VictoriaLogs datasource. - VictoriaTraces for traces. With Jaeger API support, so it intergrates with Grafana using Jaeger datasource. All 3 solutions support alerting, managed by same team, are Apache2 licensed, are focused on resource efficiency and simiplicity.
bean469 11mo ago
> At some point we needed a monitoring solution, and Zabbix didn’t fit well into the new and declarative world of containers and Docker. I wonder why they think that is the case, as Zabbix has native support for monitoring VMs and Docker containers, including support for discovering newly spawned ones and all that jazz
buro9 11mo ago
A reply of sorts, and a standard disclaimer that I work at Grafana Labs. > career-driven development we don't have this and promote and reward as frequently for "I've done solid operations" as we do for "I've added this feature" (I'm on promotion committees and can state this confidently). what we do have is high autonomy for engineers. This autonomy means it's a freedom that engineers have to identify problems they feel are important and to work on them, they do not need permission and leadership do not veto this. Some of the best features in the last few years have been a direct result of this autonomy, it's one of the things that makes working here so attractive to many of the engineers. But, with autonomy comes a little chaos, and not everything that is done is going to satisfy every end user of OSS or paid customer (of which these are a small percent of the whole). a lot of the innovation speed is just in the DNA of the company, even the creation of Grafana can be traced to a desire to get things done; Torkel wanted Kibana to also work for Prometheus, Kibana declined to add this, Torkel didn't stand still and added things to a fork of Kibana now called Grafana and hasn't stopped adding things since. > They also deprecated Angular within Grafana and switched to React for dashboards. This broke most existing dashboards. we did, I think the entire journey was 7 years long, communicated many times, over at least 6 major releases. maintaining dashboards in two languages increased complexity, whilst reducing compatibility, and gave a very large security surface to be worried about. we communicated clearly, provided migration tools, put it in release notes, updated docs, repeated it at conferences and on community calls. arguably we went too slow, and should've ripped the band-aid off, but we were sensitive to the fact that it was a breaking change and so we proceeded with extreme caution. it's done now, it was finally completed in the last version, only a very small number of users reported impact as a result of the time and care taken on this. > I just hope OTEL settles, gets stable and boring fast this is distinct from Grafana, but it's a good point... OTel is the product of virtually every vendor at this point, and a hell of a lot of engineers, it now has a lot of momentum and the pace is unlikely to ease up due to the sheer number of contributions and things that OTel as a community wishes to achieve. the most likely eventuality is that enough stability emerges to allow vendors (including but not limited to Grafana Labs) to abstract away the pace of innovation occurring underneath, but this is in tension with providing the benefits of the innovation to the people that use it. what I would say is that for most people the boring and slow path does still exist, and it's still good... just use Prometheus, a logging option of your choice, and simple Grafana dashboards and alerts. that combination hasn't varied in years, and those on it today are still immune from caring about the pace of innovation and change in OTel and across the Observability industry. OTel is being used in production at massive scale by lots of companies, but whether your project or company need move to it now reflects your priorities, many are adopting to gain independence from vendors, or just control over their telemetry, but many customers are also saying they're happy to stay on the slow and boring path and for everything to work predictably with low cost to keep pace... it works too.

245 points by gpi 11mo ago

eduction 11mo ago

I find the color of the text so light as to be unreadable.

StackTopherFlow 11mo ago
It’s challenging to read for me too.
pseidemann 11mo ago
The font doesn't render correctly on my device. It seems like as if some strokes are double, making lines inconsistent.
incorrect-horse 11mo ago
the "dark" theme having a *blaring* white banner at the top is an ... interesting design choice

BarryMilo 11mo ago

This is pretty interesting to me, as I do use Grafana in my current role. But none of their other products, and not their helm chart (we're on the Bitnami chart if that's a thing). So far it's pretty good. We're at least one major version behind, but hey everything still works. I cannot imagine other products support as many data sources (though I'm starting to think they all suck, I just dump what I can in InfluxDB).

pahae 11mo ago
I agree. I think OP has made the mistake of using more than just Grafana for dashboards and perhaps user queries. I operate a fairly large custom VictoriaMetrics-based Observability platform and have learned early on to only use Grafana as opposed to other Grafana products. Part of the stack used to use Mimir's frontend as caching layer but even that died with Mimir v3.0, now that it can't talk to generic Prometheus APIs anymore (vanilla Prom, VictoriaMetrics, promxy etc.). I went back to Cortex for caching. Such a custom stack is obviously not for everyone and takes much more time, knowledge and effort to deploy than some helm chart but overall I'd say it did save me some headache. At least when compared to the Google-like deprecation culture Grafana seems to have.

YZF 11mo ago

Not sure what's an alternative for Grafana in the open source world in terms of building dashboards for o11y? I'm not aware of one and Grafana is used very extensively in my company...

throwaway173738 11mo ago
We’re using Greylog+Elastic Search which would totally replace a Loki-only stack.
edoceo 11mo ago
I remember that alternative, free/FOSS products existed before Grafana (c2015) but many died, Grafana was everywhere. Now I also cannot find the old-alts. Vague memories of RRD and Nagios...
- kbenson 11mo ago
  Munin was what we used for a while, along with a smattering of smokeping. We're using a combination of Zabbix (alerting) and local Grafana/Prometheus/Loki (observability) at this point, but I've been worried about when Grafana will rug-pull for a while now. Hopefully enough people using their cloud offering sates their appetite and they leave the people running locally alone.
- linker3000 11mo ago
  I ran with Centreon for a while because you got Nagios + integrated dashboarding out of the box and a Community option. I'm out of that game now though so don't have the challenge. https://www.centreon.com/ https://www.centreon.com/
o11c 11mo ago
... ugh, they actually made an `o11[a-z]` abbreviation? When I picked this nick, the only term I ever saw in the wild was `i18n`.
- lelandfe 11mo ago
  K8s (Kubernetes), a11y (accessibility)... The kicker for me recently was hearing someone say "ally"

nacozarina 11mo ago

Sounds like grafana needed to fork

datadrivenangel 11mo ago

Building an elaborate pile of technical debt is a great way to have an elaborate pile of technical debt, but the lifespan of services being 2-3 years gets painful as you start composing a stack out of enough products that every quarter you need to replace something big.

valyala 11mo ago
There are products, which fight against the software bloat with bells and whistles, and against breaking backwards compatibility with every new release. They put user experience and stability as the top priority - https://docs.victoriametrics.com/victoriametrics/goals/ https://docs.victoriametrics.com/victoriametrics/goals/
- slekker 11mo ago
  Please disclose you're the founder, you've been advertising your company throughout this thread
  - valyala 11mo ago
    I'm the core developer at VictoriaMetrics - you can find this information by clicking my account name here. Now it's your turn to disclose your employer name.
    - slekker 11mo ago
      I'm a freelancer. And I do use VictoriaMetrics + Grafana on projects for clients, have done so for years now. It what put bread on the table. I like your product, it's standard etiquette in the community to disclose these affiliations, I recognize your name :)

stym06 11mo ago

off topic, but prometheus pushgateway is such a bad implementation (once you push the metrics, it always stays there until it's restarted, like counter does not increase, it just pushes a new metric with the new value) that we had to write our own metrics collector endpoint.

mickeyp 11mo ago
That is literally how it is supposed to work. Prometheus grabs metrics --- that is how it works. If you for some reason find yourself unable to host an endpoint with metrics, you can use the fallback pushgateway to push metrics where yes they will stay until restarted. Ask yourself how it could ever work if they are subsequently deleted after read. How would multiple prometheus agents be able to read from the same source?
deleted 11mo ago
[deleted]
jordanb 11mo ago
The pushgateway is itself a horrible hack for the fact that prometheus is designed only for metrics scraping. Unfortunately the whole ecosystem around it is an utter mess.
- pahae 11mo ago
  Remote Write is a viable alternative in Prometheus and its drop-in replacements. I'm not a massive fan of it myself as I feel the pull-based approach is superior overall but still make heavy use of it. The pushgateway's documentation itself calls out that there are only very limited cirumstances where it makes sense. I personally only used it in $old_job and only for batch jobs that could not use the node_exporter's textfile collector. I would not use it again and would even advise against it.
- valyala 11mo ago
  You can use other monitoring systems, which support both pull-based and push-based models for metrics' collection. See, for example, https://docs.victoriametrics.com/victoriametrics/#how-to-import-time-series-data https://docs.victoriametrics.com/victoriametrics/#how-to-imp...

stym06 11mo ago

> "But I got it all working; now I can finally stop explaining to my boss why we need to re-structure the monitoring stack every year." Prometheus and Grafana have been progressing in their own ways and each of them is trying to have a fullstack solution and then the OTEL thingy came and ruined the party for everyone

jamesblonde 11mo ago
I still haven't got my head around how OTEL fits into a good open-source monitoring stack. Afaik, it is a protocol for metrics, traces, and logs. And we want our open-source monitoring services/dbs to support it, so they become pluggable. But, afaik, there's no one good DB for logs and metrics, so most of us use Prometheus for metrics and OpenSearch for logs. Does OTEL mean we just need to replace all our collectors (like logstash for logs and all the native metrics collectors and pushgateway crap) and then reconfigure Prometheus and OpenSearch?
- rsanheim 11mo ago
  I think the answer is it doesn't fit in any definition of a _good_ monitoring stack, but we are stuck with it. It has largely become the blessed protocol, specification, and standard for OSS monitoring, along every axis (logging, tracing, collecting, instrumentation, etc)...its a bit like the efforts that resulted in J2EE and EJBs back in the day, only more diffuse and with more varied implementations. And we don't really have a simpler alternative in sight...at least in the java days there was the disgust and reaction via struts, spring, EJB3+, and of course other languages and communities. Not sure how we exactly we got into such an over-engineered mono-culture in terms of operations and monitoring and deployment for 80%+ of the industry (k8s + graf/loki/tempo + endless supporting tools or flavors), but it is really a sad state. Then you have endless implementations handling bits and pieces of various parts of the spec, and of course you have the tools to actually ingest and analyze and report on them.
- pas 11mo ago
  logs, spans and metrics are stored as time-stamped stuff. sure simple fixed-width columnar storage is faster, and makes sense to special case for numbers (add downsampling and aggregations, and histogram maintenance and whatnot), but any write-optimized storage engine can handle this, it's not the hard part (basically LevelDB, and if there's need for scaling out it'll look like Cassandra, Aerospike, ScyllaDB, or ClickHouse ... see also https://docs.greptime.com/user-guide/concepts/data-model/ https://docs.greptime.com/user-guide/concepts/data-model/ and specialized storage engines https://docs.greptime.com/reference/about-greptimedb-engines/ https://docs.greptime.com/reference/about-greptimedb-engines... )

gtrealejandro 11mo ago

As someone who runs SaaS products, this post resonates painfully well. The author is 100% correct: Monitoring should be the most boring tool in the stack. Its one and only job is to be more reliable than the thing it's monitoring. The moment your monitoring stack requires a complex dependency like Kafka, or changes its entire agent flow every 18 months, it has failed its primary purpose. It has become the problem. This sounds less like a technical evolution and more like the classic VC-funded push to get everyone onto a high-margin cloud product, even at the cost of the open-source soul.

valyala 11mo ago
Boring monitoring software is rare, since it is hard to refuse adding yet another shiny feature (and breaking the core functionality along the way) instead of focusing on improving the usability for the core functionality. We at VictoriaMetrics try making boring monitoring solution which just works out of the box - https://docs.victoriametrics.com/victoriametrics/goals/ https://docs.victoriametrics.com/victoriametrics/goals/

seymon 11mo ago

But are there good alternatives to grafana in the foss space nowadays?

pahae 11mo ago
I only know of https://perses.dev/ https://perses.dev/ but haven't had a look at it for ~half a year. It was very barebones back then but I'm hopeful it can replace Grafana for at least basic dashboarding soon.

zer00eyz 11mo ago

FTA > "I know for a fact that that pace is partially driven by career-driven development." This isn't a Grafana problem, this is an industry wide problem. Resume driven product design, resume driven engineering, resume driven marketing. DO your 2-3 years, pump out something big to inflate your resume. Apply elsewhere to get the pay bump that almost no company is handing out. After the departures there is no one left who knows the system and the next people in want to replace the things they don't understand to pad their resume for the next job. Wash, rinse, repete. Loyalty, simply goes unrewarded in a lot of places in our industry (and at a many corporations). And the people who do stay... in many cases they turn into furniture that ends up holding potential good evolution back. They loose out to the technological magpies the bring shiny things to management because it will "move the needle". Sadly this is just one facet of the problems we are facing, from how we interview to how we run (or rent) our infrastructure things have gotten rather silly...

mingus88 11mo ago
without any stability, you really can’t blame the player for playing this game. The days where you could devote your career to a firm and retire with a pension are long gone The author of this article wants a boring tech stack that just works, and honestly after everything we’ve been through in the last five years, I kinda want a boring job I can keep until I retire, too

Szpadel 11mo ago

what are tested and fairly lightweight alternatives for Loki? elastic stack is so heavy it's out of question for smaller clusters, loki integration with grafana is nice to have but separate capable dashboard would be also fine

valyala 11mo ago
VictoriaLogs - https://www.truefoundry.com/blog/victorialogs-vs-loki https://www.truefoundry.com/blog/victorialogs-vs-loki
prabhatsharma 11mo ago
OpenObserve would be the simplest and most performant

morganherlocker 11mo ago

I frequently use a docker-compose template with prometheus pushgateway + grafana for deploying on single node servers, as described at the start of the article. It works well and is trivial to setup, but the complexity explodes once your metric volume or cardinality requires more scale like prometheus alternatives a la mimir. I think this would not need to be an issue as frequently if prometheus had a more efficient publish/scraping mechanism. iirc there was once a protobuf metric format that was dropped, and now there is just the text format. While it wouldn't handle billions of unique labels like mimir, a compact binary metric format could certainly allow for millions at reasonable resolution instead of wasting all that scale potential on repeated name strings. I should be able to push or expose a bulk blob all at once with ordered labels or at least raw int keys.

tamnd 11mo ago

Signoz is good, and active development https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz

dabinat 11mo ago
I second SigNoz. I was paying a fortune for a cloud observability platform that cost more and more every month. Then I switched to self-hosted SigNoz on a cheap Hetzner box and now my observability stack costs $10 a month.

mgedde 11mo ago

What's the most promising alternative to Prometheus/Grafana if you're developing a new solution around OTEL? If you could start today and pick tools, what would you go for?

sohooo 11mo ago
We also started with the typical kube-prometheus-stack, but we don’t like Prometheus/PromQL. Moreover, it only solves the „metrics“ part - to handle logs and traces, more quite heavy and complex components have to be added to the observability stack. This didn‘t feel right, so we looked around and found greptimedb https://github.com/GreptimeTeam/greptimedb https://github.com/GreptimeTeam/greptimedb, which simplifies the whole stack. It‘s designed to handle metrics, logs, and traces. We collect metrics and logs via OpenTelemetry, and visualize them with Grafana. It provides endpoints for Postgres, MySQL, PromQL; we‘re happy to be able to build dashboards using SQL as that’s where we have the most knowledge. The benchmarks look promising, but our k8s clusters aren’t huge anyway. As a platform engineer, we appreciate the simplicity of our observability stack. Any other happy greptimedb users around here? Together with OTel, we think we can handle all future obs needs.
- killme2008 11mo ago
  Hi there, I’m from the GreptimeDB team. Thank you for giving GreptimeDB a shout-out—it means a lot to us. We created GreptimeDB to simplify the observability data stack with an all-in-one database, and we’re glad to hear it’s been helpful. OpenTelemetry-native is a requirement, not an option, for the new observability data stack. I believe otel-arrow (https://github.com/open-telemetry/otel-arrow https://github.com/open-telemetry/otel-arrow) has strong future potential, and we are committed to supporting and improving it. FYI: I think SQL is great for building everything—dashboards, alerting rules, and complex analytics—but PromQL still has unique value in the Prometheus ecosystem. To be transparent, GreptimeDB still has some performance issues with PromQL, which we’ll address before the 1.0 GA.
  - bminor13 11mo ago
    > I think SQL is great for building everything Are you saying that you prefer SQL over PromQL for metrics queries? I haven't tried querying metrics via SQL yet, but generally speaking have found PromQL to be one of the easier query languages to learn - more straightforward and concise IME. What advantages does SQL offer here?

solatic 11mo ago

Mimir is just architected for a totally different order of magnitude of metrics. At that scale, yeah, Kafka is actually necessary. There are no other open-source solutions offering the same scalability, period. That's besides the point that most customers will never need that level of scale. If you're not running Mimir on a dedicated Kubernetes cluster (or at least a dedicated-to-Grafana / observability cluster) then it's probably over-engineered for your use-case. Just use Prometheus.

vulkoingim 11mo ago
Have a look at Victoria Metrics - have run it at a relatively high scale with much more success than any other metric stores. It's one of those things that just work. It's extremely easy to run at in a single-instance mode and handles much more than you would expect. Scaling it is a breeze too. (I'm not affiliated, but a very happy user across multiple orgs and personal projects)
- solatic 11mo ago
  The project where I looked at Mimir was a 500+ million timeseries project, with the desire to support scaling to the ten-figure level of timeseries (working for a BigCo supporting hundreds of product development teams). All of these systems that store metrics in object storage - you have to remember that object storage is not file storage. Generally speaking (stuff like S3 One Zone being a relatively recent exception) you cannot append to object files. Metrics queries are resolved by querying historical metrics in object storage plus a stateful service hosting the latest 2 hours of data before it can be compressed and uploaded to object storage as a single block. At a certain scale, you simply need to choose which is more important - being able to answer queries or being able to insert more timeseries. And if you don't prioritize insertion, it just results in the backlog getting bigger and bigger, which especially in the eventual case (Murphy's Law guarantees it) of a sudden flood of metrics to ingest will cause several hour ingestion delays during which you are blind. And if you do prioritize insertion, well the component simply won't respond to queries, which makes you blind anyway. Lose-lose. Mimir built in Kafka because it's quite literally necessary at scale. You need the stateful query component (with the latest 2 hours) to prioritize queries, then pull from the Kafka topic on a lower priority thread, when there's spare time to do so. Kafka soaks up the sudden ingestion floods so that they don't result in the stateful query component getting DoS'd. I took a quick look at VictoriaMetrics - no Kafka or Kafka-like component to soak up ingestion floods? DOA. Again, most companies are not BigCos. If you're a startup/scaleup with one VP supervising several development teams, you likely don't need that scale, probably VictoriaMetrics is just fine, you're not the first person I've heard recommend it. But I would say 80% of companies are small enough to be served with a simple Prometheus or Thanos Query over HA Prometheus setup, 17% of companies will get a lot of value out of Victoria Metrics, the last 3% really need Mimir's scalability.

didierbreedt 11mo ago

I have found Grafana to be a decent product, but Prom needs a better horizontally scalable solution. We use Vector and Clickhouse for logging and works really well.

hagen1778 11mo ago
There are plenty of ways to scale Prometheus: - Thanos - Mimir - VictoriaMetrics All of them provide a way to scale monitoring to insane numbers. The difference is in architecture, maintainability and performance. But make your own choices here. Before, I remember there was m3db from Uber. But the project seems pretty dead now. And there was a Cortex project, mostly maintaned by GrafanaLabs. But at some point they forked Cortex and named it Mimir. And Cortex is now maintained by Amazon and, as I undersand, is powering Amazon Managed Prometheus. However, I would avoid using Cortex ecaxctly because it is now maintained by Amazon.

arandomhuman 11mo ago

This article comes off sort of low effort and mentions a lot grievances without actual pinpointing precise issues. I think leveraging OTEL as a general processor with a generic output is a good idea, but discounting Grafana for implementing multi tenancy solutions and alloy which is pretty fucking good is kind of pointless.

8n4vidtmkvmk 11mo ago

This reads like a satire. There's so much jargon and so many products involved to just do a little bit of logging. It's ridiculous. That is to say I agree with the author.

logifail 11mo ago

> I want stability for my monitoring; I want it boring, and that’s something Grafana is not offering I used to be a fan of InfluxDB (back in the days of v1.x) then I went off it for exactly this reason.

bboreham 11mo ago

> Mimir in version 3.0 needs Apache Kafka to work. I’d like to adjust this understanding. Kafka is the big new thing, but it’s optional. The previous way using gRPC still works. I work on Mimir and other things at Grafana Labs.

kodama-lens 11mo ago
Author here. I know the old way still works and I respect that. Given the history I ask myself how long will it work, since ist not the default anymore.
rickette 11mo ago
Well the docs say the old way (no Kafka) is on its way out: "However, this architecture is set to be deprecated in a future release." So it doesn't stay optional unfortunately. It quite a heavy dependency to include...
- skrtskrt 11mo ago
  Well it only needs to be Kafka-compatible, so personally I hope that things like RedPanda will turn out to be easier to run for this use case than actual Kafka. Having an S3-compatible store was already a fairly heavy dependency in terms of something to run correctly in production, it's just that most people don't even consider running their own object store at any real scale, they just go to cloud. Whereas running your own Kafka is something more platform teams are already attempting.

prymitive 11mo ago

IMHO in a way the constant churn of Grafana made it easier to live with it. When they break your dashboards every major release you just learn to let go. And by break it’s not just making it error and not work, it’s the constant moving of things around, refactoring the UI, replacing one component with another, all accompanied by a number of glitches every time they rewrite things. You just accept and ignore it eventually. What’s a bigger lock in for me is metrics and promql - you just can’t ever rename a poorly named metric or you face a world of pain. Or when Prometheus releases Native Histograms to replace the old ones, and suddenly everything from rules, alerts, ad-hoc queries and dashboards needs updating. And PromQL is so opaque, it just never give you an error unless there is a syntax issue. We need tools like https://github.com/cloudflare/pint https://github.com/cloudflare/pint just to know if my alert description isn’t trying to render a label that’s just not gonna be there, etc

znpy 11mo ago
> And PromQL is so opaque, it just never give you an error unless there is a syntax issue. We need tools like https://github.com/cloudflare/pint https://github.com/cloudflare/pint just to know if my alert description isn’t trying to render a label that’s just not gonna be there, etc PromQL should be blamed on prometheus though, not on grafana.

esseph 11mo ago

I feel this article in my bones. It's rough out there.

Too 11mo ago

It’s as though they deliberately make it complex and constantly moving to wear you out to give in to the convenient SaaS offer.

yomismoaqui 11mo ago

If your observability stack works and you are fine with it, do you need to update it? I understand updating some front facing service due to a vulnerability... But for a thing that it's internally accessible?

plqbfbv 11mo ago

I'm also seriously considering dropping Grafana for good for the same reasons stated in the post. Every year I need to rebuild a dashboard, reconfigure alerts, use the shiny new toy, etc etc. I'm tired. I just want the thing to alert me when something's down, and ideally if the check doesn't change and the datasource and metric don't change, the dashboard definition and the alert definition should be the same for the last and the next 10 years. The UI used to have the most 4-5 important links in the sidebar, now it's 10 menus with submenus of submenus, and I never know where to find the basics: Dashboards and Alerts. When something goes off I don't have time to re-learn the UI I look at maybe once a month.

valyala 11mo ago
Take a look at Perses - https://perses.dev/ https://perses.dev/

atoav 11mo ago

This is what keeps me from using things like these. Meanwhile I have uinx-oid scripts that keep chugging on up to date servers for more than a decade without any need to change them. I know there is always the temptation to make it really shiny and nice. But the more moving parts your system has the liklier it becomes that something will fail eventually. And as it happens these failures happen usually at the time when it is most inconvenient for all people involved. That doesn't mean that complex software cannot work reliably, but it takes more effort for the developer side to honor that unwritten contract with their users (if they are even aware of it). This is why sometimes doing it yourself, on your own servers can be benefitial because it gives you more control.

paxys 11mo ago

I don't know why software developers feel the urge to stay on the bleeding edge of every product and update their setup every week then turn around and compain "this stuff isn't stable!" I've had a grafana + prometheus setup on my servers since like 2017. It worked then and works today. I log in maybe once every year or two to update to a newer LTS version. Every dashboard is still pristine, and nothing has ever broken. I don't understand most of the words in the linked post and don't need to. The core package is the boring solution that 99% of people here need, and that works great.

hoherd 11mo ago
> Every dashboard is still pristine How did you handle the angular deprecation in grafana? Or are you just staying in an older version that still supports it?
- tekla 11mo ago
  I've done it and it wasn't too bad? The auto migrate worked pretty well for 99% of stuff (hundreds of dashboards). Though I won't say I loved doing it.
- paxys 11mo ago
  All of my dashboards were auto updated so I didn't really have to do anything.

piterrro 11mo ago

Who remembers Graphite and Carbon? This was 2010 era…

mt42or 11mo ago

Victoriametrics and never change. Avoid grafana ingestion tools. Grafana for ui, alerting.

blueybingo 11mo ago

Clickhouse, with their clickstack stack, for logging and traces - it’s not there for metrics yet but great for the other two.

blueybingo 11mo ago
Also mit and apache if you’re not fond of agpl
akvadrako 11mo ago
Victoria metrics which has similar database design to clickhouse is good for metrics. But it doesn't have a complete dashboard UI like Grafana.
- hagen1778 11mo ago
  Just to add, VictoriaMetrics covers all 3 signals: - VictoriaMetrics for metrics. With Prometheus API support, so it integrates with Grafana using Prometheus datasource. It has its own Grafana datasource with extra functionality too. - VictoriaLogs for logs. Integrates natively with Grafana using VictoriaLogs datasource. - VictoriaTraces for traces. With Jaeger API support, so it intergrates with Grafana using Jaeger datasource. All 3 solutions support alerting, managed by same team, are Apache2 licensed, are focused on resource efficiency and simiplicity.

bean469 11mo ago

> At some point we needed a monitoring solution, and Zabbix didn’t fit well into the new and declarative world of containers and Docker. I wonder why they think that is the case, as Zabbix has native support for monitoring VMs and Docker containers, including support for discovering newly spawned ones and all that jazz

buro9 11mo ago

A reply of sorts, and a standard disclaimer that I work at Grafana Labs. > career-driven development we don't have this and promote and reward as frequently for "I've done solid operations" as we do for "I've added this feature" (I'm on promotion committees and can state this confidently). what we do have is high autonomy for engineers. This autonomy means it's a freedom that engineers have to identify problems they feel are important and to work on them, they do not need permission and leadership do not veto this. Some of the best features in the last few years have been a direct result of this autonomy, it's one of the things that makes working here so attractive to many of the engineers. But, with autonomy comes a little chaos, and not everything that is done is going to satisfy every end user of OSS or paid customer (of which these are a small percent of the whole). a lot of the innovation speed is just in the DNA of the company, even the creation of Grafana can be traced to a desire to get things done; Torkel wanted Kibana to also work for Prometheus, Kibana declined to add this, Torkel didn't stand still and added things to a fork of Kibana now called Grafana and hasn't stopped adding things since. > They also deprecated Angular within Grafana and switched to React for dashboards. This broke most existing dashboards. we did, I think the entire journey was 7 years long, communicated many times, over at least 6 major releases. maintaining dashboards in two languages increased complexity, whilst reducing compatibility, and gave a very large security surface to be worried about. we communicated clearly, provided migration tools, put it in release notes, updated docs, repeated it at conferences and on community calls. arguably we went too slow, and should've ripped the band-aid off, but we were sensitive to the fact that it was a breaking change and so we proceeded with extreme caution. it's done now, it was finally completed in the last version, only a very small number of users reported impact as a result of the time and care taken on this. > I just hope OTEL settles, gets stable and boring fast this is distinct from Grafana, but it's a good point... OTel is the product of virtually every vendor at this point, and a hell of a lot of engineers, it now has a lot of momentum and the pace is unlikely to ease up due to the sheer number of contributions and things that OTel as a community wishes to achieve. the most likely eventuality is that enough stability emerges to allow vendors (including but not limited to Grafana Labs) to abstract away the pace of innovation occurring underneath, but this is in tension with providing the benefits of the innovation to the people that use it. what I would say is that for most people the boring and slow path does still exist, and it's still good... just use Prometheus, a logging option of your choice, and simple Grafana dashboards and alerts. that combination hasn't varied in years, and those on it today are still immune from caring about the pace of innovation and change in OTel and across the Observability industry. OTel is being used in production at massive scale by lots of companies, but whether your project or company need move to it now reflects your priorities, many are adopting to gain independence from vendors, or just control over their telemetry, but many customers are also saying they're happy to stay on the slow and boring path and for everything to work predictably with low cost to keep pace... it works too.