5 ms·

It's hard because it's being pushed and done by people with different agendas that also aren't necessarily the right agendas to be prioritizing. - mistake #1,

It's hard because it's being pushed and done by people with different agendas that also aren't necessarily the right agendas to be prioritizing. - mistake #1, assuming it's a technical thingy that some techie person should do and they should just get on with it and do it. Whatever it is. The mistake here is that without guidance this person is going to be selecting some random tools mostly focused on operational things (logging, infrastructure). These have low value for other stakeholders. Yes you need that but these are also commodities. - mistake #2, each of the stakeholders adds their preferred tools to the mix and the end result is a bunch of poorly integrated tools with a lot of complexity. - mistake #3, assuming that selecting a tool means the job is done. It's not. Most of these tools assume that you have some notion of how to use the tool and what you want out of the tool. Getting the tool is a the beginning, not the end. The combination is lethal. Lots of complexity, lots of data, lots of cost, and not a whole lot of value being delivered. The fix is an observability mindset that exists in product, tech, and business departments. Select 1 tool, not 2, 5, or 10. The more tools, the more fragmented the effort and the more duplication of effort and mistakes. Have a plan, engineer for the system to be observable, have a notion of things that are important to track, etc. Don't dump the responsibility on some techie but actually reflect a bit on what you are looking to get out of this and what this is worth to you.

juancn 3y ago
That, and the tooling sucks. The good tools are expensive, both Sumologic and Wavefront are amazing, but past a certain scale it's hard to justify the cost. On the open source side Grafana and its stack (tempo, loki and prometheus) all just suck, they kinda scale and check all the boxes, but in a really stupid way (data nodes should probably be query nodes, rather than moving everything from storage and caching, everything is supposed to be a dashboard, don't get me started on the query languages). The Grafana UI was built by the enemy and explorations are incredibly annoying. I'm more of an exploratory user, not a dashboard one (think level 3 on-call).
- lazyasciiart 3y ago
  We just moved to grafana from New Relic and it makes me angry just to try and look at monitoring, we have lost so much utility. And the company spent a year on the transition and a shit ton of work, so it’s not (just) that we had good workflows set up that haven’t come across. (I am also a heavy exploratory user)
  - ksaxena 3y ago
    Have you tried Signoz (https://signoz.io/ https://signoz.io/) yet?
  - llama052 3y ago
    What are you missing specifically that New Relic has? We are a full grafana prometheus shop and the New Relic/Datadog price tag is insanity compared to the utilization we get out of the free tooling. Once you get over the learning curve of promQL or whatever you're looking for it's not bad at all.
  - Jedd 3y ago
    If you were heavy users of APM, then that's going to be a step backwards, I'm sure - but Grafana's just a visualisation tool, so I'm not sure what actual stack you're comparing there. I find NewRelic's UI to be more frustrating, but I've spent a lot more time with Grafana's (while very much not a fan of their relentless tinkering with UI layout).
  - mikeshi42 3y ago
    Would love to learn what kind of exploration workflows you've missed in Grafana? Context: Founder building an affordable alternative in the SaaS observability market (hyperdx.io) - I assume your company might have moved when NR introduced new seat-based pricing?
- llama052 3y ago
  > On the open source side Grafana and its stack (tempo, loki and prometheus) all just suck, they kinda scale and check all the boxes, but in a really stupid way (data nodes should probably be query nodes, rather than moving everything from storage and caching, everything is supposed to be a dashboard, don't get me started on the query languages). hmm, you generally move prometheus data into a higher engine like Thanos. It does split layers up pretty well into separate components. Honestly it does have a learning curve but the cost is generally free and compared to New Relic or something like DataDog they aren't all that different learning curve wise. Either way it beats the days of ganglia or nagios static screenshots of some half baked python metric. Or even kibana where you'd have to run a gigantic ELK stack to get any data out of it.
  - EdwardDiego 3y ago
    > Either way it beats the days of ganglia or nagios static screenshots of some half baked python metric. Or even kibana where you'd have to run a gigantic ELK stack to get any data out of it. Holy hell, if you'd mentioned Munin also, I'd swear we worked for the same company. Spent ages making Munin play nice with Nagios. Ganglia was okay for monitoring Spark jobs running in a Yarn cluster, I guess, back in the early days it the GangliaSink shipped by default, and it was far better than trying to scrape every worker instance via JMX... And then yeah, we moved to ELK for metrics. Spent more time fixing up indexing and ingestion issues. Then I introduced Prometheus + Grafana, it was far simpler to run (at least until we got big enough to need Thanos...) and far simpler to create dashboards and write queries in.
    - llama052 3y ago
      Oh man, Munin.. I had forgotten about that! We are currently running a databricks cluster and avoiding the static-ish ganglia screenshots and injecting prometheus exporters to get data into grafana. Honestly wish the entire data space was more open in that regard. Really glad things are evolving, it's not perfect but it's sure as hell better than it was!
- smcleod 3y ago
  Sumologic is one of the worst tools I’ve ever had to use for monitoring, its “ok” for looking at a pile of logs, not great but “ok”, but when companies try to use it for monitoring - gosh, it’s like going back to the dark ages.
- Jedd 3y ago
  Prometheus isn't Grafana's - so you can't hold Grafana responsible for PromQL either - but for scaling you want their Mimir product anyway. What scaling issues are you suffering - limits, performance, costs? Exploration is pretty good in Grafana - sounds like you're using dashboards to do discovery / exploration rather than their explore tool, which got substantially better in recent versions with their metrics encyclopaedia feature.
- applied_heat 3y ago
  What do you use for exploring? KST-plot has been my go to for real time monitoring and after the fact analysis of time series data. It is super fast, I can pan, I can zoom, with keyboard shortcuts, and I can perform functions on the data to create new time series. Airbus is/was a big user of KST for their air frame test data. I haven’t found anything that can replace it
eschneider 3y ago
All excellent points. I'd just like to add that when tackling monitoring, know exactly what problem(s) you're trying to solve. Monitoring-for-monitoring's-sake is a fool's game. Are you monitoring to get usage patters? Debug production problems? Troubleshoot for customer service? Know what actionable information you want/need from your monitoring solution and let that guide the approach.
tetha 3y ago
> - mistake #1, assuming it's a technical thingy that some techie person should do and they should just get on with it and do it. Whatever it is. The mistake here is that without guidance this person is going to be selecting some random tools mostly focused on operational things (logging, infrastructure). These have low value for other stakeholders. Yes you need that but these are also commodities. I'm honestly pretty much giving up on monitoring at work. I enjoy working on monitoring, and I think good monitoring and tracing is a strong edge you can have at a technical level. After a certain point, I'm pretty sure you will lose control without good monitoring, tracing and alerting. But practically, whenever there is a glimmer of hope, aka spare time, exists to do some work on the monitoring, the entire product stack realizes: There is spare time. Let's dump any combination of (i) introducing entirely new tech into the stack, (ii) Requesting weeks of analysis if postgres/any underutilized upstream component might be the cause of slowness for an individual database of a cluster on a group of postgres servers could be the cause of slowness, (iii) elevate something to "critical we cannot work and customers are crying just now" and (iv) some production outage by a shitty change due to an architectural weakness the product had since 3 years and we've been pointed it out, but we need a "task force" of "all qualified people on deck" except that the issue isn't related to e.g. the database, and as such, the postgres DBAs are twiddling thumbs. But "working together" means they must be part of "all hands on deck". Oh, and then everyone starts yelling why the we can't just pile new topics on infra every 3 month, and why we can't have an up-to-date modern monitoring stack at the same time from the same guys. Monitoring and alerting just has zero visible value outside of infrastructure and operations imo and I'm pretty much done fighting for it. Sorry for being somewhat sour about this, I can be about things I really like.
- javier2 3y ago
  I am the same, I love good monitoring. My place has been very focused on it, and we struggle to find time to work on it beyond basic «does it work». Yet We have so much monitoring we are drowning in metrics.
emmelaich 3y ago
The answer is to have clean lines of separation between collecting, reporting, alerting, analytics. For collecting, choose a compact unambiguous format for the data and never change it. Then you can change the other stuff without boiling the ocean.
- lelanthran 3y ago
  > The answer is to have clean lines of separation between collecting, reporting, alerting, analytics. What's the difference between reporting and analytics? I think I know but I'm not sure it matches what you mean.[1] [1] I assume that reporting is for detecting trends, capacity planning, etc and analytics is for diagnosing issues.
  - emmelaich 3y ago
    Nothing really I suppose. Maybe reporting is periodic and analysis is ad-hoc?
    - lelanthran 3y ago
      After thinking about it and read what the various tools do, I decided that reporting is for ops purposes (capacity planning , etc) and analytics is for product purposes (engagement, downloads, signups, etc).

juancn 3y ago

That, and the tooling sucks. The good tools are expensive, both Sumologic and Wavefront are amazing, but past a certain scale it's hard to justify the cost. On the open source side Grafana and its stack (tempo, loki and prometheus) all just suck, they kinda scale and check all the boxes, but in a really stupid way (data nodes should probably be query nodes, rather than moving everything from storage and caching, everything is supposed to be a dashboard, don't get me started on the query languages). The Grafana UI was built by the enemy and explorations are incredibly annoying. I'm more of an exploratory user, not a dashboard one (think level 3 on-call).

lazyasciiart 3y ago
We just moved to grafana from New Relic and it makes me angry just to try and look at monitoring, we have lost so much utility. And the company spent a year on the transition and a shit ton of work, so it’s not (just) that we had good workflows set up that haven’t come across. (I am also a heavy exploratory user)
- ksaxena 3y ago
  Have you tried Signoz (https://signoz.io/ https://signoz.io/) yet?
- llama052 3y ago
  What are you missing specifically that New Relic has? We are a full grafana prometheus shop and the New Relic/Datadog price tag is insanity compared to the utilization we get out of the free tooling. Once you get over the learning curve of promQL or whatever you're looking for it's not bad at all.
- Jedd 3y ago
  If you were heavy users of APM, then that's going to be a step backwards, I'm sure - but Grafana's just a visualisation tool, so I'm not sure what actual stack you're comparing there. I find NewRelic's UI to be more frustrating, but I've spent a lot more time with Grafana's (while very much not a fan of their relentless tinkering with UI layout).
- mikeshi42 3y ago
  Would love to learn what kind of exploration workflows you've missed in Grafana? Context: Founder building an affordable alternative in the SaaS observability market (hyperdx.io) - I assume your company might have moved when NR introduced new seat-based pricing?
llama052 3y ago
> On the open source side Grafana and its stack (tempo, loki and prometheus) all just suck, they kinda scale and check all the boxes, but in a really stupid way (data nodes should probably be query nodes, rather than moving everything from storage and caching, everything is supposed to be a dashboard, don't get me started on the query languages). hmm, you generally move prometheus data into a higher engine like Thanos. It does split layers up pretty well into separate components. Honestly it does have a learning curve but the cost is generally free and compared to New Relic or something like DataDog they aren't all that different learning curve wise. Either way it beats the days of ganglia or nagios static screenshots of some half baked python metric. Or even kibana where you'd have to run a gigantic ELK stack to get any data out of it.
- EdwardDiego 3y ago
  > Either way it beats the days of ganglia or nagios static screenshots of some half baked python metric. Or even kibana where you'd have to run a gigantic ELK stack to get any data out of it. Holy hell, if you'd mentioned Munin also, I'd swear we worked for the same company. Spent ages making Munin play nice with Nagios. Ganglia was okay for monitoring Spark jobs running in a Yarn cluster, I guess, back in the early days it the GangliaSink shipped by default, and it was far better than trying to scrape every worker instance via JMX... And then yeah, we moved to ELK for metrics. Spent more time fixing up indexing and ingestion issues. Then I introduced Prometheus + Grafana, it was far simpler to run (at least until we got big enough to need Thanos...) and far simpler to create dashboards and write queries in.
  - llama052 3y ago
    Oh man, Munin.. I had forgotten about that! We are currently running a databricks cluster and avoiding the static-ish ganglia screenshots and injecting prometheus exporters to get data into grafana. Honestly wish the entire data space was more open in that regard. Really glad things are evolving, it's not perfect but it's sure as hell better than it was!
smcleod 3y ago
Sumologic is one of the worst tools I’ve ever had to use for monitoring, its “ok” for looking at a pile of logs, not great but “ok”, but when companies try to use it for monitoring - gosh, it’s like going back to the dark ages.
Jedd 3y ago
Prometheus isn't Grafana's - so you can't hold Grafana responsible for PromQL either - but for scaling you want their Mimir product anyway. What scaling issues are you suffering - limits, performance, costs? Exploration is pretty good in Grafana - sounds like you're using dashboards to do discovery / exploration rather than their explore tool, which got substantially better in recent versions with their metrics encyclopaedia feature.
applied_heat 3y ago
What do you use for exploring? KST-plot has been my go to for real time monitoring and after the fact analysis of time series data. It is super fast, I can pan, I can zoom, with keyboard shortcuts, and I can perform functions on the data to create new time series. Airbus is/was a big user of KST for their air frame test data. I haven’t found anything that can replace it

eschneider 3y ago

All excellent points. I'd just like to add that when tackling monitoring, know exactly what problem(s) you're trying to solve. Monitoring-for-monitoring's-sake is a fool's game. Are you monitoring to get usage patters? Debug production problems? Troubleshoot for customer service? Know what actionable information you want/need from your monitoring solution and let that guide the approach.

tetha 3y ago

> - mistake #1, assuming it's a technical thingy that some techie person should do and they should just get on with it and do it. Whatever it is. The mistake here is that without guidance this person is going to be selecting some random tools mostly focused on operational things (logging, infrastructure). These have low value for other stakeholders. Yes you need that but these are also commodities. I'm honestly pretty much giving up on monitoring at work. I enjoy working on monitoring, and I think good monitoring and tracing is a strong edge you can have at a technical level. After a certain point, I'm pretty sure you will lose control without good monitoring, tracing and alerting. But practically, whenever there is a glimmer of hope, aka spare time, exists to do some work on the monitoring, the entire product stack realizes: There is spare time. Let's dump any combination of (i) introducing entirely new tech into the stack, (ii) Requesting weeks of analysis if postgres/any underutilized upstream component might be the cause of slowness for an individual database of a cluster on a group of postgres servers could be the cause of slowness, (iii) elevate something to "critical we cannot work and customers are crying just now" and (iv) some production outage by a shitty change due to an architectural weakness the product had since 3 years and we've been pointed it out, but we need a "task force" of "all qualified people on deck" except that the issue isn't related to e.g. the database, and as such, the postgres DBAs are twiddling thumbs. But "working together" means they must be part of "all hands on deck". Oh, and then everyone starts yelling why the we can't just pile new topics on infra every 3 month, and why we can't have an up-to-date modern monitoring stack at the same time from the same guys. Monitoring and alerting just has zero visible value outside of infrastructure and operations imo and I'm pretty much done fighting for it. Sorry for being somewhat sour about this, I can be about things I really like.

javier2 3y ago
I am the same, I love good monitoring. My place has been very focused on it, and we struggle to find time to work on it beyond basic «does it work». Yet We have so much monitoring we are drowning in metrics.

emmelaich 3y ago

The answer is to have clean lines of separation between collecting, reporting, alerting, analytics. For collecting, choose a compact unambiguous format for the data and never change it. Then you can change the other stuff without boiling the ocean.

lelanthran 3y ago
> The answer is to have clean lines of separation between collecting, reporting, alerting, analytics. What's the difference between reporting and analytics? I think I know but I'm not sure it matches what you mean.[1] [1] I assume that reporting is for detecting trends, capacity planning, etc and analytics is for diagnosing issues.
- emmelaich 3y ago
  Nothing really I suppose. Maybe reporting is periodic and analysis is ad-hoc?
  - lelanthran 3y ago
    After thinking about it and read what the various tools do, I decided that reporting is for ops purposes (capacity planning , etc) and analytics is for product purposes (engagement, downloads, signups, etc).