4 ms·

Good luck trying to monitor using debugging tools like logs and metrics. Devs thinking they can do Ops because reasons. What you really need is state monitoring

Good luck trying to monitor using debugging tools like logs and metrics. Devs thinking they can do Ops because reasons. What you really need is state monitoring. You know, the old Nagios style that everybody thinks is out of date because, again, reasons. Just start by having e2e tests for everything and add intermediate tests when you discover points of failure and bottlenecks. And don't give me that "Nagios does not scale" bullshit. You don't need to and can't monitor 9 trillion things. You also have Icinga and other options that scale for state monitoring. In place of argumentation which would be too long, let me give you an analogy. Monitoring is telling you the fire alarm went off. The exact reason is left for a investigation and that's where saving data like logs and metrics helps. But first you need to evacuate the building and call the fire department. If instead of detecting the smoke (e2e) you try to monitor all the data like pressure in gas pipes, all the pipes fittings, how many lighters are in the building and where, etc, you will wake up every night 10 times for nothing and when a fire starts you will burn sound asleep. Oh, and don't try to scale Prometheus, that's what things like victoriametrics are for.

wspeirs 3y ago
>If instead of detecting the smoke (e2e) you try to monitor all the data like pressure in gas pipes, all the pipes fittings, how many lighters are in the building and where, etc, you will wake up every night 10 times for nothing Without this you'll always be awoken by smoke, and at that point it's too late... there's already a fire. However, if you can monitor other things (gas pipes, lighters, etc), you might be able to remediate a problem before it starts smoking or burning. There's no one-size-fits-all. Start with some obvious failure (e2e tests/checks; aka smoke), and when you root-cause, add in additional checks to hopefully catch things before the smoke. However, this also requires you update (or remove) these additional checks as your software and infra change... that's what most people forget, and then alert fatigue sets in, and everything becomes a non-alert.
- teddyh 3y ago
  > when you root-cause, add in additional checks to hopefully catch things before the smoke. No, what you should do is update the software to handle or avoid the problem gracefully. Keep the smoke detectors, but don’t keep adding tiny things to monitor and keep in sync with your ever-changing software.
  - trabant00 3y ago
    That can't always be done. CAP theorem is still true for example.
- trabant00 3y ago
  > this also requires you update (or remove) these additional checks as your software and infra change... that's what most people forget, and then alert fatigue sets in, and everything becomes a non-alert This is what I forgot to mention, monitoring is a process, not a tool you put in place and forget about. And I agree that this is the main cause monitoring goes to shit. But if you think about it a little deeper you arrive at the requirement to have monitoring done by somebody who understands the business and the infra/software stack and has been at that company a few years at least. While in reality monitoring is mostly an afterthought cost center and employees are rotated far too often. It's not a glamorous position either, I don't think I once met somebody who wants to work there.
pphysch 3y ago
> What you really need is state monitoring. You know, the old Nagios style that everybody thinks is out of date because, again, reasons. Just start by having e2e tests for everything and add intermediate tests when you discover points of failure and bottlenecks. "State monitoring" are just different metrics. State of X at time Y was Z. Blackbox Exporter covers a lot of ground here. If you need more advanced application instrumentation, you can build that, but there is real value in having all of your metrics in one place instead of siloing different bits out to Nagios.
- trabant00 3y ago
  > "State monitoring" are just different metrics True but also misleading. In the end everything is really math but that is unhelpful. What matters is the model you apply to your numbers. And state monitoring as thought by Nagios is that you have 4 states: ok, warning, critical, unknown. And you have hard and soft states. This can be replicated with prometheus queries but it's incredibly complex and error prone, not to mention you will destroy your metrics storage IO. Theory crafting aside I've created in a few months a monitoring system using Icinga that had virtually no false positives or negatives. That you could trust to phone you straight though without fear it will miss something important for business or wake you up for nothing. I never saw anything like this done with metrics or logs done by anybody.
  - preseinger 3y ago
    icinga? nagios? you're stuck in 1999 friend these tools, and the patterns they enable, have been outmoded for over a decade
    - trabant00 3y ago
      Your arguments won me over
  - pphysch 3y ago
    > I never saw anything like this done with metrics or logs done by anybody. Well, I assure you it can be done and it is not nearly as scary as you presume. I migrated an old Nagios+Cacti system to Prometheus (well, VictoriaMetrics) for a DC with thousands of servers. Nagios using 4-valued datatypes for virtually EVERYTHING is frankly just poor design. As soon as you move away from binary state, the semantics becomes ambiguous. Consider these two monitoring systems: - Nagios-style: 1+ state metric for your HTTP service. "Warning" indicates something wrong but not critical. What exactly? Usually you will see the warning and then look at metrics in sysstat or Cacti as a follow up. - Prometheus-style: 1+ state metric for HTTP probe, and a few relevant performance metrics. RPS, latency, etc. The merit of the latter is you use the exact same alerting + visualization infrastructure for both types of data. You create a CRITICAL alert if the service is down, and WARNING alerts if RPS/latency crosses thresholds. Importantly, this data is not siloed into Nagios and can be used for other purposes such as usage analytics and cross-domain incident analysis. The alerting service is just one possible consumer.

wspeirs 3y ago

>If instead of detecting the smoke (e2e) you try to monitor all the data like pressure in gas pipes, all the pipes fittings, how many lighters are in the building and where, etc, you will wake up every night 10 times for nothing Without this you'll always be awoken by smoke, and at that point it's too late... there's already a fire. However, if you can monitor other things (gas pipes, lighters, etc), you might be able to remediate a problem before it starts smoking or burning. There's no one-size-fits-all. Start with some obvious failure (e2e tests/checks; aka smoke), and when you root-cause, add in additional checks to hopefully catch things before the smoke. However, this also requires you update (or remove) these additional checks as your software and infra change... that's what most people forget, and then alert fatigue sets in, and everything becomes a non-alert.

teddyh 3y ago
> when you root-cause, add in additional checks to hopefully catch things before the smoke. No, what you should do is update the software to handle or avoid the problem gracefully. Keep the smoke detectors, but don’t keep adding tiny things to monitor and keep in sync with your ever-changing software.
- trabant00 3y ago
  That can't always be done. CAP theorem is still true for example.
trabant00 3y ago
> this also requires you update (or remove) these additional checks as your software and infra change... that's what most people forget, and then alert fatigue sets in, and everything becomes a non-alert This is what I forgot to mention, monitoring is a process, not a tool you put in place and forget about. And I agree that this is the main cause monitoring goes to shit. But if you think about it a little deeper you arrive at the requirement to have monitoring done by somebody who understands the business and the infra/software stack and has been at that company a few years at least. While in reality monitoring is mostly an afterthought cost center and employees are rotated far too often. It's not a glamorous position either, I don't think I once met somebody who wants to work there.

pphysch 3y ago

> What you really need is state monitoring. You know, the old Nagios style that everybody thinks is out of date because, again, reasons. Just start by having e2e tests for everything and add intermediate tests when you discover points of failure and bottlenecks. "State monitoring" are just different metrics. State of X at time Y was Z. Blackbox Exporter covers a lot of ground here. If you need more advanced application instrumentation, you can build that, but there is real value in having all of your metrics in one place instead of siloing different bits out to Nagios.

trabant00 3y ago
> "State monitoring" are just different metrics True but also misleading. In the end everything is really math but that is unhelpful. What matters is the model you apply to your numbers. And state monitoring as thought by Nagios is that you have 4 states: ok, warning, critical, unknown. And you have hard and soft states. This can be replicated with prometheus queries but it's incredibly complex and error prone, not to mention you will destroy your metrics storage IO. Theory crafting aside I've created in a few months a monitoring system using Icinga that had virtually no false positives or negatives. That you could trust to phone you straight though without fear it will miss something important for business or wake you up for nothing. I never saw anything like this done with metrics or logs done by anybody.
- preseinger 3y ago
  icinga? nagios? you're stuck in 1999 friend these tools, and the patterns they enable, have been outmoded for over a decade
  - trabant00 3y ago
    Your arguments won me over
- pphysch 3y ago
  > I never saw anything like this done with metrics or logs done by anybody. Well, I assure you it can be done and it is not nearly as scary as you presume. I migrated an old Nagios+Cacti system to Prometheus (well, VictoriaMetrics) for a DC with thousands of servers. Nagios using 4-valued datatypes for virtually EVERYTHING is frankly just poor design. As soon as you move away from binary state, the semantics becomes ambiguous. Consider these two monitoring systems: - Nagios-style: 1+ state metric for your HTTP service. "Warning" indicates something wrong but not critical. What exactly? Usually you will see the warning and then look at metrics in sysstat or Cacti as a follow up. - Prometheus-style: 1+ state metric for HTTP probe, and a few relevant performance metrics. RPS, latency, etc. The merit of the latter is you use the exact same alerting + visualization infrastructure for both types of data. You create a CRITICAL alert if the service is down, and WARNING alerts if RPS/latency crosses thresholds. Importantly, this data is not siloed into Nagios and can be used for other purposes such as usage analytics and cross-domain incident analysis. The alerting service is just one possible consumer.