5 ms·

Prometheus metrics saves us from painful kernel debugging (2022)

116 points by goranmoomin 2y ago

deleted 2y ago
[deleted]
lordnacho 2y ago
Sounds like they have node on their machines. Not the js framework, but the prometheus/grafana package that gives you all the meters for a generic system monitoring dashboard. Disk usage, CPU, memory, it's all set up already, just plug and play. In fact, I found a memory leak this way not long ago. Super useful having this on your infra, saves a lot of time.
- luckman212 2y ago
  Anyone know of a good guide for configuring windows_exporter and setting up alerts based on that data? I was tasked with monitoring some Windows endpoints for a memory leak and I failed in my attempts so far.
- gouthamve 2y ago
  Yup, https://github.com/prometheus/node_exporter https://github.com/prometheus/node_exporter is the standard way to monitor machine metrics with Prometheus.
- alfons_foobar 2y ago
  FYI, it's called "node_exporter" ;)
tass 2y ago
This awakened a memory from last year where a colleague and I were trying to understand where an increase in Linux memory was coming from in machines that hadn’t been rebooted in a while. This was alerted to by Prometheus metrics. Even after all apps had been restarted, it persisted. Turned out to be a leak of slab memory allocations by a kernel module. That kernel module had since been updated, but all previous versions were still loaded by the kernel so the leak persisted until the next reboot. The leaky kernel module was CrowdStrike’s falcon sensor. It started a discussion where engineering had no option but to run these things for the sake of security, there were no instances where it actually caught anything, but it had the potential to cause incidents and outages.
- josephcsible 2y ago
  Do you mean "engineering had no option but to run these things for the sake of compliance"?
  - tass 2y ago
    Compliance was the reasoning they raised, yes. Whether you require this particular tool to meet compliance, and whether that tool needs to be deployed across the entire stack might be open to interpretation though.
    - atomicnumber3 2y ago
      Open to interpretation but it often ends up something like this: 1. "Do we need to use this shovelware? Can't we achieve compliance via <a few sane steps>? 2. CTO/CEO: We can either use this software and you don't have to think about it, or we can do your thing and I'm going to hang you out to dry the second literally anything goes wrong with it. 3. "Ah. Ok nevermind." [starts updating resume] IMO it's part of this toxic culture against DIYing literally anything. Yes NIH can be bad, but so is the complete opposite where you make insane decisions (technical or money-wise) because of a culture where people are punished for trying to DIY when it does make sense. At a particularly toxic past org of mine, we actually had a "joke" (not the ha-ha kind of joke, the i can't believe we actually do this kind) where for a few key decisions we appointed someone to be the "Jesus" for that decision - someone who was already planning to quit, so they could make a decision that was correct but would get them punished, then "die for our sins", leaving us with the benefits of a correct decision having been made but without any of the political fallout of having done something that offends the sensibilities of leadership. Yes eventually you run out of messiahs to sacrifice and yes it sucked to work where a lot. But damn did they pay really well, so, yknow.
      - znpy 2y ago
        > against DIYing literally anything I get your feeling but I largely disagree: I’ve seen too many DYI mini-projects that implemented just enough and became too much important, and when they broke no one was around to fix that (the implementer had left a long ago and left scarce to no documentation at all). The nice thing about using an off-the-shelf product is that usually either it comes with support or there’s some kind of community where you can go and ask for help). There is this insidious cognitive bias in (software?) engineers to only consider the happy path when thinking about the consequences of their actions.
spiffytech 2y ago
My team spent weeks using log-aggregated metrics to gradually figure out why servers' clocks would go out of whack. It turned out Docker Swarm made undocumented† use of a UDP port that some VMware product also used, and once in a while they'd cross the streams. We only figured it out because we put every system event we could find onto a Grafana graph and narrowed down which ones kept happening at the same time. † I think? It's been a while, might have just been hard to find.
louwrentius 2y ago
As a side note, I’m into storage performance and the node-exporter data is absolutely spot on. I performed storage benchmarks with FIO and the metrics matched the loads and reported os metrics (iostat) perfectly. I actually made a Grafana dashboard[0] for it, but haven’t used this in a while myself. [0]: https://grafana.com/grafana/dashboards/11801-i-o-statistics/ https://grafana.com/grafana/dashboards/11801-i-o-statistics/
- nsguy 2y ago
  It's all standard Linux metrics that node-exporter exposes. Mostly from procfs.
  - louwrentius 2y ago
    I’ve tried to get the same results with telegraf and influxdb and never got the metrics right

116 points by goranmoomin 2y ago

deleted 2y ago

[deleted]

Sounds like they have node on their machines. Not the js framework, but the prometheus/grafana package that gives you all the meters for a generic system monitoring dashboard. Disk usage, CPU, memory, it's all set up already, just plug and play. In fact, I found a memory leak this way not long ago. Super useful having this on your infra, saves a lot of time.

luckman212 2y ago
Anyone know of a good guide for configuring windows_exporter and setting up alerts based on that data? I was tasked with monitoring some Windows endpoints for a memory leak and I failed in my attempts so far.
gouthamve 2y ago
Yup, https://github.com/prometheus/node_exporter https://github.com/prometheus/node_exporter is the standard way to monitor machine metrics with Prometheus.
alfons_foobar 2y ago
FYI, it's called "node_exporter" ;)

tass 2y ago

This awakened a memory from last year where a colleague and I were trying to understand where an increase in Linux memory was coming from in machines that hadn’t been rebooted in a while. This was alerted to by Prometheus metrics. Even after all apps had been restarted, it persisted. Turned out to be a leak of slab memory allocations by a kernel module. That kernel module had since been updated, but all previous versions were still loaded by the kernel so the leak persisted until the next reboot. The leaky kernel module was CrowdStrike’s falcon sensor. It started a discussion where engineering had no option but to run these things for the sake of security, there were no instances where it actually caught anything, but it had the potential to cause incidents and outages.

josephcsible 2y ago
Do you mean "engineering had no option but to run these things for the sake of compliance"?
- tass 2y ago
  Compliance was the reasoning they raised, yes. Whether you require this particular tool to meet compliance, and whether that tool needs to be deployed across the entire stack might be open to interpretation though.
  - atomicnumber3 2y ago
    Open to interpretation but it often ends up something like this: 1. "Do we need to use this shovelware? Can't we achieve compliance via <a few sane steps>? 2. CTO/CEO: We can either use this software and you don't have to think about it, or we can do your thing and I'm going to hang you out to dry the second literally anything goes wrong with it. 3. "Ah. Ok nevermind." [starts updating resume] IMO it's part of this toxic culture against DIYing literally anything. Yes NIH can be bad, but so is the complete opposite where you make insane decisions (technical or money-wise) because of a culture where people are punished for trying to DIY when it does make sense. At a particularly toxic past org of mine, we actually had a "joke" (not the ha-ha kind of joke, the i can't believe we actually do this kind) where for a few key decisions we appointed someone to be the "Jesus" for that decision - someone who was already planning to quit, so they could make a decision that was correct but would get them punished, then "die for our sins", leaving us with the benefits of a correct decision having been made but without any of the political fallout of having done something that offends the sensibilities of leadership. Yes eventually you run out of messiahs to sacrifice and yes it sucked to work where a lot. But damn did they pay really well, so, yknow.
    - znpy 2y ago
      > against DIYing literally anything I get your feeling but I largely disagree: I’ve seen too many DYI mini-projects that implemented just enough and became too much important, and when they broke no one was around to fix that (the implementer had left a long ago and left scarce to no documentation at all). The nice thing about using an off-the-shelf product is that usually either it comes with support or there’s some kind of community where you can go and ask for help). There is this insidious cognitive bias in (software?) engineers to only consider the happy path when thinking about the consequences of their actions.

spiffytech 2y ago

My team spent weeks using log-aggregated metrics to gradually figure out why servers' clocks would go out of whack. It turned out Docker Swarm made undocumented† use of a UDP port that some VMware product also used, and once in a while they'd cross the streams. We only figured it out because we put every system event we could find onto a Grafana graph and narrowed down which ones kept happening at the same time. † I think? It's been a while, might have just been hard to find.

louwrentius 2y ago

As a side note, I’m into storage performance and the node-exporter data is absolutely spot on. I performed storage benchmarks with FIO and the metrics matched the loads and reported os metrics (iostat) perfectly. I actually made a Grafana dashboard[0] for it, but haven’t used this in a while myself. [0]: https://grafana.com/grafana/dashboards/11801-i-o-statistics/ https://grafana.com/grafana/dashboards/11801-i-o-statistics/

nsguy 2y ago
It's all standard Linux metrics that node-exporter exposes. Mostly from procfs.
- louwrentius 2y ago
  I’ve tried to get the same results with telegraf and influxdb and never got the metrics right