4 ms·

I've thought about creating this about 5 times; the existing services really haven't gotten it quite right, IMO. For me, exactly what metrics to measure are not

by bhuga 16y ago

I've thought about creating this about 5 times; the existing services really haven't gotten it quite right, IMO. For me, exactly what metrics to measure are not as important as the poor correlation tools. The usual suspects are fine for 'easy' and the non-usual suspects will always require custom scripts. More important for me is letting go of 'server stats'. The existing tools you mention are stuck in pre-cloud days, assuming that I care about a graph for a particular server as opposed to the average/95th %tile/99 %tile across a group of servers filling a role. The best solution I've found is OpenTSDB, which lets me tag any data point, which lets me tag web server CPU use metrics as 'web' and not worry about when machines go up and down when I'm looking at web server CPU use. But OpenTSDB is a pain to setup and if someone jazzed up the graphs and made it a service, I'd buy it.

Skywing 16y ago
Correlation is also why we started this. A little story: we were using Cacti on a cluster of ours. I was showing a friend of mine the graphs, only because I was explaining our infrastructure setup. I noticed one had unusually high CPU load. The server with the high load was one of our SQL database servers. I SSHd into it and sure enough it was MySQL causing the load. But ... why? So, after so log file browsing, we found that our site was being scraped by some bots, and the particular page being scraped was doing a sorted query on a small range of data, which was causing the entire table to be sorted then scanned, which was causing high CPU usage, we thought. So, the correlation there between high CPU usage and the cause was less direct, and a simple graph was only enough to alert me of the fact that there was a problem on that machine. Plus, I wasnt even alerted - I stumbled upon it. So, this is something we're aiming to fix.
- bhuga 16y ago
  If you can correlate semantically-parsed log data with time series to automatically understand that particular problem, you're solving problems harder than server monitoring and applying them to server monitoring, IMO :) Not that there's anything wrong with that!

by bhuga 16y ago

Skywing 16y ago

Correlation is also why we started this. A little story: we were using Cacti on a cluster of ours. I was showing a friend of mine the graphs, only because I was explaining our infrastructure setup. I noticed one had unusually high CPU load. The server with the high load was one of our SQL database servers. I SSHd into it and sure enough it was MySQL causing the load. But ... why? So, after so log file browsing, we found that our site was being scraped by some bots, and the particular page being scraped was doing a sorted query on a small range of data, which was causing the entire table to be sorted then scanned, which was causing high CPU usage, we thought. So, the correlation there between high CPU usage and the cause was less direct, and a simple graph was only enough to alert me of the fact that there was a problem on that machine. Plus, I wasnt even alerted - I stumbled upon it. So, this is something we're aiming to fix.

bhuga 16y ago
If you can correlate semantically-parsed log data with time series to automatically understand that particular problem, you're solving problems harder than server monitoring and applying them to server monitoring, IMO :) Not that there's anything wrong with that!