4 ms·

> But because they are also used for system-wide monitoring, they are read from all the time. And this is the billion dollars mistake of the current devop cult

by rixed 7y ago

> But because they are also used for system-wide monitoring, they are read from all the time. And this is the billion dollars mistake of the current devop culture. I believe we are doing monitoring wrong. Real time monitoring need no persistent storage. Troubleshooting does need persistent storage, but not monitoring, and unless your infrastructure is broken all the time then querying past data must occur only rarely. From what i have seen, this mistake seems to stem from the web culture that tends to favor designs centered on a database. Whereas you want your real-time monitoring to be centered on stream processing, with one output to the persistent store for later retrieval. There is no good reason for your dashboards nor your alerts to hit a database every few minutes, this design is just wrong. I'm actually working on the prototype of a stream processor tailored at small scale network monitoring and would welcome any discussion/criticism on this topic. Notice that "small scale" for a stream processor is much larger than "small scale" for a database, and that i believe a good stream processor capable of running arbitrary persistent queries for monitoring on the order of a million data points per second should fit a single server and be more than enough to monitor an above the average sized business infrastructure.

closeparen 7y ago
When you have very high cardinality series, it's decently common to slot in a "rollup" or "reducer" query between collection and persistence. This is similar to the insight you mentioned: I don't actually need to store a separate series per hostname in perpetuity just to know when the worst one is out of control. I can ask it to persist the top five individually, and then an average. But when responding to a PagerDuty alert, the first thing I want to see is a plot of the alerting series. The next thing I want to see is a plot of every series about that service (bonus points if you can find the ones that have discontinuities with similar timing). So you're really not getting out of the "fast serving" business, just reducing the load on it.
- rixed 7y ago
  > But when responding to a PagerDuty alert, the first thing I want to see is a plot of the alerting series. When you do receive a page, you can then hit a database. This happens rarely enough (hopefully!) so that your database can be optimised for writes only, not for writes and reads, as $parent suggested. I would even argue that this is a similar use case than dashboarding: when paged you want to see the last 3 days or so of accurate data + some longer term averages/trends/baseline. All of this could fit in RAM. At least in the current prototype I mentioned above the ambition is to be able to serve recent data for such dashboards out of RAM without bothering with a DB, and have the DB completely out of the way for the monitoring use case (while still having a persistent store for long term capacity planning / business analyses). Assuming doubles are compressed and a collection rate of 1/10s, 3 days of data is <100KiB per timeseries, ie 0.5GiB for 10k source timeseries.

by rixed 7y ago

closeparen 7y ago

When you have very high cardinality series, it's decently common to slot in a "rollup" or "reducer" query between collection and persistence. This is similar to the insight you mentioned: I don't actually need to store a separate series per hostname in perpetuity just to know when the worst one is out of control. I can ask it to persist the top five individually, and then an average. But when responding to a PagerDuty alert, the first thing I want to see is a plot of the alerting series. The next thing I want to see is a plot of every series about that service (bonus points if you can find the ones that have discontinuities with similar timing). So you're really not getting out of the "fast serving" business, just reducing the load on it.

rixed 7y ago
> But when responding to a PagerDuty alert, the first thing I want to see is a plot of the alerting series. When you do receive a page, you can then hit a database. This happens rarely enough (hopefully!) so that your database can be optimised for writes only, not for writes and reads, as $parent suggested. I would even argue that this is a similar use case than dashboarding: when paged you want to see the last 3 days or so of accurate data + some longer term averages/trends/baseline. All of this could fit in RAM. At least in the current prototype I mentioned above the ambition is to be able to serve recent data for such dashboards out of RAM without bothering with a DB, and have the DB completely out of the way for the monitoring use case (while still having a persistent store for long term capacity planning / business analyses). Assuming doubles are compressed and a collection rate of 1/10s, 3 days of data is <100KiB per timeseries, ie 0.5GiB for 10k source timeseries.