3 ms·

Logs are very much not a time series so storing them in a time series database is not something that I would recommend. Time series are essentially a sequence o

by daenney 10y ago

Logs are very much not a time series so storing them in a time series database is not something that I would recommend. Time series are essentially a sequence of numeric data points in chronological order. Though events are usually in chronological order, they're not numeric data points (though they can contain that too). As a consequence of being two different things, storing and analysing them efficiently need different solutions. I've seen people stuff everything in Elastic Search and it's certainly possible if you really want to.

bbrazil 10y ago
A time series database is anything that stores data with time attached, so log storage does count. The more pertinent difference is event logging vs. metrics. For the former the ELK stack is popular, and Prometheus.io is suitable for the latter.
- eis 10y ago
  Prometheus unfortunately has limited long term storage support and it's explicitly not one of their goals. Their goal is "operational monitoring", not analytics. I found that by trying to see if it could be used as a metrics DB but stumbled upon a few issues like configuring retention times per target. See for example https://github.com/prometheus/prometheus/issues/1381 https://github.com/prometheus/prometheus/issues/1381 Prometheus has lots of potential but it's not a metrics DB at this point or in the near future. I just wish they'd made that a bit clearer on the page. "We make design decisions that presume that Promtheus data is ephemeral, and can be lost/blown away with no impact." That viewpoint pretty much limits them to be an ops tool. I wish they'd reconsider this point.
Shish2k 10y ago
> Though events are usually in chronological order, they're not numeric data points (though they can contain that too) I think the thing is that the primary use-case for a lot of people combines the two - eg they want to log a whole key:value map where values can be numeric or string, and then dynamically generate time series. We can do this with SQL, like if I want a graph of traffic per domain name, "SELECT time_bucket, domain_name, sum(bytes) FROM access_log GROUP BY time_bucket, domain_name" - but all SQL servers are totally non-optimised for this use case. Surely there must be some software out there which can do this efficiently?

by daenney 10y ago

bbrazil 10y ago

A time series database is anything that stores data with time attached, so log storage does count. The more pertinent difference is event logging vs. metrics. For the former the ELK stack is popular, and Prometheus.io is suitable for the latter.

eis 10y ago
Prometheus unfortunately has limited long term storage support and it's explicitly not one of their goals. Their goal is "operational monitoring", not analytics. I found that by trying to see if it could be used as a metrics DB but stumbled upon a few issues like configuring retention times per target. See for example https://github.com/prometheus/prometheus/issues/1381 https://github.com/prometheus/prometheus/issues/1381 Prometheus has lots of potential but it's not a metrics DB at this point or in the near future. I just wish they'd made that a bit clearer on the page. "We make design decisions that presume that Promtheus data is ephemeral, and can be lost/blown away with no impact." That viewpoint pretty much limits them to be an ops tool. I wish they'd reconsider this point.

Shish2k 10y ago

> Though events are usually in chronological order, they're not numeric data points (though they can contain that too) I think the thing is that the primary use-case for a lot of people combines the two - eg they want to log a whole key:value map where values can be numeric or string, and then dynamically generate time series. We can do this with SQL, like if I want a graph of traffic per domain name, "SELECT time_bucket, domain_name, sum(bytes) FROM access_log GROUP BY time_bucket, domain_name" - but all SQL servers are totally non-optimised for this use case. Surely there must be some software out there which can do this efficiently?