3 ms·
How many dimensions does each metric support?
by otterley 10y ago
How many dimensions does each metric support?
- RobAtticus 10y agoSorry, not sure I follow. Do you mean how many partitions are supported? We allow up to two dimensions of partitioning: we always partition by time, and we allow you to specify another key on which to partition. Allowing multiple keys to partition by is probably far down the road. As for the number of partitions on that non-time key: I don't think we have a limit (or its something like 32,000). Not sure if that answered your question.
- mfreed 10y agoNot sure if this is what you're asking, but some monitoring focused systems have you associate each individual time-series metric with a bunch of metadata/labels (either encoding into a single string ("dc1.rack15.server46.port2") or in some json/array. We expose a more traditional postgres schema, which is a different way of thinking about the data model if you come from a monitoring world. So you can shove data into a postgres JSONB or array type if you want, but you might more commonly have all your data items stored as individual columns. And different/many metrics can be stored in a single row, so (1) you aren't paying for the label cost with each metric, (2) our query planner makes it efficient to support complex predicates on these metrics. Like: SELECT date_trunc('minute', time) AS minute, COUNT(device_id) FROM logs WHERE cpu_level > 0.9 AND free_mem < 1024 AND time > now() - interval '24 hours' GROUP BY minute ORDER BY COUNT(device_id) DESC LIMIT 25;
- otterley 10y agoThe real power of time series databases comes from the ability to aggregate metrics based on arbitrary dimensions. For example, to create a dashboard containing stack graphs of http requests summed across all servers of a particular application name, grouped by response code, one for every application name, then separately sectioned in the dashboard by region - but only for the servers in the production environment. And none of the dimension keys or values are known to the database in advance. Datadog, SignalFx, Prometheus, and Wavefront all have this ability. It's now standard functionality for any reasonably advanced time series monitoring implementation. The fact that your implementation happens to be backed by Postgres is interesting, but it's merely an implementation detail as far as power users are concerned.
- mfreed 10y agoYou can query by arbitrary dimension (what we would think of as time aggregates). In the future, you'd be able to materialize to arbitrary dimensions with continuous queries.
- otterley 10y agoHow do you support arbitrary dimensions in a tabular database system without massively blowing up the number of tables, the number of columns, or number of sparse rows? This kind of problem is really an OLAP cube problem, yet you're throwing an OLTP solution at it. I've seen it tried before at several sites and it's been abandoned each time for having crippling performance and management behaviors. What are you doing differently that will make this attempt successful?
- deleted 10y ago[deleted]