7 ms·

Does "Time Series Database" mean anything technical, or is this just some Uber marketing? In statistics, time series has a technical meaning.

idunno246 7y ago
A db that backs lots of time based graphs. They generally hit some pathological cases for general dbs. Frequent write of small data, most recent time is very hot so sharding is tricky, queries generally pull lots of little bits of data in from long time ranges, resolution of past data can often be lowered, etc. Opentsdb for instance was built on top of Hbase because implementing one naively in hbase hits tons of performance issues.
steve_adams_86 7y ago
There many flavours of time series databases, but my understanding is that at their core they're optimised for storing and querying based on time stamp/time series data. Typically very quickly in large volumes. That's maybe the primary criteria to define a database as a time series database. Someone could probably elaborate on this a massive amount. I'm sure there is some nuance and a lot of relevant details around how that optimization is done.
namanaggarwal 7y ago
It's not a marketing term. It's a database optimised for storing and querying time based metrics. Uber didn't invent the term, there are a lot of existing products in market. Question is why none of them worked for them. I have used OpenTSDB and it worked great at Mastercard scale. What issues did Uber had?
- rossjudson 7y ago
  "Mastercard scale" doesn't mean anything in particular, unless you quantify it. How many metrics? Write rate? Query rate? Query complexity?
abvdasker 7y ago
A time series database is specialized for use cases where the data and query patterns are solely temporal in nature and must show the latest data in real-time (performance metrics/monitoring and stock prices come to mind). Relational and NoSQL databases tend to degrade rapidly with these query patterns at scale (think of the complexity of SQL queries to bucket rows by timestamp). https://en.m.wikipedia.org/wiki/Time_series_database https://en.m.wikipedia.org/wiki/Time_series_database
- roskilli 7y ago
  I touch on this a little in the podcast I did with Jeff[0], but it boils down to OpenTSDSB for us when we benchmarked could only do low tens of thousands of writes per second per node, whereas M3DB is hyper optimized and can do hundreds of thousands to millions of writes per second per node depending on compute/disk. Also with a fast inverted index we were able to achieve much faster query times than OpenTSDB at scale. [0]: https://softwareengineeringdaily.com/2019/08/21/time-series-databases-with-rob-skillington/ https://softwareengineeringdaily.com/2019/08/21/time-series-...
- refset 7y ago
  Note that temporal databases are also a thing, so it's probably wise to avoid using the word "temporal" when discussing time series databases. As far as I know kdb+ is the only technology that has a foot in both camps. https://en.m.wikipedia.org/wiki/Temporal_database https://en.m.wikipedia.org/wiki/Temporal_database
  - CharlesW 7y ago
    > As far as I know kdb+ is the only technology that has a foot in both camps. Teradata Vantage also supports both. And you're absolutely right, it's important not to conflate "temporal" and "time series" support.
bostik 7y ago
TSDBs are a special case of databases. And oh boy, time-series is hard. Your regular RDBMS is going to be either write-heavy or read-heavy. You can pretty easily[ß] optimise the database for one of these utilisation patterns. But a TSDB basically combines the worst of both worlds: telemetry at any scale is important, and monitoring reliability in an always-online system is not optional. TSDBs are written to very frequently; even at a reasonably low scale we could be talking about couple of hundred thousand writes every few seconds. But because they are also used for system-wide monitoring, they are read from all the time. ß: a read-heavy regular DB has the ratio of reads:writes in thousands, perhaps millions; a write-heavy DB can be read from a couple of times every few seconds, but can be written to at a rate of tens of thousands of entries per second. You - or your expensive DBA - can optimise the DB for one of these patterns, but not for both. TSDBs have to support both patterns at the same time, so their internals have been geared to this one specific domain.
- rixed 7y ago
  > But because they are also used for system-wide monitoring, they are read from all the time. And this is the billion dollars mistake of the current devop culture. I believe we are doing monitoring wrong. Real time monitoring need no persistent storage. Troubleshooting does need persistent storage, but not monitoring, and unless your infrastructure is broken all the time then querying past data must occur only rarely. From what i have seen, this mistake seems to stem from the web culture that tends to favor designs centered on a database. Whereas you want your real-time monitoring to be centered on stream processing, with one output to the persistent store for later retrieval. There is no good reason for your dashboards nor your alerts to hit a database every few minutes, this design is just wrong. I'm actually working on the prototype of a stream processor tailored at small scale network monitoring and would welcome any discussion/criticism on this topic. Notice that "small scale" for a stream processor is much larger than "small scale" for a database, and that i believe a good stream processor capable of running arbitrary persistent queries for monitoring on the order of a million data points per second should fit a single server and be more than enough to monitor an above the average sized business infrastructure.
  - 7y ago

idunno246 7y ago

A db that backs lots of time based graphs. They generally hit some pathological cases for general dbs. Frequent write of small data, most recent time is very hot so sharding is tricky, queries generally pull lots of little bits of data in from long time ranges, resolution of past data can often be lowered, etc. Opentsdb for instance was built on top of Hbase because implementing one naively in hbase hits tons of performance issues.

steve_adams_86 7y ago

There many flavours of time series databases, but my understanding is that at their core they're optimised for storing and querying based on time stamp/time series data. Typically very quickly in large volumes. That's maybe the primary criteria to define a database as a time series database. Someone could probably elaborate on this a massive amount. I'm sure there is some nuance and a lot of relevant details around how that optimization is done.

namanaggarwal 7y ago

It's not a marketing term. It's a database optimised for storing and querying time based metrics. Uber didn't invent the term, there are a lot of existing products in market. Question is why none of them worked for them. I have used OpenTSDB and it worked great at Mastercard scale. What issues did Uber had?

rossjudson 7y ago
"Mastercard scale" doesn't mean anything in particular, unless you quantify it. How many metrics? Write rate? Query rate? Query complexity?

abvdasker 7y ago

A time series database is specialized for use cases where the data and query patterns are solely temporal in nature and must show the latest data in real-time (performance metrics/monitoring and stock prices come to mind). Relational and NoSQL databases tend to degrade rapidly with these query patterns at scale (think of the complexity of SQL queries to bucket rows by timestamp). https://en.m.wikipedia.org/wiki/Time_series_database https://en.m.wikipedia.org/wiki/Time_series_database

roskilli 7y ago
I touch on this a little in the podcast I did with Jeff[0], but it boils down to OpenTSDSB for us when we benchmarked could only do low tens of thousands of writes per second per node, whereas M3DB is hyper optimized and can do hundreds of thousands to millions of writes per second per node depending on compute/disk. Also with a fast inverted index we were able to achieve much faster query times than OpenTSDB at scale. [0]: https://softwareengineeringdaily.com/2019/08/21/time-series-databases-with-rob-skillington/ https://softwareengineeringdaily.com/2019/08/21/time-series-...
refset 7y ago
Note that temporal databases are also a thing, so it's probably wise to avoid using the word "temporal" when discussing time series databases. As far as I know kdb+ is the only technology that has a foot in both camps. https://en.m.wikipedia.org/wiki/Temporal_database https://en.m.wikipedia.org/wiki/Temporal_database
- CharlesW 7y ago
  > As far as I know kdb+ is the only technology that has a foot in both camps. Teradata Vantage also supports both. And you're absolutely right, it's important not to conflate "temporal" and "time series" support.

bostik 7y ago

TSDBs are a special case of databases. And oh boy, time-series is hard. Your regular RDBMS is going to be either write-heavy or read-heavy. You can pretty easily[ß] optimise the database for one of these utilisation patterns. But a TSDB basically combines the worst of both worlds: telemetry at any scale is important, and monitoring reliability in an always-online system is not optional. TSDBs are written to very frequently; even at a reasonably low scale we could be talking about couple of hundred thousand writes every few seconds. But because they are also used for system-wide monitoring, they are read from all the time. ß: a read-heavy regular DB has the ratio of reads:writes in thousands, perhaps millions; a write-heavy DB can be read from a couple of times every few seconds, but can be written to at a rate of tens of thousands of entries per second. You - or your expensive DBA - can optimise the DB for one of these patterns, but not for both. TSDBs have to support both patterns at the same time, so their internals have been geared to this one specific domain.

rixed 7y ago
> But because they are also used for system-wide monitoring, they are read from all the time. And this is the billion dollars mistake of the current devop culture. I believe we are doing monitoring wrong. Real time monitoring need no persistent storage. Troubleshooting does need persistent storage, but not monitoring, and unless your infrastructure is broken all the time then querying past data must occur only rarely. From what i have seen, this mistake seems to stem from the web culture that tends to favor designs centered on a database. Whereas you want your real-time monitoring to be centered on stream processing, with one output to the persistent store for later retrieval. There is no good reason for your dashboards nor your alerts to hit a database every few minutes, this design is just wrong. I'm actually working on the prototype of a stream processor tailored at small scale network monitoring and would welcome any discussion/criticism on this topic. Notice that "small scale" for a stream processor is much larger than "small scale" for a database, and that i believe a good stream processor capable of running arbitrary persistent queries for monitoring on the order of a million data points per second should fit a single server and be more than enough to monitor an above the average sized business infrastructure.
- 7y ago