3 ms·

How does it compare to other time series db? I've used influxdb, prometheus, graphite, Elasticsearch and even Splunk. I mean in terms of complexity, manageabil

How does it compare to other time series db? I've used influxdb, prometheus, graphite, Elasticsearch and even Splunk. I mean in terms of complexity, manageability, performance, scalability...

jmngomes 9y ago
Also curious about this, and would add OpenTSDB to that list.
- akulkarni 9y ago
  At a high-level, we've taken a unique approach to the time-series data problem. While every other time-series db has taken a non-relational approach (which also means no or poor SQL support), we've built ours on Postgres. As a result, not only do we support full SQL (JOINs, complex WHEREs/GROUPBYs, window functions, etc), but we also inherit the reliability and vast ecosystem of Postgres. At a more detailed level, comparisons between TimescaleDB and others varies depending on the database. Typically though, if you are most concerned with data compression or simple column scans, some of the other options might be more suitable than ours. But otherwise you should see similar (if not better) insert and query performance, while maintaining all the other benefits of a relational DB / Postgres. And with your relational data living alongside time-series data, many people have been able to simplify their stack (from multiple dbs into one db), which makes things easier to manage operationally and at the application level. Re: OpenTSDB, we hear that managing an HDFS cluster / Zookeeper / etc can be a headache. You obviously don't have that problem with TimescaleDB. Re: Prometheus, I should add that we've open-sourced `pg_prometheus` [1], which allows you to write Prometheus data into Postgres / TimescaleDB. So with regards to Prometheus, we are more of a complement than a replacement. [1] https://github.com/timescale/pg_prometheus https://github.com/timescale/pg_prometheus
assface 9y ago
See CMU's Time-Series Database Lectures from last year: http://db.cs.cmu.edu/seminar2017/ http://db.cs.cmu.edu/seminar2017/
LogicX 9y ago
We were an early adopter of InfluxDB and ran it for almost two years. Some of our challenges: Not being able to change your DB structure (like indexed columns) after the fact. Indexes on very unique data blowing up the DB (OOM, slowdown, still even after they ‘fixed’ the cardinality problem. Performance writing data while also querying against it for analytics. We solved all these problems with timescale, use far fewer resources, are a long way away from needing a cluster, have lots of room for growth and development had sped up with devs who are already familiar with postgresql and can easily make changes.
camel_gopher 9y ago
Fred from IRONdb (http://irondb.io http://irondb.io) here. First off, it's worth mentioning that "scalability" is a very relative term. Some people need to process a billion measurements per second.. there are very few databases out there that can handle that sort of task. If you're only idling along at something small that a single machine can easily handle (like a million data points per second), then the product doesn't need to scale very much at all. At that point you'd be looking for operability and manageability... how does it behave when everything else in the world aims to ruin your day? When we started scaling Circonus, we realized we needed to solve this billion measurements per second problem on a system that scales near-linear with added nodes and has manageable failure and recovery scenarios... if we didn't solve that, we'd be worried about data all the time instead of building value on data. That's IRONdb. What all of these TSDBs you mentioned have in common is that they tend to handle increasing workloads by just throwing a bigger box at the problem, but run into technical challenges when adding more nodes. How do they ensure data consistency and availability when multiple nodes are involved? Most of them accomplish this via consensus algorithms (paxos/raft/etc). This introduces a number of technical problems that readers of the 'Call Me Maybe' blog series by Aphyr (https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen) are familiar with. Essentially you end up with data inconsistency and corruption at high workloads to some degree for almost all of them, and sacrifice performance for consensus. At serious scale with petabytes of data, how does your TSDB detect and recover from the inevitable fallibility of disks (bit error rates being non-zero)? These are manageability questions that you run into at scale. This was an issue we recognized was best avoided when we designed IRONdb, so we approached the problem by avoiding the need for consensus altogether through use of commutative operations. This means that it doesn't matter what order the end time series data is applied in; the result is the same. It allowed us to focus our efforts on read vs write performance, operational simplicity and the other points you mentioned. I gave a high level technical talk on these issues a few months ago, https://www.youtube.com/watch?v=QBatpIFii7M https://www.youtube.com/watch?v=QBatpIFii7M As always, I'd recommend taking each for a spin and seeing for yourself. None are perfect and each will surprise and disappoint you in unique ways when you use it in anger. This coming from someone with ample anger. Disclaimer; I've been a Postgres user for 18 years.

How does it compare to other time series db? I've used influxdb, prometheus, graphite, Elasticsearch and even Splunk. I mean in terms of complexity, manageability, performance, scalability...

jmngomes 9y ago

Also curious about this, and would add OpenTSDB to that list.

akulkarni 9y ago
At a high-level, we've taken a unique approach to the time-series data problem. While every other time-series db has taken a non-relational approach (which also means no or poor SQL support), we've built ours on Postgres. As a result, not only do we support full SQL (JOINs, complex WHEREs/GROUPBYs, window functions, etc), but we also inherit the reliability and vast ecosystem of Postgres. At a more detailed level, comparisons between TimescaleDB and others varies depending on the database. Typically though, if you are most concerned with data compression or simple column scans, some of the other options might be more suitable than ours. But otherwise you should see similar (if not better) insert and query performance, while maintaining all the other benefits of a relational DB / Postgres. And with your relational data living alongside time-series data, many people have been able to simplify their stack (from multiple dbs into one db), which makes things easier to manage operationally and at the application level. Re: OpenTSDB, we hear that managing an HDFS cluster / Zookeeper / etc can be a headache. You obviously don't have that problem with TimescaleDB. Re: Prometheus, I should add that we've open-sourced `pg_prometheus` [1], which allows you to write Prometheus data into Postgres / TimescaleDB. So with regards to Prometheus, we are more of a complement than a replacement. [1] https://github.com/timescale/pg_prometheus https://github.com/timescale/pg_prometheus

assface 9y ago

See CMU's Time-Series Database Lectures from last year: http://db.cs.cmu.edu/seminar2017/ http://db.cs.cmu.edu/seminar2017/

LogicX 9y ago

We were an early adopter of InfluxDB and ran it for almost two years. Some of our challenges: Not being able to change your DB structure (like indexed columns) after the fact. Indexes on very unique data blowing up the DB (OOM, slowdown, still even after they ‘fixed’ the cardinality problem. Performance writing data while also querying against it for analytics. We solved all these problems with timescale, use far fewer resources, are a long way away from needing a cluster, have lots of room for growth and development had sped up with devs who are already familiar with postgresql and can easily make changes.

camel_gopher 9y ago

Fred from IRONdb (http://irondb.io http://irondb.io) here. First off, it's worth mentioning that "scalability" is a very relative term. Some people need to process a billion measurements per second.. there are very few databases out there that can handle that sort of task. If you're only idling along at something small that a single machine can easily handle (like a million data points per second), then the product doesn't need to scale very much at all. At that point you'd be looking for operability and manageability... how does it behave when everything else in the world aims to ruin your day? When we started scaling Circonus, we realized we needed to solve this billion measurements per second problem on a system that scales near-linear with added nodes and has manageable failure and recovery scenarios... if we didn't solve that, we'd be worried about data all the time instead of building value on data. That's IRONdb. What all of these TSDBs you mentioned have in common is that they tend to handle increasing workloads by just throwing a bigger box at the problem, but run into technical challenges when adding more nodes. How do they ensure data consistency and availability when multiple nodes are involved? Most of them accomplish this via consensus algorithms (paxos/raft/etc). This introduces a number of technical problems that readers of the 'Call Me Maybe' blog series by Aphyr (https://aphyr.com/tags/jepsen https://aphyr.com/tags/jepsen) are familiar with. Essentially you end up with data inconsistency and corruption at high workloads to some degree for almost all of them, and sacrifice performance for consensus. At serious scale with petabytes of data, how does your TSDB detect and recover from the inevitable fallibility of disks (bit error rates being non-zero)? These are manageability questions that you run into at scale. This was an issue we recognized was best avoided when we designed IRONdb, so we approached the problem by avoiding the need for consensus altogether through use of commutative operations. This means that it doesn't matter what order the end time series data is applied in; the result is the same. It allowed us to focus our efforts on read vs write performance, operational simplicity and the other points you mentioned. I gave a high level technical talk on these issues a few months ago, https://www.youtube.com/watch?v=QBatpIFii7M https://www.youtube.com/watch?v=QBatpIFii7M As always, I'd recommend taking each for a spin and seeing for yourself. None are perfect and each will surprise and disappoint you in unique ways when you use it in anger. This coming from someone with ample anger. Disclaimer; I've been a Postgres user for 18 years.