5 ms·

I think what OP was getting at is why it has to be specialized. Time-series features can be implemented in a general purpose DB engine like postgres or mongo.

nonameiguess 3y ago
How? I just tried to look for time series in Postgres and it looks like TimescaleDB is the plugin you'd want? At least web search seems to love it. Their landing page is just marketing fluff and says next to nothing about their feature set. In a database that isn't column-oriented, how can they do automatic downsampling? Or do they just not?
- hnmullany 3y ago
  You might want to check out their columnar compression & their continuous aggregate blog posts: https://www.timescale.com/blog/timescaledb-2-3-improving-columnar-compression-for-time-series-on-postgresql/ https://www.timescale.com/blog/timescaledb-2-3-improving-col... https://www.timescale.com/blog/massive-scale-for-time-series-workloads-introducing-continuous-aggregates-for-distributed-hypertables-in-timescaledb-2-5/ https://www.timescale.com/blog/massive-scale-for-time-series...
  - lordnacho 3y ago
    Have used it for finance, can recommend. No need for kdb, at least for now. Their Slack is very responsive too, top service.
marginalia_nu 3y ago
They kinda can't though, not without losing a lot of performance and capability. Time series databases should be viewed in the context of being a fairly niche product however. The archetypal use case is in finance where you have a constant firehose of tick data and trades across a wide array of instruments you need to be able to index instantaneously and aggregate easily. The sheer volume of data can't be overstated here. To make matters worse, the records are often both wide and sparsely populated. A non-specialized really DBMS doesn't cut it in this case. You need to build the DBMS from the ground up to cater to this sort of a usecase. Slapping a columnar backing store onto something existing does not cut it.
nhourcard 3y ago
Time-series features can be implemented in a general purpose DB engine, but the architecture of the database will be limiting for lots of use cases requiring performance, especially for high cardinality datasets. A columnar database engineered from the ground up for time-series should have better foundations to allows fast ingest (to the tune of million of rows/sec per server) and also be very efficient for time-based queries that can be done via languages such as SQL. Using QuestDB as an example: Data is stored in chronological order and is optimized for sequential ingestion with a timestamp component (re-ordering data on the fly if it comes out-of-order). The data is also partitioned by time. The InfluxDB Line Protocol is better suited for streaming type of ingest versus transactional inserts via Postgres.

nonameiguess 3y ago

How? I just tried to look for time series in Postgres and it looks like TimescaleDB is the plugin you'd want? At least web search seems to love it. Their landing page is just marketing fluff and says next to nothing about their feature set. In a database that isn't column-oriented, how can they do automatic downsampling? Or do they just not?

hnmullany 3y ago
You might want to check out their columnar compression & their continuous aggregate blog posts: https://www.timescale.com/blog/timescaledb-2-3-improving-columnar-compression-for-time-series-on-postgresql/ https://www.timescale.com/blog/timescaledb-2-3-improving-col... https://www.timescale.com/blog/massive-scale-for-time-series-workloads-introducing-continuous-aggregates-for-distributed-hypertables-in-timescaledb-2-5/ https://www.timescale.com/blog/massive-scale-for-time-series...
- lordnacho 3y ago
  Have used it for finance, can recommend. No need for kdb, at least for now. Their Slack is very responsive too, top service.

marginalia_nu 3y ago

They kinda can't though, not without losing a lot of performance and capability. Time series databases should be viewed in the context of being a fairly niche product however. The archetypal use case is in finance where you have a constant firehose of tick data and trades across a wide array of instruments you need to be able to index instantaneously and aggregate easily. The sheer volume of data can't be overstated here. To make matters worse, the records are often both wide and sparsely populated. A non-specialized really DBMS doesn't cut it in this case. You need to build the DBMS from the ground up to cater to this sort of a usecase. Slapping a columnar backing store onto something existing does not cut it.

nhourcard 3y ago

Time-series features can be implemented in a general purpose DB engine, but the architecture of the database will be limiting for lots of use cases requiring performance, especially for high cardinality datasets. A columnar database engineered from the ground up for time-series should have better foundations to allows fast ingest (to the tune of million of rows/sec per server) and also be very efficient for time-based queries that can be done via languages such as SQL. Using QuestDB as an example: Data is stored in chronological order and is optimized for sequential ingestion with a timestamp component (re-ordering data on the fly if it comes out-of-order). The data is also partitioned by time. The InfluxDB Line Protocol is better suited for streaming type of ingest versus transactional inserts via Postgres.