3 ms·
Hi @georgewfraser: We understand this point, and you aren't the first to mention it on HN =) But our users, and the users of most so-called "time series databa
by mfreed 7y ago
Hi @georgewfraser: We understand this point, and you aren't the first to mention it on HN =)
But our users, and the users of most so-called "time series databases", do not typically generate the classic scan-oriented workloads of OLAP systems that motivated C-Store and classic columnar data warehouses.
I talk about this in the post: sure, there are deep and narrow queries, but there are also a /lot/ of shallow and wide queries. Most real-time dashboarding, which is a huge use-case for IT/devops monitoring and IoT sensor monitoring, is about "what happened very recently".
But you do occasionally want to look back, so you want to keep old data sitting around, just in a cost-effective manner. And there is a lot of data: time-series and monitoring applications are typically write (not read) heavy, and ingest high volumes of data continuously.
Enter compression and data tiering, both of which appeared in today's v1.5 release.
As such, the primary motivation behind our work on compression was actually saving storage overhead, not improving query performance. Although we'll never pass up on performance improvements when we get them! =)
(And we are familiar with the citation -- I've known Daniel and Sam for many years, and one of the engineers who led this work at Timescale was actually Daniel's former PhD advisee Josh Lockerman.)
- georgewfraser 7y agoThank you for responding. The “shallow and wide” query you describe is not a challenge for any recently built column store, which will partition the data into blocks of 10-100 MB. So when you run a query like: select * from time_series where ingest_time > ? You will only scan the most recent block, and it will be extremely fast. I have no doubt that there are individual queries where a time-series database will “win”, but I earnestly question whether there’s a real-world use case, with a mix of queries, where a special purpose time-series db is preferable to a good column store with time-based partitioning.