9 ms·

How Time Series Databases Work, and Where They Don’t

228 points by telotortium 5y ago

OmarIsmail 5y ago
These articles are so good. Very informative for a product engineer such as myself. Allows me to create a much clearer and more expansive mental model of the running systems.
- thor24 5y ago
  +1 Read their other articles on the blog as well. Very informative.
- cgearhart 5y ago
  It’s hard for me to rate the content. I couldn’t get past the sketchy dark pattern of a site that tracks me and sells my data for marketing with an “accept all cookies” button, while making it much harder to find the process for opting out. At that point it’s easier for me to just opt out of using the site. It’s too bad because I’d love to read some good technical articles, and this sounds like an interesting topic.
  - dominotw 5y ago
    > sells my data for marketing with an “accept all cookies” button most likely just poorly implemented gdpr.
    - bostik 5y ago
      I used to wear the DPO hat, so this comes from a vantage point of having been in the weeds: ignorance is no excuse for incompetence. "Doing GDPR" badly because you don't know any better? Then don't even try to walk that tightrope. Don't track. At all. If you use on third parties who don't give you the option to do it right, then dump those third parties.
  - rendall 5y ago
    Agreed. I open nearly every HN article from an unknown domain in incognito/private mode. At least half implement this kind of dark pattern. I find it spectacularly baffling for this kind of content-marketing website, where income presumably does not come from advertising, nor trading in visitor data, but from their DBaaS. Why would such a company risk alienating their visitors like this? It makes me question the decision-making in their unrelated, primary service; their conscientiousness; even their security. Truly mystifying.
    - s_fischer 5y ago
      I think there's a universe where there is no overlap between engineers/PMs who work on the actual DB product and website/content marketing site. The marketing space is filled with all kinds of "plug and play" SaaS providers which offer detailed customer journey data and sometimes it's just straight up easier to add an "accept all" consent banner than to try and allow for hot loading specific 3rd party libraries based on customized consent options. Is it the right thing to do? In my opinion, no. But I can also understand a situation where decisions were made on marketing tech before understanding the technical privacy implications. And then the implementation is handled by a team (potentially much smaller) that does not work on the actual product.
      - rendall 5y ago
        Sure. I believe I understand your explanation and appreciate it. It's possible for their core service to be rock solid and their marketing side to be of lesser priority and so not to receive the thought and resources of their core service. It makes sense. Still, were I to be responsible for evaluating competing services, such choices would definitely be a ding. Not unrecoverable, but it would make me wonder unnecessarily about their corporate culture and customer care. All things being equal, I think it would be wise to go with the service that didn't do that.
- neilsense 5y ago
  I wouldn't take this as the one truth, they are describing their database more than they are describing all timeseries databases. Just like MySQL and Postgres can behave wildly differently depending on use, so can timeseriesdbs, there is no one truth here.
proddata 5y ago
That article takes various concepts from typical TSDB solutions and seemingly only looks at the bad sides. Time series data has many different forms, not every form works for every TSDB solution. For the 3 caveats at the top, there are already two TS solutions that look promising (QuestDB, TimescaleDB). Often an operational analytics DB (Clickhouse, CrateDB) might also be a solution.
- camel_gopher 5y ago
  Take a look at IronDB too. High scale distributed implementationd.
- jpgvm 5y ago
  If you want something like Honeycomb but scales better then maybe look at Druid.
  - dikei 5y ago
    Last time I checked, Druid were not very good at ad-hoc tasks because it lacked join and SQL supports was sketchy. How is it now ?
    - jpgvm 5y ago
      Limited JOIN support. SQL is now very good. JOINs vs no JOINs isn't an adhoc vs not-adhoc thing but more of a schema thing. If you try jam a star schema into it you aren't going to have a good time. This is true for pretty all of these more optimised stores. If you have a star schema and want to do those sorts of queries (and performance or cost aren't your #1 driving factors) then the better tool is a traditional data warehouse like BigQuery. This probably won't be the case forever though, there is significant progress in the Presto/Trino world to enable push-down for stores like Druid which would allow you to source most of your fact table data from other sources and then join into your events/time-orientated data from Druid very efficiently.
- oconnore 5y ago
  > Often an operational analytics DB (Clickhouse, CrateDB) might also be a solution This might be a bit off topic, but speaking of gaps in common observability tooling: is an OLAP database a common go-to for longer-timescale analytics (as in [1])? We're using BigQuery, but on ~600GB of log/event data I start hitting memory limits even with fairly small analytical windows. In this context I have seen other references to: Sawzall (google), Lingo (google), MapReduce/Pig/Cascading/Scalding. Are people using Spark for this sort of thing now? Perhaps a combined workflow would be ideal: filter/group/extract interesting data in Hadoop/Spark, and then load into OLAP for ad-hoc querying? [1]: https://danluu.com/metrics-analytics/ https://danluu.com/metrics-analytics/
tkzed49 5y ago
Pointless aside but that's probably the most annoying cookie banner I've seen. Your options are to accept the cookies or read their privacy policy, which is also covered by the cookie banner.
jldugger 5y ago
I'm really struggling to understand the marketing from Honeycomb. They have a lot of reputable and smart staff, seemingly making conflicting public statements. For example, in this post, the author describes how expensive cardinality is for TSDB, but doesn't really address how Honeycomb's column store implementation efficiently stores data with full cardinality anywhere near as cheaply. Like, I fully appreciate how having traces and structured event logs will improve observability, but this 'actually, Prometheus is expensive too' angle is weird, and their pricing page's 'starts at $100' doesn't really make it easy to compare.
- jldugger 5y ago
  Thinking about this a bit more, there's really two factors you have to balance when setting these systems up: event volume and cardinality. It feels like Honeycomb basically has you pre-pay for cardinality. Every field you add is equally efficient at storing data regardless of type or possible values (well, probably table compression helps some), and the event size is not a factor in billing(?). Since there's no pre-calculated aggregation, you bear the full cost of every event.
nivertech 5y ago
Cardinality is only expensive if the metric dimensions/tags/labels are a part of the builtin metric data model (like for example in InfluxDB). It's not expensive when dimensions are just columns in the custom table schema, like for example in TimescaleDB or kdb+/q. Also in the latter case a single database "row" can contain multiple metrics (each modeled as a separate column in a table).
- hardwaresofton 5y ago
  > It's not expensive when dimensions are just columns in the custom table schema, like for example in TimescaleDB or kdb+/q. kdb+/q (not exactly sure) is the one that is used heavily in finance, correct? [EDIT] - Yup[0][1]. I'm not sure how many competitors there are but last time I heard about kdb it seemed like they were pretty much the go-to choice outside comp sci land (which is normally preoccupied with influx, prometheus, timescale, etc). There's probably a reason it's trusted by those other fields. [0]: https://kx.com/ https://kx.com/ [1]: https://code.kx.com/home/ https://code.kx.com/home/
  - liveoneggs 5y ago
    kdb is crazy expensive which is why storing your 5-min load average inside of it doesn't make much sense
  - nivertech 5y ago
    kdb+/q was originally heavily used in finance/trading, but nowadays also in IoT/IIoT. The q language itself is pretty-much general purpose PL, but of course it shines for timeseries processing workloads.
- claytonjy 5y ago
  This is true, but given this post and the one linked about their custom column store, I don't think Timescale would fit. Timescale is still row-oriented while in memory (compressed chunks are columnar), though that may not be a big deal in practice; the bigger deal might be the ability to arbitrarily add columns. With Timescale you either have to issue DDL to add a column (which can be expensive and doesn't work in some cases), or resign yourself to JSONB. My worry with the JSONB is that it'll be expensive to search w/o indices, but if you need indices that goes against the Honeycomb philosophy. Very curious to hear if folks more deeply familiar with TimescaleDB internals have a different take!
gilbetron 5y ago
This is a very narrow definition of a time series DB, really more of a pure metrics store that requires individual time series to be kept separately. I find it odd, as I wouldn't define time series DB, in general, to be what they are. Rather, a time series DB is usually simple some sort of DB that supports a time column as the main index. There's tons that fall in that category (including Postgres, which you should absolutely use unless you have a compelling reason not to) that can easily handle many of the situations the article claims won't work. Druid can handle high-cardinality, for instance.
- digbybk 5y ago
  Can you talk more about why you should use Postgres unless you have a compelling reason not to? I'm currently investigating options for storing high-frequency data. Postgres is looking like a good option.
  - gilbetron 5y ago
    Postgres has a lot of engineering work put into it to handle all kinds of use cases, plus massive effort by the community building tutorials, videos, wrappers, libraries, etc. If you have a problem with Postgres config, or are trying to get it to do something odd, there is undoubtedly a bunch of Stack Overflow discussions about that thing. For most other databases, the selection of all of those is much thinner. Additionally, often we believe our application needs feature X, and there is some database tech that purports to excel at X, the fact is postgres is probably able to do X unless you get to some extreme velocity or volume. Furthermore, by going with another database you are almost always giving up on features W, Y, and Z that you don't realize you need that Postgres supports and isn't supported by the "Exotic" database you are thinking of. In short, Postgres has amazing breadth and depth in features and support and tooling. Be sure you are ok giving that up!
    - akulkarni 5y ago
      If you like Postgres, you may want to try TimescaleDB, which is a time-series database built on Postgres (packaged as a Postgres extension). Postgres database + time-series database all in one. This btw is one of the reasons I love Postgres - its extensibility. (Disclaimer: TimescaleDB cofounder)
ncmncm 5y ago
There is a lot of development work left to do to optimize time-series database designs. They will get more useful as that work gets done.

228 points by telotortium 5y ago

These articles are so good. Very informative for a product engineer such as myself. Allows me to create a much clearer and more expansive mental model of the running systems.

thor24 5y ago
+1 Read their other articles on the blog as well. Very informative.
cgearhart 5y ago
It’s hard for me to rate the content. I couldn’t get past the sketchy dark pattern of a site that tracks me and sells my data for marketing with an “accept all cookies” button, while making it much harder to find the process for opting out. At that point it’s easier for me to just opt out of using the site. It’s too bad because I’d love to read some good technical articles, and this sounds like an interesting topic.
- dominotw 5y ago
  > sells my data for marketing with an “accept all cookies” button most likely just poorly implemented gdpr.
  - bostik 5y ago
    I used to wear the DPO hat, so this comes from a vantage point of having been in the weeds: ignorance is no excuse for incompetence. "Doing GDPR" badly because you don't know any better? Then don't even try to walk that tightrope. Don't track. At all. If you use on third parties who don't give you the option to do it right, then dump those third parties.
- rendall 5y ago
  Agreed. I open nearly every HN article from an unknown domain in incognito/private mode. At least half implement this kind of dark pattern. I find it spectacularly baffling for this kind of content-marketing website, where income presumably does not come from advertising, nor trading in visitor data, but from their DBaaS. Why would such a company risk alienating their visitors like this? It makes me question the decision-making in their unrelated, primary service; their conscientiousness; even their security. Truly mystifying.
  - s_fischer 5y ago
    I think there's a universe where there is no overlap between engineers/PMs who work on the actual DB product and website/content marketing site. The marketing space is filled with all kinds of "plug and play" SaaS providers which offer detailed customer journey data and sometimes it's just straight up easier to add an "accept all" consent banner than to try and allow for hot loading specific 3rd party libraries based on customized consent options. Is it the right thing to do? In my opinion, no. But I can also understand a situation where decisions were made on marketing tech before understanding the technical privacy implications. And then the implementation is handled by a team (potentially much smaller) that does not work on the actual product.
    - rendall 5y ago
      Sure. I believe I understand your explanation and appreciate it. It's possible for their core service to be rock solid and their marketing side to be of lesser priority and so not to receive the thought and resources of their core service. It makes sense. Still, were I to be responsible for evaluating competing services, such choices would definitely be a ding. Not unrecoverable, but it would make me wonder unnecessarily about their corporate culture and customer care. All things being equal, I think it would be wise to go with the service that didn't do that.
neilsense 5y ago
I wouldn't take this as the one truth, they are describing their database more than they are describing all timeseries databases. Just like MySQL and Postgres can behave wildly differently depending on use, so can timeseriesdbs, there is no one truth here.

proddata 5y ago

That article takes various concepts from typical TSDB solutions and seemingly only looks at the bad sides. Time series data has many different forms, not every form works for every TSDB solution. For the 3 caveats at the top, there are already two TS solutions that look promising (QuestDB, TimescaleDB). Often an operational analytics DB (Clickhouse, CrateDB) might also be a solution.

camel_gopher 5y ago
Take a look at IronDB too. High scale distributed implementationd.
jpgvm 5y ago
If you want something like Honeycomb but scales better then maybe look at Druid.
- dikei 5y ago
  Last time I checked, Druid were not very good at ad-hoc tasks because it lacked join and SQL supports was sketchy. How is it now ?
  - jpgvm 5y ago
    Limited JOIN support. SQL is now very good. JOINs vs no JOINs isn't an adhoc vs not-adhoc thing but more of a schema thing. If you try jam a star schema into it you aren't going to have a good time. This is true for pretty all of these more optimised stores. If you have a star schema and want to do those sorts of queries (and performance or cost aren't your #1 driving factors) then the better tool is a traditional data warehouse like BigQuery. This probably won't be the case forever though, there is significant progress in the Presto/Trino world to enable push-down for stores like Druid which would allow you to source most of your fact table data from other sources and then join into your events/time-orientated data from Druid very efficiently.
oconnore 5y ago
> Often an operational analytics DB (Clickhouse, CrateDB) might also be a solution This might be a bit off topic, but speaking of gaps in common observability tooling: is an OLAP database a common go-to for longer-timescale analytics (as in [1])? We're using BigQuery, but on ~600GB of log/event data I start hitting memory limits even with fairly small analytical windows. In this context I have seen other references to: Sawzall (google), Lingo (google), MapReduce/Pig/Cascading/Scalding. Are people using Spark for this sort of thing now? Perhaps a combined workflow would be ideal: filter/group/extract interesting data in Hadoop/Spark, and then load into OLAP for ad-hoc querying? [1]: https://danluu.com/metrics-analytics/ https://danluu.com/metrics-analytics/

tkzed49 5y ago

Pointless aside but that's probably the most annoying cookie banner I've seen. Your options are to accept the cookies or read their privacy policy, which is also covered by the cookie banner.

jldugger 5y ago

I'm really struggling to understand the marketing from Honeycomb. They have a lot of reputable and smart staff, seemingly making conflicting public statements. For example, in this post, the author describes how expensive cardinality is for TSDB, but doesn't really address how Honeycomb's column store implementation efficiently stores data with full cardinality anywhere near as cheaply. Like, I fully appreciate how having traces and structured event logs will improve observability, but this 'actually, Prometheus is expensive too' angle is weird, and their pricing page's 'starts at $100' doesn't really make it easy to compare.

jldugger 5y ago
Thinking about this a bit more, there's really two factors you have to balance when setting these systems up: event volume and cardinality. It feels like Honeycomb basically has you pre-pay for cardinality. Every field you add is equally efficient at storing data regardless of type or possible values (well, probably table compression helps some), and the event size is not a factor in billing(?). Since there's no pre-calculated aggregation, you bear the full cost of every event.

nivertech 5y ago

Cardinality is only expensive if the metric dimensions/tags/labels are a part of the builtin metric data model (like for example in InfluxDB). It's not expensive when dimensions are just columns in the custom table schema, like for example in TimescaleDB or kdb+/q. Also in the latter case a single database "row" can contain multiple metrics (each modeled as a separate column in a table).

hardwaresofton 5y ago
> It's not expensive when dimensions are just columns in the custom table schema, like for example in TimescaleDB or kdb+/q. kdb+/q (not exactly sure) is the one that is used heavily in finance, correct? [EDIT] - Yup[0][1]. I'm not sure how many competitors there are but last time I heard about kdb it seemed like they were pretty much the go-to choice outside comp sci land (which is normally preoccupied with influx, prometheus, timescale, etc). There's probably a reason it's trusted by those other fields. [0]: https://kx.com/ https://kx.com/ [1]: https://code.kx.com/home/ https://code.kx.com/home/
- liveoneggs 5y ago
  kdb is crazy expensive which is why storing your 5-min load average inside of it doesn't make much sense
- nivertech 5y ago
  kdb+/q was originally heavily used in finance/trading, but nowadays also in IoT/IIoT. The q language itself is pretty-much general purpose PL, but of course it shines for timeseries processing workloads.
claytonjy 5y ago
This is true, but given this post and the one linked about their custom column store, I don't think Timescale would fit. Timescale is still row-oriented while in memory (compressed chunks are columnar), though that may not be a big deal in practice; the bigger deal might be the ability to arbitrarily add columns. With Timescale you either have to issue DDL to add a column (which can be expensive and doesn't work in some cases), or resign yourself to JSONB. My worry with the JSONB is that it'll be expensive to search w/o indices, but if you need indices that goes against the Honeycomb philosophy. Very curious to hear if folks more deeply familiar with TimescaleDB internals have a different take!

gilbetron 5y ago

This is a very narrow definition of a time series DB, really more of a pure metrics store that requires individual time series to be kept separately. I find it odd, as I wouldn't define time series DB, in general, to be what they are. Rather, a time series DB is usually simple some sort of DB that supports a time column as the main index. There's tons that fall in that category (including Postgres, which you should absolutely use unless you have a compelling reason not to) that can easily handle many of the situations the article claims won't work. Druid can handle high-cardinality, for instance.

digbybk 5y ago
Can you talk more about why you should use Postgres unless you have a compelling reason not to? I'm currently investigating options for storing high-frequency data. Postgres is looking like a good option.
- gilbetron 5y ago
  Postgres has a lot of engineering work put into it to handle all kinds of use cases, plus massive effort by the community building tutorials, videos, wrappers, libraries, etc. If you have a problem with Postgres config, or are trying to get it to do something odd, there is undoubtedly a bunch of Stack Overflow discussions about that thing. For most other databases, the selection of all of those is much thinner. Additionally, often we believe our application needs feature X, and there is some database tech that purports to excel at X, the fact is postgres is probably able to do X unless you get to some extreme velocity or volume. Furthermore, by going with another database you are almost always giving up on features W, Y, and Z that you don't realize you need that Postgres supports and isn't supported by the "Exotic" database you are thinking of. In short, Postgres has amazing breadth and depth in features and support and tooling. Be sure you are ok giving that up!
  - akulkarni 5y ago
    If you like Postgres, you may want to try TimescaleDB, which is a time-series database built on Postgres (packaged as a Postgres extension). Postgres database + time-series database all in one. This btw is one of the reasons I love Postgres - its extensibility. (Disclaimer: TimescaleDB cofounder)

ncmncm 5y ago

There is a lot of development work left to do to optimize time-series database designs. They will get more useful as that work gets done.