8 ms·

A Comparison of Time Series Databases and Netsil’s Use of Druid

107 points by fangjin 10y ago

bbrazil 10y ago
Great to see an analysis of the various options. A few corrections on Prometheus: > an AlertManager which handles generating and transmitting alerts Alerts are generated in Prometheus, the Alertmanager then groups, deduplicates and transmits them. See https://www.robustperception.io/prometheus-and-alertmanager-architecture/ https://www.robustperception.io/prometheus-and-alertmanager-... > Data Storage Model: Prometheus uses LevelDB as the storage backend LevelDB is only used for metadata (we used to use LevelDB for everything, that didn't work out too well). The primary timeseries storage is a custom format, which you can find more details about in https://www.youtube.com/watch?v=HbnGSNEjhUc&index=6&list=PLoz-W_CUquUlCq-Q0hy53TolAhaED9vmU https://www.youtube.com/watch?v=HbnGSNEjhUc&index=6&list=PLo... > It has an active community but no enterprise support. There's several providers of support, my company being one of them. See https://prometheus.io/community/ https://prometheus.io/community/ And some others: > InfluxDB supports ingesting data via a CLI, client libraries and an HTTP API. It also accepts a custom TCP protocol, the graphite protocol and the OpenTSDB protocol. > Influxdb .. supports GroupBy but not TopN queries. https://docs.influxdata.com/influxdb/v0.13/query_language/functions/#top https://docs.influxdata.com/influxdb/v0.13/query_language/fu...
- smb06 10y ago
  Thanks for your comments. They have been incorporated in the blog.
mattdeboard 10y ago
Am I completely alone in thinking ElasticSearch is out of place here? Am I misunderstanding something about ES? It's a NoSQL doc store but is there something about its indexing or storage tech (Lucene, right?) that specifically makes it optimal for time-series data?
- gillh 10y ago
  There are a bunch of companies that use ElasticSearch for storing timeseries data for monitoring use-cases. Reference: 1. https://www.opsclarity.com/riak-elasticsearch-scaled-monitoring-backend-opsclarity/ https://www.opsclarity.com/riak-elasticsearch-scaled-monitor... 2. https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store https://www.elastic.co/blog/elasticsearch-as-a-time-series-d...
- lobster_johnson 10y ago
  ES is quite good at time-series data, actually. Lucene is effectively a sparse column store. Its storage format is quite compact, and they do some clever things to balance between speed and memory usage. Lucene uses append-only index files (segments) that are compacted (merged) after a while; not too dissimilar from how Cassandra or LevelDB work. The aggregation API in ES is quite extensive, too. Date histograms, top-K, etc., all composable and extremely fast, particularly when parallelized over multiple shards (though you have to prepare for how sharded aggregation can skew some aggregation calculations). ES is also impressively fast at bulk indexing. There are some downsides. ES has a global field cache of some sort (I forget the details) that makes it increasingly bloated as you add more unique fields, which can be a problem for some applications, particularly if you federate many different apps in one cluster. In general, ES -- being on the JVM -- is RAM-hungry, and it's a little too easy to cause it to run out of memory, even with a huge heap and all caches etc. configured with hard limits. ES's schema system (the "mappings") also leaves a lot to be desired, but this problem is less applicable to an analytics app. Lastly: If you ever want to compact historical data by merging events to a lower granularity (is there an industry term for this?), ES doesn't give you any tools to do it. You will have to write the logic yourself.
  - iampims 10y ago
    > If you ever want to compact historical data by merging events to a lower granularity (is there an industry term for this?) I believe this is called a roll-up.
- laichzeit0 10y ago
  There's a comparison between InfluxDB and Elastic (albeit written by Influx themselves, so take it with a grain of salt), but it would appear there's a strong case to be made that there are better alternatives than Elastic if want to store time-series metrics. https://www.influxdata.com/influxdb-markedly-elasticsearch-in-time-series-data-metrics-benchmark/ https://www.influxdata.com/influxdb-markedly-elasticsearch-i...
marknadal 10y ago
Any comparison on the cost to run those systems? Disclosure: Reason why I ask is because we're a competitor, however currently less focused on timeseries (we don't nearly have all those interesting features yet), but our latest load testing saved 100M+ records for $10/day[1] which is a pretty disruptive price, for over 100GB of data. Next up is TB scale! So it would be useful to see price comparisons of other systems. [1] https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8
alex_hirner 10y ago
Not to forget chronix.io, which was shown to have favorable performance for more complex queries and space efficiency at last week's http://munich-datageeks.de/dadada/ http://munich-datageeks.de/dadada/ (slides yet to come).
dataloopio 10y ago
Broader comparison of the facts: https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/edit https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCE... Also a blog comparison: https://blog.dataloop.io/top10-open-source-time-series-databases https://blog.dataloop.io/top10-open-source-time-series-datab... From reading benchmarks and various articles on how to setup druid for real-time analytics it isn't great. Druid was designed for batch workloads. Feeding the database slowly and then once complete performing analytics over a large data set. Real-time streaming analytics are dreadful to setup and use.
- fangjin 10y ago
  While Druid was initially built for batch loads, the architecture has evolved substantially as the project has matured. Today, Druid supports exactly-once streaming ingestion from Kafka, and large production deployments routinely stream millions of events per second into Druid.
  - dataloopio 10y ago
    Can you point me to a source for 'routinely stream millions of events per second into druid'? While it is true that druid is great at querying billions of rows per second it's not very good at ingress. Here is a mailing list discussion for some background. https://groups.google.com/forum/#!searchin/druid-user/benchmark%7Csort:relevance/druid-user/90BMCxz22Ko/73D8HidLCgAJ https://groups.google.com/forum/#!searchin/druid-user/benchm...
    - packetslave 10y ago
      well, he's is a druid committer and CEO of a company built on top of it, so...
      - dataloopio 10y ago
        No bias there then
    - 10y ago
deleted 10y ago
[deleted]
mej10 10y ago
What amount of data are you ingesting per day and what is the cost? How much time do you spend maintaining Druid? Do you have dedicated SRE/devops people managing it or is it something a dev could do with minimal time investment? Answers to any of these would be super helpful -- it is hard to find details on these aspects of real world uses of Druid.
- gillh 10y ago
  We are packaging our product as a self-hosted machine images (www.netsil.com/download). We are using DC/OS runtime to orchestrate all our internal services and have written various health check scripts to make sure the data pipeline components such as our stream-processor, Kafka and Druid stay operational. This blog post describes Netsil's architecture: https://blog.netsil.com/listen-to-your-apis-see-your-apps-a91ece5aa5bd https://blog.netsil.com/listen-to-your-apis-see-your-apps-a9... In-terms of data being ingested -- we are stream-processing service interactions (network packets!) in real-time, storing interaction logs in memory and rolling them up to generate multidimensional timeseries data emitted every second. Therefore, the actual timeseries ingestion rates by Druid vary across deployments.

107 points by fangjin 10y ago

Great to see an analysis of the various options. A few corrections on Prometheus: > an AlertManager which handles generating and transmitting alerts Alerts are generated in Prometheus, the Alertmanager then groups, deduplicates and transmits them. See https://www.robustperception.io/prometheus-and-alertmanager-architecture/ https://www.robustperception.io/prometheus-and-alertmanager-... > Data Storage Model: Prometheus uses LevelDB as the storage backend LevelDB is only used for metadata (we used to use LevelDB for everything, that didn't work out too well). The primary timeseries storage is a custom format, which you can find more details about in https://www.youtube.com/watch?v=HbnGSNEjhUc&index=6&list=PLoz-W_CUquUlCq-Q0hy53TolAhaED9vmU https://www.youtube.com/watch?v=HbnGSNEjhUc&index=6&list=PLo... > It has an active community but no enterprise support. There's several providers of support, my company being one of them. See https://prometheus.io/community/ https://prometheus.io/community/ And some others: > InfluxDB supports ingesting data via a CLI, client libraries and an HTTP API. It also accepts a custom TCP protocol, the graphite protocol and the OpenTSDB protocol. > Influxdb .. supports GroupBy but not TopN queries. https://docs.influxdata.com/influxdb/v0.13/query_language/functions/#top https://docs.influxdata.com/influxdb/v0.13/query_language/fu...

smb06 10y ago
Thanks for your comments. They have been incorporated in the blog.

mattdeboard 10y ago

Am I completely alone in thinking ElasticSearch is out of place here? Am I misunderstanding something about ES? It's a NoSQL doc store but is there something about its indexing or storage tech (Lucene, right?) that specifically makes it optimal for time-series data?

gillh 10y ago
There are a bunch of companies that use ElasticSearch for storing timeseries data for monitoring use-cases. Reference: 1. https://www.opsclarity.com/riak-elasticsearch-scaled-monitoring-backend-opsclarity/ https://www.opsclarity.com/riak-elasticsearch-scaled-monitor... 2. https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store https://www.elastic.co/blog/elasticsearch-as-a-time-series-d...
lobster_johnson 10y ago
ES is quite good at time-series data, actually. Lucene is effectively a sparse column store. Its storage format is quite compact, and they do some clever things to balance between speed and memory usage. Lucene uses append-only index files (segments) that are compacted (merged) after a while; not too dissimilar from how Cassandra or LevelDB work. The aggregation API in ES is quite extensive, too. Date histograms, top-K, etc., all composable and extremely fast, particularly when parallelized over multiple shards (though you have to prepare for how sharded aggregation can skew some aggregation calculations). ES is also impressively fast at bulk indexing. There are some downsides. ES has a global field cache of some sort (I forget the details) that makes it increasingly bloated as you add more unique fields, which can be a problem for some applications, particularly if you federate many different apps in one cluster. In general, ES -- being on the JVM -- is RAM-hungry, and it's a little too easy to cause it to run out of memory, even with a huge heap and all caches etc. configured with hard limits. ES's schema system (the "mappings") also leaves a lot to be desired, but this problem is less applicable to an analytics app. Lastly: If you ever want to compact historical data by merging events to a lower granularity (is there an industry term for this?), ES doesn't give you any tools to do it. You will have to write the logic yourself.
- iampims 10y ago
  > If you ever want to compact historical data by merging events to a lower granularity (is there an industry term for this?) I believe this is called a roll-up.
laichzeit0 10y ago
There's a comparison between InfluxDB and Elastic (albeit written by Influx themselves, so take it with a grain of salt), but it would appear there's a strong case to be made that there are better alternatives than Elastic if want to store time-series metrics. https://www.influxdata.com/influxdb-markedly-elasticsearch-in-time-series-data-metrics-benchmark/ https://www.influxdata.com/influxdb-markedly-elasticsearch-i...

marknadal 10y ago

Any comparison on the cost to run those systems? Disclosure: Reason why I ask is because we're a competitor, however currently less focused on timeseries (we don't nearly have all those interesting features yet), but our latest load testing saved 100M+ records for $10/day[1] which is a pretty disruptive price, for over 100GB of data. Next up is TB scale! So it would be useful to see price comparisons of other systems. [1] https://www.youtube.com/watch?v=x_WqBuEA7s8 https://www.youtube.com/watch?v=x_WqBuEA7s8

alex_hirner 10y ago

Not to forget chronix.io, which was shown to have favorable performance for more complex queries and space efficiency at last week's http://munich-datageeks.de/dadada/ http://munich-datageeks.de/dadada/ (slides yet to come).

dataloopio 10y ago

Broader comparison of the facts: https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCEWdPtAoccJ4a-IuZv4fXDHxM/edit https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCE... Also a blog comparison: https://blog.dataloop.io/top10-open-source-time-series-databases https://blog.dataloop.io/top10-open-source-time-series-datab... From reading benchmarks and various articles on how to setup druid for real-time analytics it isn't great. Druid was designed for batch workloads. Feeding the database slowly and then once complete performing analytics over a large data set. Real-time streaming analytics are dreadful to setup and use.

fangjin 10y ago
While Druid was initially built for batch loads, the architecture has evolved substantially as the project has matured. Today, Druid supports exactly-once streaming ingestion from Kafka, and large production deployments routinely stream millions of events per second into Druid.
- dataloopio 10y ago
  Can you point me to a source for 'routinely stream millions of events per second into druid'? While it is true that druid is great at querying billions of rows per second it's not very good at ingress. Here is a mailing list discussion for some background. https://groups.google.com/forum/#!searchin/druid-user/benchmark%7Csort:relevance/druid-user/90BMCxz22Ko/73D8HidLCgAJ https://groups.google.com/forum/#!searchin/druid-user/benchm...
  - packetslave 10y ago
    well, he's is a druid committer and CEO of a company built on top of it, so...
    - dataloopio 10y ago
      No bias there then
  - 10y ago

deleted 10y ago

[deleted]

mej10 10y ago

What amount of data are you ingesting per day and what is the cost? How much time do you spend maintaining Druid? Do you have dedicated SRE/devops people managing it or is it something a dev could do with minimal time investment? Answers to any of these would be super helpful -- it is hard to find details on these aspects of real world uses of Druid.

gillh 10y ago
We are packaging our product as a self-hosted machine images (www.netsil.com/download). We are using DC/OS runtime to orchestrate all our internal services and have written various health check scripts to make sure the data pipeline components such as our stream-processor, Kafka and Druid stay operational. This blog post describes Netsil's architecture: https://blog.netsil.com/listen-to-your-apis-see-your-apps-a91ece5aa5bd https://blog.netsil.com/listen-to-your-apis-see-your-apps-a9... In-terms of data being ingested -- we are stream-processing service interactions (network packets!) in real-time, storing interaction logs in memory and rolling them up to generate multidimensional timeseries data emitted every second. Therefore, the actual timeseries ingestion rates by Druid vary across deployments.