9 ms·

Apache Druid vs. Time-Series Databases

150 points by wochiquan 7y ago

rb808 7y ago
I stumbled across Druid (with superset) and it looks great but has a really low profile, even though it is in a hot area. Does anyone actually use it?
- alexbanks 7y ago
  https://netflixtechblog.com/how-netflix-uses-druid-for-real-time-insights-to-ensure-a-high-quality-experience-19e1e8568d06 https://netflixtechblog.com/how-netflix-uses-druid-for-real-...
- wochiquan 7y ago
  https://medium.com/walmartlabs/event-stream-analytics-at-walmart-with-druid-dcf1a37ceda7 https://medium.com/walmartlabs/event-stream-analytics-at-wal...
- peterhunt 7y ago
  We use it a lot at Twitter, though it's a fairly recent addition to the stack.
- switzer 7y ago
  I used it a few years ago with my former company. It effortlessly injected 10K rows of data per second, and we configured Druid over 50 dimensions and about 20 different metrics. In that configuration, Druid was able to respond to most queries within a second. I am not sure how it works but it is an amazing product. We are evaluating it now for use at my current company.
  - shaklee3 7y ago
    To be fair, 10k rows per second is extremely slow. Clickhouse can do millions per second on a single server in our tests.
Nihilartikel 7y ago
I built out a Druid backend for interactive graphing & aggregation of web traffic and application security metrics a few years back. Users could choose arbitrary filters, aggregations, and time slicing. This was a second system replacing a Spark cluster running over timeseries events in Cassandra, which wasn't really practically scalable. Tuning and debugging the Spark queries and Cassandra performance was an endless time sink. Druid worked really well for almost all use cases, reliably getting sub-second results against many billions of records even on a pretty modest deployment. Being able to use arbitrary Javascript functions in queries was fantastic, so we could do things like filtering on IP subnets or case insensitive prefix/suffix matching, and the like, as needed. The Docker setup that Druid ships with is deceptively simple - making a production installation was an effort. My thoughts are: - Build templating and code generation for the many config files for each component early on so you can edit constants in a single place and have all of the sundry config files update to reflect them, and also manage per-host overrides in a sane, version controlled, way. - Druid will use as much ram as you can throw at it, but in a pinch, reading directly from fast NVME storage is pretty good. - If you have realtime data ingestion, you will also have to build tooling to re-ingest older data that has changed or needed to be amended. This will end up looking like a 'lambda architecture'
- jcims 7y ago
  This sounds kind of like splunk. Any similarity?
- battery_cowboy 7y ago
  I'm not a data guy, can you explain this part a bit more if you have time? > you have realtime data ingestion, you will also have to build tooling to re-ingest older data that has changed or needed to be amended. This will end up looking like a 'lambda architecture'
  - ignoramous 7y ago
    Lambda architecture for data processing, as popularized by Nathan Marz et al [0], has two components, the Batch layer and the Stream layer. At a high level, Batch trades quality for staleness whilst Stream optimises for freshness at the expense of quality [1]. I believe what GP means by Lambda is that, you'd need a system that batch processes the data to be amended / changed (reprocess older data) but stream processes whatever that's required for real-time [2]. An alternative is the Kappa architecture proposed initially by Jay Kreps [3][4], co-creator of Apache Kafka. --- [0] https://www.amazon.com/dp/1617290343 https://www.amazon.com/dp/1617290343 [1] https://en.wikipedia.org/wiki/Lambda_architecture https://en.wikipedia.org/wiki/Lambda_architecture [2] https://speakerdeck.com/druidio/real-time-analytics-with-open-source-technologies-1?slide=41 https://speakerdeck.com/druidio/real-time-analytics-with-ope... [3] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... [4] https://dataintensive.net/ https://dataintensive.net/
polskibus 7y ago
Why choose Druid over clickhouse?
- manigandham 7y ago
  Most people shouldn't. Clickhouse or other column-store data warehouses (redshift, bigquery, etc) are very fast and have all the features to handle time-series and other data. Druid is good if you (1) make use of native integrations like Kafka (2) Need every field indexed for fast seeking to a few rows (3) can use the JSON API to make up for the still-in-beta SQL interface (4) don't need every single event/row as they are pre-aggregated (5) always have a time-column to partition by (6) want to use the native S3 tiering for older data (7) dont need joins and complex analysis Imply's distribution is better than core Druid but it's still more operationally complex than Clickhouse and alternatives.
  - leetbulb 7y ago
    1) Most people handling large data streams are already using Kafka or similar. If you aren't, Druid's has pretty wide support for event ingress, including HTTP. 2) If you're looking to provide this type of analytics in the first place, you probably do want this. Being able to execute extremely fast, granular, ad-hoc queries over highly dimensional data is very powerful. I designed a reporting frontend that really took advantage of this and I always felt guilty when people would complimented me how fast it drilled down in to 10+ dimensions. 3) There are plenty of mature libraries for the Druid API. 4) This is almost inherent with any OLAP system. Although, even with high-cardinality data, Druid performs extremely well. Either way, you should have a backing data warehouse and offline, batch jobs if you need to perform BI analytics on row-by-row / non-aggregated data. Remember, OLAP sits on top of a data warehouse. 5) Yes, but technically you just need a key that increments. 6) Not sure if this is an argument against Druid. I've found the S3 deep storage / tiering to be very efficient and powerful, especially because you can create and push segments directly to storage and not run it through an indexer. S3 is also just an object store protocol specification now. There are lots of people who run S3-compatible object stores in-house. HDFS is also natively supported and another widely used storage backend in this space. Also, there are plenty of community extensions for other object stores. 7) Again, OLAP. Data flowing in to your OLAP layer should already be denormalized and ready for indexing. Also, you can join data in to your existing indices with Hadoop, etc. Druid support joins and lookups, although I've never used them. ClickHouse and other similar systems also don't do very well with joins. Maybe we have different definitions of "complex analysis," but in my experience, you can do some pretty crazy stuff with queries including writing your own JS functions, and if you're really dedicated, you can write your own extensions. One thing that I feel like a lot of people miss is that Druid is specifically an OLAP layer designed for large scale, and I mean beyond Netflix size scale (they use Druid). Every individual component of Druid is designed to scale out independently and play nicely with your existing infrastructure. Similar to ES, you have nodes that have specific roles such as event ingress / real-time compute, query handlers, query brokers, historical data / cache / compute, etc. Then you also have a bunch of supporting architecture provided for you for (re)indexing data, coordinating the entire cluster, etc. Druid is huge, not an AIO (OLAP, OLTP, DWH, etc) analytics solution, and it takes more than one person to run a larger cluster, even though I did it for a few years.
starpilot 7y ago
kdb/Shakti
- quod_2058 7y ago
  Or questDB if you want that sort of performance but prefer sql
  - starpilot 7y ago
    Fast as kdb? You have a single fact to back that up?
    - quod_2058 7y ago
      Saw this last week. The examples are limited but clearly easy to reproduce. I saw in a discussion on Reddit where it seems they plan to get even faster using more techniques like prefetch. https://news.ycombinator.com/item?id=22803504 https://news.ycombinator.com/item?id=22803504
      - starpilot 7y ago
        mother of god.
jonpo 7y ago
Is it really better though? i feel like we need more data. what are the tradeoffs? seems like a sales article
- adequateness 7y ago
  It is an article published on a blog of a company talking about what said company does.
quietgolfer 7y ago
I looked into Apache Druid for my company's event aggregation and serving. We decided to go with Apache Pinot instead. A major reason was serving latency. Most of the docs I saw that talked about Druid latency (and their goal for latency) made it seem like 500ms was okay for query latency speeds. Pinot seemed more focused on query latency. 50ms was more acceptable for my team's latency goal. I ran some basic latency tests and Pinot was faster (2mil and 22mil events, no real optimizations). I think Pinot's star-tree index would perform great for my team's use case. Druid's intro flow was awesome. The tutorials worked very well. I hit hiccups with Pinot's intro flow. The Pinot team has been very responsive on Slack to issues. Here are some notes: https://docs.google.com/document/d/1GiB8zoiJ4Qs10A1LdIr6lJGWTfk-0ElU8aJbejomstY/edit https://docs.google.com/document/d/1GiB8zoiJ4Qs10A1LdIr6lJGW...
- ychb 7y ago
  Similar to @quietgolfer, we also have strict requirement on serving latency, and we benefited from Pinot's star-tree index. We had a table with more than 1 billion events, and we had to do DISTINCTCOUNT and SUM with GROUPBY. Initially without optimization, queries timed out (after 15s). After star-tree index was added, the query returned in 500ms. Based on our earlier benchmark, Pinot query latency without star-tree is already slightly better than Druid: https://image.slidesharecdn.com/lnkdmeetupslides0426-170502173242/95/pinot-near-realtime-analytics-uber-18-638.jpg?cb=1493746410 https://image.slidesharecdn.com/lnkdmeetupslides0426-1705021.... Though that benchmark was done years ago, the result seems consistent with the recent test run by @quietgolfer.
yumraj 7y ago
Just curious what are people using Druid, Clickhouse, Pinot... for? would love to learn by knowing about the use-cases where I should perhaps consider such technologies..
- hodgesrm 7y ago
  Some common ClickHouse use cases and verticals shown below. Basically any case where you have relatively structured data, very large fact tables, and a need for low latency response. Names of companies that have given public talks are shown. * Analyzing netflow logs (network management, numerous users) * Content delivery (Mux.com) * Web analytics (ContentSquare, CloudFlare) * Generating optimized parameters for real time ad bidding (various, not too many recent talks on this) * Log management and system observability (surprisingly common across industries-Sentry, Cloudflare, Uber) * Valuing assets from market data, e.g. tick data (financial services, not many companies will discuss publicly) Druid is used for some of these. I don't know that much about Pinot, hence cannot comment there.

150 points by wochiquan 7y ago

I stumbled across Druid (with superset) and it looks great but has a really low profile, even though it is in a hot area. Does anyone actually use it?

alexbanks 7y ago
https://netflixtechblog.com/how-netflix-uses-druid-for-real-time-insights-to-ensure-a-high-quality-experience-19e1e8568d06 https://netflixtechblog.com/how-netflix-uses-druid-for-real-...
wochiquan 7y ago
https://medium.com/walmartlabs/event-stream-analytics-at-walmart-with-druid-dcf1a37ceda7 https://medium.com/walmartlabs/event-stream-analytics-at-wal...
peterhunt 7y ago
We use it a lot at Twitter, though it's a fairly recent addition to the stack.
switzer 7y ago
I used it a few years ago with my former company. It effortlessly injected 10K rows of data per second, and we configured Druid over 50 dimensions and about 20 different metrics. In that configuration, Druid was able to respond to most queries within a second. I am not sure how it works but it is an amazing product. We are evaluating it now for use at my current company.
- shaklee3 7y ago
  To be fair, 10k rows per second is extremely slow. Clickhouse can do millions per second on a single server in our tests.

Nihilartikel 7y ago

I built out a Druid backend for interactive graphing & aggregation of web traffic and application security metrics a few years back. Users could choose arbitrary filters, aggregations, and time slicing. This was a second system replacing a Spark cluster running over timeseries events in Cassandra, which wasn't really practically scalable. Tuning and debugging the Spark queries and Cassandra performance was an endless time sink. Druid worked really well for almost all use cases, reliably getting sub-second results against many billions of records even on a pretty modest deployment. Being able to use arbitrary Javascript functions in queries was fantastic, so we could do things like filtering on IP subnets or case insensitive prefix/suffix matching, and the like, as needed. The Docker setup that Druid ships with is deceptively simple - making a production installation was an effort. My thoughts are: - Build templating and code generation for the many config files for each component early on so you can edit constants in a single place and have all of the sundry config files update to reflect them, and also manage per-host overrides in a sane, version controlled, way. - Druid will use as much ram as you can throw at it, but in a pinch, reading directly from fast NVME storage is pretty good. - If you have realtime data ingestion, you will also have to build tooling to re-ingest older data that has changed or needed to be amended. This will end up looking like a 'lambda architecture'

jcims 7y ago
This sounds kind of like splunk. Any similarity?
battery_cowboy 7y ago
I'm not a data guy, can you explain this part a bit more if you have time? > you have realtime data ingestion, you will also have to build tooling to re-ingest older data that has changed or needed to be amended. This will end up looking like a 'lambda architecture'
- ignoramous 7y ago
  Lambda architecture for data processing, as popularized by Nathan Marz et al [0], has two components, the Batch layer and the Stream layer. At a high level, Batch trades quality for staleness whilst Stream optimises for freshness at the expense of quality [1]. I believe what GP means by Lambda is that, you'd need a system that batch processes the data to be amended / changed (reprocess older data) but stream processes whatever that's required for real-time [2]. An alternative is the Kappa architecture proposed initially by Jay Kreps [3][4], co-creator of Apache Kafka. --- [0] https://www.amazon.com/dp/1617290343 https://www.amazon.com/dp/1617290343 [1] https://en.wikipedia.org/wiki/Lambda_architecture https://en.wikipedia.org/wiki/Lambda_architecture [2] https://speakerdeck.com/druidio/real-time-analytics-with-open-source-technologies-1?slide=41 https://speakerdeck.com/druidio/real-time-analytics-with-ope... [3] https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... [4] https://dataintensive.net/ https://dataintensive.net/

polskibus 7y ago

Why choose Druid over clickhouse?

manigandham 7y ago
Most people shouldn't. Clickhouse or other column-store data warehouses (redshift, bigquery, etc) are very fast and have all the features to handle time-series and other data. Druid is good if you (1) make use of native integrations like Kafka (2) Need every field indexed for fast seeking to a few rows (3) can use the JSON API to make up for the still-in-beta SQL interface (4) don't need every single event/row as they are pre-aggregated (5) always have a time-column to partition by (6) want to use the native S3 tiering for older data (7) dont need joins and complex analysis Imply's distribution is better than core Druid but it's still more operationally complex than Clickhouse and alternatives.
- leetbulb 7y ago
  1) Most people handling large data streams are already using Kafka or similar. If you aren't, Druid's has pretty wide support for event ingress, including HTTP. 2) If you're looking to provide this type of analytics in the first place, you probably do want this. Being able to execute extremely fast, granular, ad-hoc queries over highly dimensional data is very powerful. I designed a reporting frontend that really took advantage of this and I always felt guilty when people would complimented me how fast it drilled down in to 10+ dimensions. 3) There are plenty of mature libraries for the Druid API. 4) This is almost inherent with any OLAP system. Although, even with high-cardinality data, Druid performs extremely well. Either way, you should have a backing data warehouse and offline, batch jobs if you need to perform BI analytics on row-by-row / non-aggregated data. Remember, OLAP sits on top of a data warehouse. 5) Yes, but technically you just need a key that increments. 6) Not sure if this is an argument against Druid. I've found the S3 deep storage / tiering to be very efficient and powerful, especially because you can create and push segments directly to storage and not run it through an indexer. S3 is also just an object store protocol specification now. There are lots of people who run S3-compatible object stores in-house. HDFS is also natively supported and another widely used storage backend in this space. Also, there are plenty of community extensions for other object stores. 7) Again, OLAP. Data flowing in to your OLAP layer should already be denormalized and ready for indexing. Also, you can join data in to your existing indices with Hadoop, etc. Druid support joins and lookups, although I've never used them. ClickHouse and other similar systems also don't do very well with joins. Maybe we have different definitions of "complex analysis," but in my experience, you can do some pretty crazy stuff with queries including writing your own JS functions, and if you're really dedicated, you can write your own extensions. One thing that I feel like a lot of people miss is that Druid is specifically an OLAP layer designed for large scale, and I mean beyond Netflix size scale (they use Druid). Every individual component of Druid is designed to scale out independently and play nicely with your existing infrastructure. Similar to ES, you have nodes that have specific roles such as event ingress / real-time compute, query handlers, query brokers, historical data / cache / compute, etc. Then you also have a bunch of supporting architecture provided for you for (re)indexing data, coordinating the entire cluster, etc. Druid is huge, not an AIO (OLAP, OLTP, DWH, etc) analytics solution, and it takes more than one person to run a larger cluster, even though I did it for a few years.

starpilot 7y ago

kdb/Shakti

quod_2058 7y ago
Or questDB if you want that sort of performance but prefer sql
- starpilot 7y ago
  Fast as kdb? You have a single fact to back that up?
  - quod_2058 7y ago
    Saw this last week. The examples are limited but clearly easy to reproduce. I saw in a discussion on Reddit where it seems they plan to get even faster using more techniques like prefetch. https://news.ycombinator.com/item?id=22803504 https://news.ycombinator.com/item?id=22803504
    - starpilot 7y ago
      mother of god.

jonpo 7y ago

Is it really better though? i feel like we need more data. what are the tradeoffs? seems like a sales article

adequateness 7y ago
It is an article published on a blog of a company talking about what said company does.

quietgolfer 7y ago

I looked into Apache Druid for my company's event aggregation and serving. We decided to go with Apache Pinot instead. A major reason was serving latency. Most of the docs I saw that talked about Druid latency (and their goal for latency) made it seem like 500ms was okay for query latency speeds. Pinot seemed more focused on query latency. 50ms was more acceptable for my team's latency goal. I ran some basic latency tests and Pinot was faster (2mil and 22mil events, no real optimizations). I think Pinot's star-tree index would perform great for my team's use case. Druid's intro flow was awesome. The tutorials worked very well. I hit hiccups with Pinot's intro flow. The Pinot team has been very responsive on Slack to issues. Here are some notes: https://docs.google.com/document/d/1GiB8zoiJ4Qs10A1LdIr6lJGWTfk-0ElU8aJbejomstY/edit https://docs.google.com/document/d/1GiB8zoiJ4Qs10A1LdIr6lJGW...

ychb 7y ago
Similar to @quietgolfer, we also have strict requirement on serving latency, and we benefited from Pinot's star-tree index. We had a table with more than 1 billion events, and we had to do DISTINCTCOUNT and SUM with GROUPBY. Initially without optimization, queries timed out (after 15s). After star-tree index was added, the query returned in 500ms. Based on our earlier benchmark, Pinot query latency without star-tree is already slightly better than Druid: https://image.slidesharecdn.com/lnkdmeetupslides0426-170502173242/95/pinot-near-realtime-analytics-uber-18-638.jpg?cb=1493746410 https://image.slidesharecdn.com/lnkdmeetupslides0426-1705021.... Though that benchmark was done years ago, the result seems consistent with the recent test run by @quietgolfer.

yumraj 7y ago

Just curious what are people using Druid, Clickhouse, Pinot... for? would love to learn by knowing about the use-cases where I should perhaps consider such technologies..

hodgesrm 7y ago
Some common ClickHouse use cases and verticals shown below. Basically any case where you have relatively structured data, very large fact tables, and a need for low latency response. Names of companies that have given public talks are shown. * Analyzing netflow logs (network management, numerous users) * Content delivery (Mux.com) * Web analytics (ContentSquare, CloudFlare) * Generating optimized parameters for real time ad bidding (various, not too many recent talks on this) * Log management and system observability (surprisingly common across industries-Sentry, Cloudflare, Uber) * Valuing assets from market data, e.g. tick data (financial services, not many companies will discuss publicly) Druid is used for some of these. I don't know that much about Pinot, hence cannot comment there.