5 ms·

Would be interesting to see a comparison to Druid: http://druid.io/ http://druid.io/ They both seem to cater the same use case: scalable real-time analytics.

by eis 11y ago

boothead 11y ago
I came here to ask for a comparison between the two. I'll add the request here instead instead of starting a new top level thread. Both look really interesting!
- fangjin 11y ago
  At a high level, the architecture seems to be very similar to Druid, although the Pinot team has yet to release more technical information about the project. Druid is farther along in terms of development, community, adoption, and scale. It is curious why the team choose to rebuild the architecture versus just using Druid.
deleted 11y ago
[deleted]
ddlatham 11y ago
The architecture is extremely similar.
lgsilver 11y ago
Has anyone tested query volumes on Druid.io? We're thinking about using it in production (may now also consider Pinot), but we're concerned about the lack documentation around production load query volumes. Really like that LinkedIn uses Pinot at runtime.
- cheddar 11y ago
  lgsilver, it really depends on what kinds of queries you are doing. For something like "who viewed my profile" where you are likely to have a very selective where clause that matches only a few hundred rows, getting thousands of queries a second should be possible. Most of the use cases where I've seen Druid used are scanning and aggregating billions of rows per query, in that world it requires a bit more hardware to achieve thousands of queries per second. Fwiw, we also have user-facing flows at Yahoo that are ingesting multiple billions of events per day powered by Druid. One thing that I think makes it really hard to compare these systems is the lack of a meaningful benchmark. One of the difficulties with coming up with a meaningful benchmark is the wide variance in use cases and different ways of shaping the same problems.
cheddar 11y ago
I'm biased in that I had the honor of writing the first line of code of Druid, but from looking at what I could find on Pinot it looks like architecture-wise, we share a lot of similarities. At a minimum, the thought behind how to scale and how to ingest data appears to be the exact same. Druid clusters are presently ingesting tens of billions of events per day (with a 50b number from Metamarkets in the venturebeat article) and housing trillions of rows. From a high level architecture, I don't see a reason Pinot couldn't as well (when they are the same, it's kinda hard to say that one can't handle what the other can ;)). On the query API-side, it appears like Pinot exposes "PQL" which is a SQL-like query language. In Druid, for better or worse, we use JSON-based HTTP calls. Once you are using a client for querying in a programmatic manner, the difference doesn't really matter, but it is true that something like PQL makes it simpler for humans to write their own queries. For management of distributed state, it looks like Pinot depends on Apache Helix, which is an interesting framework. This was the first time I had heard about it and, I really would've liked to have something like that to build on when we started Druid. We instead had to implement it all ourselves. Helix sounds like it provides all the hooks to enable the various levels of integration that we have needs for in Druid, so I can only imagine that there is little difference in capability there. My biggest fear is that the majority of operational issues that we run into with Druid is around zookeeper integration, losing an entire ZK cluster is painful (and does happen) and from my read of the Helix docs, that would likely cause a loss of all shards of data as well (it looks like they are persistent znodes?). In terms of storage format, Druid enforces bitmap indexing and the Pinot docs appear to indicate that they have multiple types of indexes available, which is pretty cool. It looks like Pinot doesn't have a way to plug in user-defined column formats, though, so it would likely be require a bit of extra work to extend the system to be able to handle more complex types of aggregators like sketches. And, really, enabling sketching extensions is one of the most important features of any "scalable realtime processing" tool, in my opinion. On the ingestion side, it appears that Pinot reads data from kafka in real-time and then restating from batch, which is the model we've employed in Druid for the last 4 years, so we know it works. I'd love to know how Pinot guarantees high availability and replication of the real-time stream, there are many ways of doing that (over the years, we've implemented 3 different ways in Druid and are still not particularly happy with any of them). Anyway, I still haven't had a chance to play with Pinot, so this could be completely off-base, but those are my initial reactions.

Would be interesting to see a comparison to Druid: http://druid.io/ http://druid.io/ They both seem to cater the same use case: scalable real-time analytics.

by eis 11y ago

Would be interesting to see a comparison to Druid: http://druid.io/ http://druid.io/ They both seem to cater the same use case: scalable real-time analytics.

boothead 11y ago

I came here to ask for a comparison between the two. I'll add the request here instead instead of starting a new top level thread. Both look really interesting!

fangjin 11y ago
At a high level, the architecture seems to be very similar to Druid, although the Pinot team has yet to release more technical information about the project. Druid is farther along in terms of development, community, adoption, and scale. It is curious why the team choose to rebuild the architecture versus just using Druid.

deleted 11y ago

[deleted]

ddlatham 11y ago

The architecture is extremely similar.

lgsilver 11y ago

Has anyone tested query volumes on Druid.io? We're thinking about using it in production (may now also consider Pinot), but we're concerned about the lack documentation around production load query volumes. Really like that LinkedIn uses Pinot at runtime.

cheddar 11y ago
lgsilver, it really depends on what kinds of queries you are doing. For something like "who viewed my profile" where you are likely to have a very selective where clause that matches only a few hundred rows, getting thousands of queries a second should be possible. Most of the use cases where I've seen Druid used are scanning and aggregating billions of rows per query, in that world it requires a bit more hardware to achieve thousands of queries per second. Fwiw, we also have user-facing flows at Yahoo that are ingesting multiple billions of events per day powered by Druid. One thing that I think makes it really hard to compare these systems is the lack of a meaningful benchmark. One of the difficulties with coming up with a meaningful benchmark is the wide variance in use cases and different ways of shaping the same problems.

cheddar 11y ago

I'm biased in that I had the honor of writing the first line of code of Druid, but from looking at what I could find on Pinot it looks like architecture-wise, we share a lot of similarities. At a minimum, the thought behind how to scale and how to ingest data appears to be the exact same. Druid clusters are presently ingesting tens of billions of events per day (with a 50b number from Metamarkets in the venturebeat article) and housing trillions of rows. From a high level architecture, I don't see a reason Pinot couldn't as well (when they are the same, it's kinda hard to say that one can't handle what the other can ;)). On the query API-side, it appears like Pinot exposes "PQL" which is a SQL-like query language. In Druid, for better or worse, we use JSON-based HTTP calls. Once you are using a client for querying in a programmatic manner, the difference doesn't really matter, but it is true that something like PQL makes it simpler for humans to write their own queries. For management of distributed state, it looks like Pinot depends on Apache Helix, which is an interesting framework. This was the first time I had heard about it and, I really would've liked to have something like that to build on when we started Druid. We instead had to implement it all ourselves. Helix sounds like it provides all the hooks to enable the various levels of integration that we have needs for in Druid, so I can only imagine that there is little difference in capability there. My biggest fear is that the majority of operational issues that we run into with Druid is around zookeeper integration, losing an entire ZK cluster is painful (and does happen) and from my read of the Helix docs, that would likely cause a loss of all shards of data as well (it looks like they are persistent znodes?). In terms of storage format, Druid enforces bitmap indexing and the Pinot docs appear to indicate that they have multiple types of indexes available, which is pretty cool. It looks like Pinot doesn't have a way to plug in user-defined column formats, though, so it would likely be require a bit of extra work to extend the system to be able to handle more complex types of aggregators like sketches. And, really, enabling sketching extensions is one of the most important features of any "scalable realtime processing" tool, in my opinion. On the ingestion side, it appears that Pinot reads data from kafka in real-time and then restating from batch, which is the model we've employed in Druid for the last 4 years, so we know it works. I'd love to know how Pinot guarantees high availability and replication of the real-time stream, there are many ways of doing that (over the years, we've implemented 3 different ways in Druid and are still not particularly happy with any of them). Anyway, I still haven't had a chance to play with Pinot, so this could be completely off-base, but those are my initial reactions.