3 ms·
Thoughts on OLAPs ClickHouse vs. Apache Druid vs. Starrocks in 2023/2024
I've been conducting a research around OLAP dbs that are relevant in 2023 with my data science team for a new project.
The ideal solution would be a db that can make use of an S3 bucket as long term storage, while being easy to deploy and manage on Kubernetes. They will be consuming kafka topics.
I have narrowed down the solutions between Apache Druid, Starrocks and of course Clickhouse.
While Druid shows some very neat tricks on real time data storing/indexing, its deployment on kubernetes is really ugly.
Starrocks is not so far, also a bit complex to deploy and manage the "front end and back end" clusters, while Clickhouse makes the k8s deployment a bliss.
What got me in Clikchouse is the kafka topics ingestion, its a lot of work to create and keep managing sql materialized things everytime I need to get some new data. Druid otherwise makes it super cool by almost automatically understanding the json received from the kafka topics.
However I feel like Druid is getting a bit old and with not so much community or development resources or attention. ChatGPT is saving my ass I must admit. Tho, a lot of "what corps are using this tech" sites show that Twitter, Neflix and Reddit itself use Druid. I'm not sure if that is true or old data.
Clickhouse otherwise feels like the industry baby, ultra-seeded and megacorp friendly, but with its own problems like painful cross table joins. Starrocks have a nice fanbase, but falls a bit in the same place as Druid, even being some years ahead of Apache Doris, plus the fact it is mantained by the Linux Foundation.
So, what are your thoughts on these three OLAP databases today?
- truth_seeker 3y agoIt all depends on your workload. BTW, Please have look at ScyllaDB too. You may use Presto or Apache Calcite on top of it for analytics. https://resources.scylladb.com/scylladb-best-practices/olap-or-oltp-why-not-both https://resources.scylladb.com/scylladb-best-practices/olap-...
- babycmd 3y agoCool, I know Scylla but it is an OLTP db isn't it?
- ryadh 3y ago(disclaimer: I work at ClickHouse) Another important dimension I'd consider as well is the broader ecosystem of integrations. In OSS, it's a byproduct of the success of the main project but an often overlooked aspect when choosing a solution. Eg. Here are some of the Kafka integrations options for CH https://clickhouse.com/docs/en/integrations/kafka https://clickhouse.com/docs/en/integrations/kafka
- HermitX 3y agoIn fact, your choice is determined by your business scenario. I believe that overall, among these databases, StarRocks offers the highest flexibility. Particularly, if your business scenario relies on performing data analysis through joins across multiple tables, then StarRocks is almost your only option. Additionally, if you want to work with ANSI SQL, then you might have to use StarRocks.