4 ms·
Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS. Then m
by threeseed 11y ago
Druid isn't even close to being the leading solution. The overwhelming majority of places doing big data analytics will be Hadoop/Spark using core HDFS.
Then most places will augment this with a range of database solutions depending on how structured/clean the data is and the various workloads. Cassandra, HBase, MongoDB, Teradata, Oracle, ElasticSearch, Greenplum are all pretty common place in most enterprises.
And of course the ridiculous array of SQL engines on top of HDFS/S3/whatever else e.g. Hive, Spark SQL, Presto, Drill, SAP etc.
- fangjin 11y agoDruid's main value add to the data infrastructure space is around power user-facing data applications at scale. The queries it is best at are OLAP/business intelligence style queries. It isn't really designed to be a general processing tool such as Hadoop or Spark. The open source data space is very complex, and there are many different solutions targeted towards many different use cases. Druid is better than other solutions at some of these use cases, and worse than other solutions at others. I wrote my interpretation of the current open source data landscape here for anyone interested: http://imply.io/post/2015/11/04/big-data-zoo.html http://imply.io/post/2015/11/04/big-data-zoo.html
- deepanchor 11y agoAs someone who hasn't yet had the opportunity to use many of these systems, this was a great high-level overview of how the various systems fit together. Thanks for writing it!
- scott_s 11y agoYour Wikipedia link to "stream processors" is for the wrong kind of stream processors. For a decades, the digital signal processing and graphics worlds have used the streaming abstraction to design programs and hardware. Their applications are typically expressed in pipelines, and do have to continually process data. "Big data" stream processing is obviously related as its a dataflow programming model, but it's still very different in practice. The streaming abstraction is generally more free-form, and not realized directly in hardware. I contrast the two kinds of streaming in Section 2 of a paper from a few years ago: http://www.scott-a-s.com/files/pact2012.pdf http://www.scott-a-s.com/files/pact2012.pdf
- joefkelley 11y agoIf one of these databases included a really good SparkSQL connector (including great predicate push-down), I could see that being hugely beneficial to growth. Part of why people are using these systems is because you can connect in many different types of data. Make your database part of that ecosystem and you can gradually take in more of the analytics workload without the users losing the ability to also throw in some plain old text files or something and join between multiple worlds.
- fangjin 11y agoDruid has a SparkSQL connector: https://github.com/SparklineData/spark-druid-olap https://github.com/SparklineData/spark-druid-olap
- joefkelley 11y agoCool! Yeah I poked around the druid site and didn't find anything originally, but this looks pretty promising. It's hard to tell how full-featured it is without getting hands-on, but I see something like this making Druid much more usable in a lot of analytics environments.
- hbutani 11y agoWe have couple of companies running Tableau on top of this. The deployment is Tableau - Spark ThriftServer(with our extension) - Druid. We push down Slice and Dice and Star Join Queries as Druid Queries; all of Spark SQL is supported with some portions of a Query Plan being executed in Spark. We are working on supporting more Spark UDFs being pushed to Druid, performance improvements, and more coverage for Tableau. Further down we will support Star Schemas where some or all dimensions are not indexed. Happy to discuss specific SQL support or deployment questions. Please reach out to us. - Harish.
- threeseed 11y agoCassandra, MongoDB and ElasticSearch all have SparkSQL DataSource drivers with predicate pushdown support. Most of the limitations in that support is from the database itself and not from the driver. The biggest problems with SparkSQL is simply in its limited support for ANSI SQL. It's getting better with every release but not nearly quick enough.