4 ms·

If one of these databases included a really good SparkSQL connector (including great predicate push-down), I could see that being hugely beneficial to growth. P

If one of these databases included a really good SparkSQL connector (including great predicate push-down), I could see that being hugely beneficial to growth. Part of why people are using these systems is because you can connect in many different types of data. Make your database part of that ecosystem and you can gradually take in more of the analytics workload without the users losing the ability to also throw in some plain old text files or something and join between multiple worlds.

fangjin 11y ago
Druid has a SparkSQL connector: https://github.com/SparklineData/spark-druid-olap https://github.com/SparklineData/spark-druid-olap
- joefkelley 11y ago
  Cool! Yeah I poked around the druid site and didn't find anything originally, but this looks pretty promising. It's hard to tell how full-featured it is without getting hands-on, but I see something like this making Druid much more usable in a lot of analytics environments.
  - hbutani 11y ago
    We have couple of companies running Tableau on top of this. The deployment is Tableau - Spark ThriftServer(with our extension) - Druid. We push down Slice and Dice and Star Join Queries as Druid Queries; all of Spark SQL is supported with some portions of a Query Plan being executed in Spark. We are working on supporting more Spark UDFs being pushed to Druid, performance improvements, and more coverage for Tableau. Further down we will support Star Schemas where some or all dimensions are not indexed. Happy to discuss specific SQL support or deployment questions. Please reach out to us. - Harish.
threeseed 11y ago
Cassandra, MongoDB and ElasticSearch all have SparkSQL DataSource drivers with predicate pushdown support. Most of the limitations in that support is from the database itself and not from the driver. The biggest problems with SparkSQL is simply in its limited support for ANSI SQL. It's getting better with every release but not nearly quick enough.
- hbutani 11y ago
  I don't know the specific state of the DataSources you mention, but yes just providing a DataSource only is not enough; you can only push Filters and Projections to the underlying engine. You need to develop Query Rewrite Rules to rewrite Joins, Group Bys, Having, Limit etc to the underlying engine. For example our Rewrite engine for Druid is here: https://github.com/SparklineData/spark-druid-olap/blob/master/src/main/scala/org/apache/spark/sql/sources/druid/DruidPlanner.scala https://github.com/SparklineData/spark-druid-olap/blob/maste... Can you be specific about ANSI SQL compliance requirements. Spark SQL is closing the gap on Hive SQL; both have decent support for analytical queries: Cubes/Rollups/Windowing etc. The only major gap between Spark and Hive SQL I know off is SubQuery predicates(exists/not exists).

fangjin 11y ago

Druid has a SparkSQL connector: https://github.com/SparklineData/spark-druid-olap https://github.com/SparklineData/spark-druid-olap

joefkelley 11y ago
Cool! Yeah I poked around the druid site and didn't find anything originally, but this looks pretty promising. It's hard to tell how full-featured it is without getting hands-on, but I see something like this making Druid much more usable in a lot of analytics environments.
- hbutani 11y ago
  We have couple of companies running Tableau on top of this. The deployment is Tableau - Spark ThriftServer(with our extension) - Druid. We push down Slice and Dice and Star Join Queries as Druid Queries; all of Spark SQL is supported with some portions of a Query Plan being executed in Spark. We are working on supporting more Spark UDFs being pushed to Druid, performance improvements, and more coverage for Tableau. Further down we will support Star Schemas where some or all dimensions are not indexed. Happy to discuss specific SQL support or deployment questions. Please reach out to us. - Harish.

threeseed 11y ago

Cassandra, MongoDB and ElasticSearch all have SparkSQL DataSource drivers with predicate pushdown support. Most of the limitations in that support is from the database itself and not from the driver. The biggest problems with SparkSQL is simply in its limited support for ANSI SQL. It's getting better with every release but not nearly quick enough.

hbutani 11y ago
I don't know the specific state of the DataSources you mention, but yes just providing a DataSource only is not enough; you can only push Filters and Projections to the underlying engine. You need to develop Query Rewrite Rules to rewrite Joins, Group Bys, Having, Limit etc to the underlying engine. For example our Rewrite engine for Druid is here: https://github.com/SparklineData/spark-druid-olap/blob/master/src/main/scala/org/apache/spark/sql/sources/druid/DruidPlanner.scala https://github.com/SparklineData/spark-druid-olap/blob/maste... Can you be specific about ANSI SQL compliance requirements. Spark SQL is closing the gap on Hive SQL; both have decent support for analytical queries: Cubes/Rollups/Windowing etc. The only major gap between Spark and Hive SQL I know off is SubQuery predicates(exists/not exists).