3 ms·
Cassandra, MongoDB and ElasticSearch all have SparkSQL DataSource drivers with predicate pushdown support. Most of the limitations in that support is from the d
by threeseed 11y ago
Cassandra, MongoDB and ElasticSearch all have SparkSQL DataSource drivers with predicate pushdown support. Most of the limitations in that support is from the database itself and not from the driver.
The biggest problems with SparkSQL is simply in its limited support for ANSI SQL. It's getting better with every release but not nearly quick enough.
- hbutani 11y agoI don't know the specific state of the DataSources you mention, but yes just providing a DataSource only is not enough; you can only push Filters and Projections to the underlying engine. You need to develop Query Rewrite Rules to rewrite Joins, Group Bys, Having, Limit etc to the underlying engine. For example our Rewrite engine for Druid is here: https://github.com/SparklineData/spark-druid-olap/blob/master/src/main/scala/org/apache/spark/sql/sources/druid/DruidPlanner.scala https://github.com/SparklineData/spark-druid-olap/blob/maste... Can you be specific about ANSI SQL compliance requirements. Spark SQL is closing the gap on Hive SQL; both have decent support for analytical queries: Cubes/Rollups/Windowing etc. The only major gap between Spark and Hive SQL I know off is SubQuery predicates(exists/not exists).