3 ms·
I don't know the specific state of the DataSources you mention, but yes just providing a DataSource only is not enough; you can only push Filters and Projection
by hbutani 11y ago
I don't know the specific state of the DataSources you mention, but yes just providing a DataSource only is not enough; you can only push Filters and Projections to the underlying engine. You need to develop Query Rewrite Rules to rewrite Joins, Group Bys, Having, Limit etc to the underlying engine. For example our Rewrite engine for Druid is here: https://github.com/SparklineData/spark-druid-olap/blob/master/src/main/scala/org/apache/spark/sql/sources/druid/DruidPlanner.scala https://github.com/SparklineData/spark-druid-olap/blob/maste...
Can you be specific about ANSI SQL compliance requirements. Spark SQL is closing the gap on Hive SQL; both have decent support for analytical queries: Cubes/Rollups/Windowing etc. The only major gap between Spark and Hive SQL I know off is SubQuery predicates(exists/not exists).