4 ms·

(Disclaimer: I work at Databricks.) I really believe that Spark's "more-than-SQL" query interface is how things should be. SQL is of course the gold standard a

(Disclaimer: I work at Databricks.) I really believe that Spark's "more-than-SQL" query interface is how things should be. SQL is of course the gold standard and probably represents north of 90% of analytic workloads, but there a lot of queries that (especially for us programmers) are much easier to express procedurally/functionally, rather than purely declaratively.

strictfp 9y ago
But can we do that in am efficient manner?
oldandtired 9y ago
Many years ago, I worked on a project that required changes made to a reporting system that the original author had used procedural means to do data selection and updates. It was a highly convoluted "mess". The section I had to update was going to take some time to analyse the code to ensure that the specific updates would work. I looked at what the code was doing and restructured into a set of queries that would do the same task in a simple sequence. It turned out that this simplification also sped up the entire update process by at least 1 if not 2 orders of magnitude. There are few cases where processing the data procedurally is better than using SQL queries. Most time that it has been done procedurally, I have found that there are better ways to do what you require by judicious thinking and using SQL (the bronze standard). I should say that my definition of procedural processing involves opening queries and processing tuples one by one. If you have a different definition then please let me know.
- elvinyung 9y ago
  I mostly agree, but I think it fundamentally relies on a good query plan optimizer to figure out the proper order to execute joins, rewriting predicates, etc. So I think I take the position that the machine is not perfect [1], and doesn't always provide a perfect abstraction of a fast declarative answering interface. Sometimes you really do need to tell it how to access the data. This is why, for example, some SQL query engines let the user provide join hints. That said, I do agree that procedural queries are mostly a quick fix, and not very future-proof (against future improvements to the query engine). And FWIW, Spark's DataFrame API [2] is not actually that procedural; it lets you specify something that feels like a direct query execution plan, but actually still gets optimized underneath. [1] http://www.vldb.org/pvldb/vol9/p204-leis.pdf http://www.vldb.org/pvldb/vol9/p204-leis.pdf [2] https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset https://spark.apache.org/docs/latest/api/scala/index.html#or...

strictfp 9y ago

But can we do that in am efficient manner?

oldandtired 9y ago

Many years ago, I worked on a project that required changes made to a reporting system that the original author had used procedural means to do data selection and updates. It was a highly convoluted "mess". The section I had to update was going to take some time to analyse the code to ensure that the specific updates would work. I looked at what the code was doing and restructured into a set of queries that would do the same task in a simple sequence. It turned out that this simplification also sped up the entire update process by at least 1 if not 2 orders of magnitude. There are few cases where processing the data procedurally is better than using SQL queries. Most time that it has been done procedurally, I have found that there are better ways to do what you require by judicious thinking and using SQL (the bronze standard). I should say that my definition of procedural processing involves opening queries and processing tuples one by one. If you have a different definition then please let me know.

elvinyung 9y ago
I mostly agree, but I think it fundamentally relies on a good query plan optimizer to figure out the proper order to execute joins, rewriting predicates, etc. So I think I take the position that the machine is not perfect [1], and doesn't always provide a perfect abstraction of a fast declarative answering interface. Sometimes you really do need to tell it how to access the data. This is why, for example, some SQL query engines let the user provide join hints. That said, I do agree that procedural queries are mostly a quick fix, and not very future-proof (against future improvements to the query engine). And FWIW, Spark's DataFrame API [2] is not actually that procedural; it lets you specify something that feels like a direct query execution plan, but actually still gets optimized underneath. [1] http://www.vldb.org/pvldb/vol9/p204-leis.pdf http://www.vldb.org/pvldb/vol9/p204-leis.pdf [2] https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.Dataset https://spark.apache.org/docs/latest/api/scala/index.html#or...