There's overhead in the flexibility, but it seems that you'll have to do the same kind of tuning all over again if you build your own query engine from scratch.

There's overhead in the flexibility, but it seems that you'll have to do the same kind of tuning all over again if you build your own query engine from scratch. And frankly it seems (at least at a high level) that Cockroach is heading towards building the exact same kind of dataflow engine anyways[1]. Spark has some support for SQL DML, and to me it actually mostly seems like a concern for the underlying KV layer to provide ACID transactions, and expose them to Spark through the Data Sources API. [1] https://www.cockroachlabs.com/blog/local-and-distributed-processing-in-cockroachdb/ https://www.cockroachlabs.com/blog/local-and-distributed-pro...