4 ms·

I'm not an expert with either system, but I think the answer is pretty clear: that's not at all an efficient solution. Spark's approach is: do the data however

I'm not an expert with either system, but I think the answer is pretty clear: that's not at all an efficient solution. Spark's approach is: do the data however you want, then use a ridiculous amount of resources getting that data into the format you want it in. You also get a ton of guarantees that you don't get with that Spark system you've proposed. Here's their comparison page[1]. If you're familiar with the technology you're suggesting it should become fairly obvious why Cockroach's nuanced approach to SQL is superior to the Spark SQL solution. For one, Spark SQL doesn't support indices. You can build one, but I'd be surprised if you could get the same general performance out of that system vs. a dedicated system like the one that's built into CockroachDB. I know the hype is to "use NoSQL" but the reality is that there's a limited set of cases where the existing solutions truly make sense. Generally speaking, SQL is the best choice for many companies. That CockroachDB is bringing some new ideas to the SQL space is something that should be lauded. And I think they're doing it in a way that can't really be replicated easily or efficiently by current systems (which is perhaps why they're receiving so much attention.) [1] https://www.cockroachlabs.com/docs/stable/cockroachdb-in-comparison.html https://www.cockroachlabs.com/docs/stable/cockroachdb-in-com...

elvinyung 9y ago
I... agree. But my argument was not "use NoSQL". I never suggested to use NoSQL. I suggested to use Spark as the compute layer. Spark has a sophisticated [1] query engine, which seems perfectly capable of pruning partitions and pushing down index scans into data sources. Yes, Spark can't do writes, so you'll still have to build a transactional KV store that sits underneath the query layer, but you won't have to implement SQL from scratch. (This seems to be similar to the approach taken by e.g. SnappyData [2] and Splice Machine [3].) [1] https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html https://databricks.com/blog/2016/05/23/apache-spark-as-a-com... [2] http://snappydatainc.github.io/snappydata/ http://snappydatainc.github.io/snappydata/ [3] http://doc.splicemachine.com/2.0/GettingStarted/ArchitectureOvervew.html http://doc.splicemachine.com/2.0/GettingStarted/Architecture...
- plamb 9y ago
  SnappyData employee here -- This is essentially what we did. The main difference is that we already had a decade old transactional K/V store, that, over time morphed into a more full fledged in-memory database. That is what we integrated with Spark versus rolling a new database. The SQL layer in this database (GemFire/Geode) already had a number of optimizations we could use to speed up Spark SQL queries, even over the native Spark cache. Like some of the other comments in this thread, the idea was to provide all the guarantees of a OLTP store (HA, ACID, Scalability, Mutations etc) with the powerful analytic capabilities of Spark.

elvinyung 9y ago

I... agree. But my argument was not "use NoSQL". I never suggested to use NoSQL. I suggested to use Spark as the compute layer. Spark has a sophisticated [1] query engine, which seems perfectly capable of pruning partitions and pushing down index scans into data sources. Yes, Spark can't do writes, so you'll still have to build a transactional KV store that sits underneath the query layer, but you won't have to implement SQL from scratch. (This seems to be similar to the approach taken by e.g. SnappyData [2] and Splice Machine [3].) [1] https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html https://databricks.com/blog/2016/05/23/apache-spark-as-a-com... [2] http://snappydatainc.github.io/snappydata/ http://snappydatainc.github.io/snappydata/ [3] http://doc.splicemachine.com/2.0/GettingStarted/ArchitectureOvervew.html http://doc.splicemachine.com/2.0/GettingStarted/Architecture...

plamb 9y ago
SnappyData employee here -- This is essentially what we did. The main difference is that we already had a decade old transactional K/V store, that, over time morphed into a more full fledged in-memory database. That is what we integrated with Spark versus rolling a new database. The SQL layer in this database (GemFire/Geode) already had a number of optimizations we could use to speed up Spark SQL queries, even over the native Spark cache. Like some of the other comments in this thread, the idea was to provide all the guarantees of a OLTP store (HA, ACID, Scalability, Mutations etc) with the powerful analytic capabilities of Spark.