3 ms·
Has it become easier to run ad hoc queries with spark? I remember a year ago that the only available solution was the job server by ooyala. Which seems to be a
by TheGuyWhoCodes 11y ago
Has it become easier to run ad hoc queries with spark?
I remember a year ago that the only available solution was the job server by ooyala.
Which seems to be a missing feature of core Spark, and isn't something I was willing to bet my product on.
Datastax evangelized people to use Spark to run queries over Cassandra but it looks so awkward and time consuming to copy jars around to the master, basically you need a dev ops team to this and even more scriptology for production.
- mydpy 11y agoInfinitely. Yes. The addition of Dataframes in 1.3 and numerous enhancements to SparkSQL have made it as easy as: val myDF = sqlContext.read.format("com.databricks.spark.csv") //allows you to read a csv file (for simplicity) .option("header", "true") .option("delimiter", "\\t") .option("mode", "PERMISSIVE") .option("inferSchema", "true") .load(filename) .where(filter_query) .cache() myDF.registerTempTable("myDF") sqlContext.sql("SELECT COUNT(*) FROM myDF").show() sqlContext.sql("SELECT COUNT(*) FROM myDF where filter_query").show()
- eip 11y ago>to run queries over Cassandra Why not just use Presto? It gives you basically full SQL capability for Cassandra with minimal effort.
- TheGuyWhoCodes 11y agoPresto seems like a good product, haven't really test it too much tho. When we started to look at Cassandra 1.5 years ago, Spark integration was all the rage, and the premise it was the missing link for doing analytic on Cassandra. We tested it thoroughly and came to the conclusion that is wasn't a mature enough solution.