2 ms·

> Finally, an easy way to query Parquet data without spinning up a local Spark instance What's the burning pain-point here? Spinning up a spark shell is extrem

> Finally, an easy way to query Parquet data without spinning up a local Spark instance What's the burning pain-point here? Spinning up a spark shell is extremely quick and easy.

wenc 5y ago
That's a fair question, and I guess my answer would be "ergonomics". I tend to work in Python not Scala, and there's a certain amount of boilerplate that is necessary to even get Spark functional. For instance, this is a code snippet to spin up local Spark (not Spark Shell) I have stored in a file which I have to look up because there's no way I can commit this to memory: import findspark findspark.init() from pyspark.sql import SparkSession sqlContext = SparkSession.builder.master("local[*]").getOrCreate() df = sqlContext.read.parquet('path-to-file/commentClusters.parquet') Not to mention fussy things you have to do in Spark to alias a table: df.createOrReplaceTempView("tableA") sqlContext.sql("SELECT count(*) from tableA").show() Compare this to these 2 (!) DuckDB commands in the Python REPL that I just ran moments ago: >>> import duckdb >>> duckdb.query("SELECT COUNT(*) FROM '*.parquet'").fetchall() [(2,)]
- nyokodo 5y ago
  > I tend to work in Python not Scala Fair enough. I’m more comfortable in Scala than Python. Have you tried using a local Jupyter notebook? That would seem to tick all the boxes. https://jupyter.org/install https://jupyter.org/install
  - wenc 5y ago
    I do use Jupyter and the boilerplate code above is what I usually deploy in Jupyter. Spark syntax is still just a touch more complicated than necessary for my needs because it's solving the much more general distributed computing problem. (not to mention the impedance mismatch -- Spark is very much a Scala native app, and it feels that way to a Python user). DuckDB on the other hand seems to provide a quick way to interact with local Parquet files in Python, which I appreciate.

> Finally, an easy way to query Parquet data without spinning up a local Spark instance What's the burning pain-point here? Spinning up a spark shell is extremely quick and easy.

wenc 5y ago

That's a fair question, and I guess my answer would be "ergonomics". I tend to work in Python not Scala, and there's a certain amount of boilerplate that is necessary to even get Spark functional. For instance, this is a code snippet to spin up local Spark (not Spark Shell) I have stored in a file which I have to look up because there's no way I can commit this to memory: import findspark findspark.init() from pyspark.sql import SparkSession sqlContext = SparkSession.builder.master("local[*]").getOrCreate() df = sqlContext.read.parquet('path-to-file/commentClusters.parquet') Not to mention fussy things you have to do in Spark to alias a table: df.createOrReplaceTempView("tableA") sqlContext.sql("SELECT count(*) from tableA").show() Compare this to these 2 (!) DuckDB commands in the Python REPL that I just ran moments ago: >>> import duckdb >>> duckdb.query("SELECT COUNT(*) FROM '*.parquet'").fetchall() [(2,)]

nyokodo 5y ago
> I tend to work in Python not Scala Fair enough. I’m more comfortable in Scala than Python. Have you tried using a local Jupyter notebook? That would seem to tick all the boxes. https://jupyter.org/install https://jupyter.org/install
- wenc 5y ago
  I do use Jupyter and the boilerplate code above is what I usually deploy in Jupyter. Spark syntax is still just a touch more complicated than necessary for my needs because it's solving the much more general distributed computing problem. (not to mention the impedance mismatch -- Spark is very much a Scala native app, and it feels that way to a Python user). DuckDB on the other hand seems to provide a quick way to interact with local Parquet files in Python, which I appreciate.