2 ms·
That's a fair question, and I guess my answer would be "ergonomics". I tend to work in Python not Scala, and there's a certain amount of boilerplate that is nec
by wenc 5y ago
That's a fair question, and I guess my answer would be "ergonomics". I tend to work in Python not Scala, and there's a certain amount of boilerplate that is necessary to even get Spark functional. For instance, this is a code snippet to spin up local Spark (not Spark Shell) I have stored in a file which I have to look up because there's no way I can commit this to memory:
import findspark
findspark.init()
from pyspark.sql import SparkSession
sqlContext = SparkSession.builder.master("local[*]").getOrCreate()
df = sqlContext.read.parquet('path-to-file/commentClusters.parquet')
Not to mention fussy things you have to do in Spark to alias a table:
df.createOrReplaceTempView("tableA")
sqlContext.sql("SELECT count(*) from tableA").show()
Compare this to these 2 (!) DuckDB commands in the Python REPL that I just ran moments ago:
>>> import duckdb
>>> duckdb.query("SELECT COUNT(*) FROM '*.parquet'").fetchall()
[(2,)]
- nyokodo 5y ago> I tend to work in Python not Scala Fair enough. I’m more comfortable in Scala than Python. Have you tried using a local Jupyter notebook? That would seem to tick all the boxes. https://jupyter.org/install https://jupyter.org/install
- wenc 5y agoI do use Jupyter and the boilerplate code above is what I usually deploy in Jupyter. Spark syntax is still just a touch more complicated than necessary for my needs because it's solving the much more general distributed computing problem. (not to mention the impedance mismatch -- Spark is very much a Scala native app, and it feels that way to a Python user). DuckDB on the other hand seems to provide a quick way to interact with local Parquet files in Python, which I appreciate.