4 ms·
val spark: SparkSession = SparkSession.builder .appName("Example") .master("local[*]") .getOrCreate() val df = spark.read.parquet("/m
by crabbone 3y ago
val spark: SparkSession = SparkSession.builder
.appName("Example")
.master("local[*]")
.getOrCreate()
val df = spark.read.parquet("/mnt/nyctaxi/parquet")
.groupBy("passenger_count")
.sum("fare_amount")
.orderBy("passenger_count")
df.show()
That's the first listing in this book. This is a disaster... Query language implemented in a pug-ugly other language which has to rely on two other query languages to function. This is the worst sales pitch in the history of sale pitches. Maybe ever.
- kristjansson 3y agoLooks like it gets quite quickly to logical plans and the actual business of query engines. The surface representation of a query is almost immaterial to the topic of the book.
- polishdude20 3y agoAgreed, I was like, oh cool I get to learn about query engines with basic SQL with postgres or MySQL.
- chrisjc 3y agoThat's the first "dataframe" listing in the book. The first listing is the SQL examples. Both SQL and dataframes have to be prepared for the query engine. You can read the whole book without ever needing to take into consideration the pug-ugly dataframe content. I don't disagree that dataframes are often/most times horrible to look at.