4 ms·

Parquet + Spark SQL makes me wonder if MapReduce and columnar MPP are really so far apart? I know they're orders of magnitude slower than e.g. Vertica today, b

Parquet + Spark SQL makes me wonder if MapReduce and columnar MPP are really so far apart? I know they're orders of magnitude slower than e.g. Vertica today, but I wonder if there is a fundamental reason for that, or is it just the implementation?

threeseed 12y ago
Both projects are still pretty nice so plenty of optimization work left to be done. Spark is starting to amass all the heavy hitters in big data so no doubt it will get much faster over time.
jandrewrogers 12y ago
Most of the performance differences are in how these systems handle representation, I/O, and execution scheduling. In other words, implementation. The challenge in closing the gap is that the architectures across systems are quite different so you can't just add performance that is largely derived from fundamental architecture. Columnar MPP needs to be carefully defined to answer the question. Some important data models that fit in these architectures operate on data types that are not meaningfully order-able at a mathematical level i.e. you can't sort them. A lot of columnar implementations, and virtually all in open source, assume sortability as a property of the represented data types. tl;dr: Columnar MPP sometimes is not far outside what you can express with Parquet/Spark/MapReduce/etc, just much faster, but there are data models supported in some advanced Columnar MPP systems that are not usefully expressible with that stack. It depends on the platform and the use case.
- rxin 12y ago
  Sorry I don't think the last paragraph you said is true. I dont see any special data types or models that can be modeled by MPP architecture but cannot be modeled in Spark. In short, I don't believe there is much difference at the physical execution level between MPP and Spark. I also don't understand what you mean by orderable. Spark does not require records to be oderable. Maybe you can elaborate?

threeseed 12y ago

Both projects are still pretty nice so plenty of optimization work left to be done. Spark is starting to amass all the heavy hitters in big data so no doubt it will get much faster over time.

jandrewrogers 12y ago

Most of the performance differences are in how these systems handle representation, I/O, and execution scheduling. In other words, implementation. The challenge in closing the gap is that the architectures across systems are quite different so you can't just add performance that is largely derived from fundamental architecture. Columnar MPP needs to be carefully defined to answer the question. Some important data models that fit in these architectures operate on data types that are not meaningfully order-able at a mathematical level i.e. you can't sort them. A lot of columnar implementations, and virtually all in open source, assume sortability as a property of the represented data types. tl;dr: Columnar MPP sometimes is not far outside what you can express with Parquet/Spark/MapReduce/etc, just much faster, but there are data models supported in some advanced Columnar MPP systems that are not usefully expressible with that stack. It depends on the platform and the use case.

rxin 12y ago
Sorry I don't think the last paragraph you said is true. I dont see any special data types or models that can be modeled by MPP architecture but cannot be modeled in Spark. In short, I don't believe there is much difference at the physical execution level between MPP and Spark. I also don't understand what you mean by orderable. Spark does not require records to be oderable. Maybe you can elaborate?