4 ms·

Blaze: Fast query execution engine for Apache Spark

92 points by sbt567 3y ago

whinvik 3y ago
Any comparisons with Databricks Spark. When we started experimenting with Spark, we initially used AWS EMR. But then the same code was way faster on Databricks than it was on EMR, which resulted in us ditching EMR.
- juliangamble 3y ago
  It would be great to have a comparison to Dataframes and RDDs as well.
  - bllchmbrs 3y ago
    DataFrames are just SQL. There will be no performance difference. RDDs will be worse, so it shouldn't matter. No vectorization, no column processing, lots of serialization and de-serialization. They're basically always slower than DataFrames barring some strange use case.
- esafak 3y ago
  Got numbers?
- AdamProut 3y ago
  Databricks has kept their Photon[1][2] query engine for Spark closed sourced thus far. Unless EMR has made equivalent changes to the Spark runtime they use Databricks should be much faster. Photon brings the standard vectorized execution techniques used in SQL data warehouses for many years to Spark. [1] https://docs.databricks.com/en/clusters/photon.html https://docs.databricks.com/en/clusters/photon.html [2] https://dl.acm.org/doi/10.1145/3514221.3526054 https://dl.acm.org/doi/10.1145/3514221.3526054
  - whinvik 3y ago
    I am a bit hazy about the exact details of how we did it since its been some time, but we definitely did not use Photon as it was too expensive. One of the issues was that we started experimenting with Delta Tables and EMR was horrible in leveraging that.
elromulous 3y ago
For a split second, I thought bazel[0] finally got externally renamed to its true name. [0] https://en.m.wikipedia.org/wiki/Bazel_(software) https://en.m.wikipedia.org/wiki/Bazel_(software)
- deleted 3y ago
  [deleted]
kristjansson 3y ago
Unfortunate name overlap with an under-loved PyData project: https://blaze.pydata.org https://blaze.pydata.org
- deleted 3y ago
  [deleted]
- tomrod 3y ago
  Same. I've always liked the blaze project.
- tjpnz 3y ago
  And Google's version of Bazel.
  - laurentlb 3y ago
    The public version was renamed Bazel because of name conflicts.
zX41ZdbW 3y ago
Interesting, looks like it is just DataFusion engine for Spark. There is a similar project: https://github.com/oap-project/gluten https://github.com/oap-project/gluten - it brings ClickHouse as an engine to Spark.
zhangyt26 3y ago
Photon, velox, and now this. Why would people use spark in the first place other than for legacy application reasons?

92 points by sbt567 3y ago

Any comparisons with Databricks Spark. When we started experimenting with Spark, we initially used AWS EMR. But then the same code was way faster on Databricks than it was on EMR, which resulted in us ditching EMR.

juliangamble 3y ago
It would be great to have a comparison to Dataframes and RDDs as well.
- bllchmbrs 3y ago
  DataFrames are just SQL. There will be no performance difference. RDDs will be worse, so it shouldn't matter. No vectorization, no column processing, lots of serialization and de-serialization. They're basically always slower than DataFrames barring some strange use case.
esafak 3y ago
Got numbers?
AdamProut 3y ago
Databricks has kept their Photon[1][2] query engine for Spark closed sourced thus far. Unless EMR has made equivalent changes to the Spark runtime they use Databricks should be much faster. Photon brings the standard vectorized execution techniques used in SQL data warehouses for many years to Spark. [1] https://docs.databricks.com/en/clusters/photon.html https://docs.databricks.com/en/clusters/photon.html [2] https://dl.acm.org/doi/10.1145/3514221.3526054 https://dl.acm.org/doi/10.1145/3514221.3526054
- whinvik 3y ago
  I am a bit hazy about the exact details of how we did it since its been some time, but we definitely did not use Photon as it was too expensive. One of the issues was that we started experimenting with Delta Tables and EMR was horrible in leveraging that.

elromulous 3y ago

For a split second, I thought bazel[0] finally got externally renamed to its true name. [0] https://en.m.wikipedia.org/wiki/Bazel_(software) https://en.m.wikipedia.org/wiki/Bazel_(software)

deleted 3y ago
[deleted]

kristjansson 3y ago

Unfortunate name overlap with an under-loved PyData project: https://blaze.pydata.org https://blaze.pydata.org

deleted 3y ago
[deleted]
tomrod 3y ago
Same. I've always liked the blaze project.
tjpnz 3y ago
And Google's version of Bazel.
- laurentlb 3y ago
  The public version was renamed Bazel because of name conflicts.

zX41ZdbW 3y ago

Interesting, looks like it is just DataFusion engine for Spark. There is a similar project: https://github.com/oap-project/gluten https://github.com/oap-project/gluten - it brings ClickHouse as an engine to Spark.

zhangyt26 3y ago

Photon, velox, and now this. Why would people use spark in the first place other than for legacy application reasons?