4 ms·

Performance is only one part of the story. The major advantage Snowflake (and to some extent Presto/Trino) brings to the table is it's pretty much plug and play

by hiyer 5y ago

Performance is only one part of the story. The major advantage Snowflake (and to some extent Presto/Trino) brings to the table is it's pretty much plug and play. Spark OTOH usually requires a lot of tweaking to work reliably for your workloads.

EdwardDiego 5y ago
Very much true. I saw a joke tweet recently something along the lines of - It's amazing how many data engineering scaling issues these days are being solved by just paying Snowflake more money. Spark does take a lot of tuning, but then I'm guessing Databricks offer that service as part of your licensing fee? (I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it)
- hiyer 5y ago
  > I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it They have some proprietary features like DBIO [1]. They also have some cloud-specific features like storage autoscaling [2] that would not be available in OSS Spark. Even Delta Lake [3] used to be proprietary, but I suspect the rise of open-source frameworks like Iceberg led them to open-source it. Shameless plug - when working at a since-shutdown competitor to Databricks, I'd come up with storage autoscaling long before them [4], so it's not unlikely that they were "inspired" by us :-) . 1. https://docs.databricks.com/spark/latest/spark-sql/dbio-commit.html https://docs.databricks.com/spark/latest/spark-sql/dbio-comm... 2. https://databricks.com/blog/2017/12/01/transparent-autoscaling-of-instance-storage.html https://databricks.com/blog/2017/12/01/transparent-autoscali... 3. https://delta.io/ https://delta.io/ 4. https://www.qubole.com/blog/auto-scaling-in-qubole-with-aws-elastic-block-storage/ https://www.qubole.com/blog/auto-scaling-in-qubole-with-aws-...
  - glogla 5y ago
    The open source Delta is not a replacement for the real thing - they did not include features like optimizing small files (small file problem is well known in big data, and much more of a problem once streaming gets involved) and others. It is more of a demo of the real thing. Which does not stop them from repeating everywhere how open they are, of course. EDIT: the delta also still keeps partitioning information in the hive metastore, while iceberg keeps it in storage, making it a far superior design. Adopting iceberg is harder due to third party tools like AWS Redshift not supporting it - you have to go 100 % of the way.
    - saj1th 5y ago
      >the delta also still keeps partitioning information in the hive metastore, while iceberg keeps it in storage, making it a far superior design. Check out https://github.com/delta-io/delta/blob/3ffb30d86c6acda9b59b9b029e6e945992ef4f42/PROTOCOL.md https://github.com/delta-io/delta/blob/3ffb30d86c6acda9b59b9... when you get a chance. You don't need hive metastore to query delta tables since all metadata for a Delta table is stored alongside the data >they did not include features like optimizing small files For optimizing small files, you could run https://docs.delta.io/latest/best-practices.html#compact-files https://docs.delta.io/latest/best-practices.html#compact-fil...
rdeboo 5y ago
I think the comparison was Snowflake vs Databricks SQL. Databricks SQL is a PaaS service just like Snowflake. Also, it uses their Photon engine, which is a proprietary engine written in C++. It is not Spark.
- hiyer 5y ago
  I'm aware that Databricks is a PaaS service, but what Databricks runs under the hood is Spark (with a few proprietary extensions). So your jobs/queries do require some tuning just like with OS Spark. Spark has had SQL engines (SparkSQL/Hive on Spark) for a long time. Photon is just a new, faster one. Photon tasks also run on Spark executors only, so it's not independent of Spark[1]. Also, while it's proprietary now, I wouldn't be surprised if Databricks open-sources it in the future, like they did with Delta Lake. 1. https://databricks.com/blog/2021/06/17/announcing-photon-public-preview-the-next-generation-query-engine-on-the-databricks-lakehouse-platform.html https://databricks.com/blog/2021/06/17/announcing-photon-pub...

by hiyer 5y ago

EdwardDiego 5y ago

Very much true. I saw a joke tweet recently something along the lines of - It's amazing how many data engineering scaling issues these days are being solved by just paying Snowflake more money. Spark does take a lot of tuning, but then I'm guessing Databricks offer that service as part of your licensing fee? (I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it)

hiyer 5y ago
> I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it They have some proprietary features like DBIO [1]. They also have some cloud-specific features like storage autoscaling [2] that would not be available in OSS Spark. Even Delta Lake [3] used to be proprietary, but I suspect the rise of open-source frameworks like Iceberg led them to open-source it. Shameless plug - when working at a since-shutdown competitor to Databricks, I'd come up with storage autoscaling long before them [4], so it's not unlikely that they were "inspired" by us :-) . 1. https://docs.databricks.com/spark/latest/spark-sql/dbio-commit.html https://docs.databricks.com/spark/latest/spark-sql/dbio-comm... 2. https://databricks.com/blog/2017/12/01/transparent-autoscaling-of-instance-storage.html https://databricks.com/blog/2017/12/01/transparent-autoscali... 3. https://delta.io/ https://delta.io/ 4. https://www.qubole.com/blog/auto-scaling-in-qubole-with-aws-elastic-block-storage/ https://www.qubole.com/blog/auto-scaling-in-qubole-with-aws-...
- glogla 5y ago
  The open source Delta is not a replacement for the real thing - they did not include features like optimizing small files (small file problem is well known in big data, and much more of a problem once streaming gets involved) and others. It is more of a demo of the real thing. Which does not stop them from repeating everywhere how open they are, of course. EDIT: the delta also still keeps partitioning information in the hive metastore, while iceberg keeps it in storage, making it a far superior design. Adopting iceberg is harder due to third party tools like AWS Redshift not supporting it - you have to go 100 % of the way.
  - saj1th 5y ago
    >the delta also still keeps partitioning information in the hive metastore, while iceberg keeps it in storage, making it a far superior design. Check out https://github.com/delta-io/delta/blob/3ffb30d86c6acda9b59b9b029e6e945992ef4f42/PROTOCOL.md https://github.com/delta-io/delta/blob/3ffb30d86c6acda9b59b9... when you get a chance. You don't need hive metastore to query delta tables since all metadata for a Delta table is stored alongside the data >they did not include features like optimizing small files For optimizing small files, you could run https://docs.delta.io/latest/best-practices.html#compact-files https://docs.delta.io/latest/best-practices.html#compact-fil...

rdeboo 5y ago

I think the comparison was Snowflake vs Databricks SQL. Databricks SQL is a PaaS service just like Snowflake. Also, it uses their Photon engine, which is a proprietary engine written in C++. It is not Spark.

hiyer 5y ago
I'm aware that Databricks is a PaaS service, but what Databricks runs under the hood is Spark (with a few proprietary extensions). So your jobs/queries do require some tuning just like with OS Spark. Spark has had SQL engines (SparkSQL/Hive on Spark) for a long time. Photon is just a new, faster one. Photon tasks also run on Spark executors only, so it's not independent of Spark[1]. Also, while it's proprietary now, I wouldn't be surprised if Databricks open-sources it in the future, like they did with Delta Lake. 1. https://databricks.com/blog/2021/06/17/announcing-photon-public-preview-the-next-generation-query-engine-on-the-databricks-lakehouse-platform.html https://databricks.com/blog/2021/06/17/announcing-photon-pub...