4 ms·

The author is suggesting that Databricks should sell a product with infinitely scalable storage and is queryable via SQL or Python... isn't that exactly what th

The author is suggesting that Databricks should sell a product with infinitely scalable storage and is queryable via SQL or Python... isn't that exactly what they're doing? Databricks has a product called Delta Lake that covers the infinitely scalable storage part. Here's a talk from a Delta Lake user that's writing hundreds of terabyes / up to a petabyte of data a day to Delta Lake: https://youtu.be/vqMuECdmXG0 https://youtu.be/vqMuECdmXG0 Databricks recently rewrote the Spark query engine in C++ (called Photon) and provides a new SQL query interface for data warehouse type developers. Databricks recently set the TPC-DS benchmark record, see this blog post: https://databricks.com/blog/2021/11/02/databricks-sets-official-data-warehousing-performance-record.html https://databricks.com/blog/2021/11/02/databricks-sets-offic... This article doesn't align with my view of reality.

riku_iki 4y ago
> Databricks has a product called Delta Lake that covers the infinitely scalable storage part I think it is just API layer on top of existing storage systems like s3 and hdfs: https://docs.delta.io/latest/delta-storage.html#amazon-s3 https://docs.delta.io/latest/delta-storage.html#amazon-s3
- alexott 4y ago
  No, it’s not just API layer, but significant addon on top of existing tech… It brings transactions, faster queries, time travel, etc.
  - riku_iki 4y ago
    Maybe, but "infinitely scalable storage" part is because of lower lever layer.
    - Epa095 4y ago
      If course, but there are many ways one could make the storage format such that it would not scale nicely with the underlying layer.
      - riku_iki 4y ago
        Like how?
  - FridgeSeal 4y ago
    It’s a parquet file, with some json as metadata to define the version boundaries. That’s all. You too can have transactions if you implement your access layer appropriately. It’s a useful format, but let’s note pretend it’s any more magic than it actually is. I’ve also not noticed any improved performance above what partitioning would give you.
    - Epa095 4y ago
      >You too can have transactions if you implement your access layer appropriately. Yes, we can all "just" make it ourself, still nice when others make it. Personally I very much like that it's conseptually easy and that it builds on an open source format (parquet). I also expect there to be a dragons, both small and large, in actually getting the acid stuff right,so I am happy we can fight them together.
    - alexott 4y ago
      Regarding performance- try version 1.2.x or higher - it includes data skipping, etc. from what was Databricks-only implementation before. Regarding “that’s all” - can you give an estimate how much time it will take to reimplement it from scratch on multiple cloud platforms?
    - manish_gill 4y ago
      Delta metadata based min/max statistics, Parquet row groups and min/max in-file, combined with the right combination of bucketing and sorting, allowed us to actually ignore all partitioning and drastically improve performance on PB scale. The metadata can be a very useful way to do indexing/allows you to skip a lot of things. It's not magic, and you can understand what the _delta_log is doing fairly easily, but I can testify that the performance improvement over partitioning can be achieved.
      - alexott 4y ago
        And some queries, like, max(col) could be answered without touching the data…
majormajor 4y ago
Delta is a fairly recent entrant, and when some of my coworkers evaluated it, it didn't really seem as compelling as Snowflake or BigQuery. Databricks was something our data science department liked, but it didn't have a compelling sell to our (much larger) analytics org.
- mr_toad 4y ago
  > Databricks was something our data science department liked, but it didn't have a compelling sell to our (much larger) analytics org. So they’ll just have to suffer running their non-SQL workloads locally on corporate issued laptops? Or they’re not really going to do data science at all?
  - majormajor 4y ago
    Who's they here? The non-data-science analysts? They were running SQL in the cloud on other products. Non-sql workloads were the domain of data engineers and data science in that org. The data scientists were using Databricks, but data eng was instructed to try to replicate a few core pieces because it was $$$$ as far as "a way to run notebooks and such" went.
- ramraj07 4y ago
  Yeah, we moved away from spark for a reason (you kinda have to babysit it or it’ll crash on anything actually big data). Snowflake took care of things much better than that. Their lake offering is like the worst of both worlds.
  - disgruntledphd2 4y ago
    Spark allows you to distributed model training though, which is really nice when you need it.
- zjaffee 4y ago
  I don't think Delta is meant to compete with Big Query, although it's been a long time since I've used Big Query, I recall upserts not being something you were able to do within BQ and that's essentially the main selling point of delta.
bayan1234 4y ago
I think the author isn't talking about changing the product, but the market positioning and how it's communicated to customers.
- Rimbo 4y ago
  Yeah. The clickbait title suggests that big data tools themselves are going away; the actual contents are that the needless hype for their companies, folded into their sales pitches, is going away.

riku_iki 4y ago

> Databricks has a product called Delta Lake that covers the infinitely scalable storage part I think it is just API layer on top of existing storage systems like s3 and hdfs: https://docs.delta.io/latest/delta-storage.html#amazon-s3 https://docs.delta.io/latest/delta-storage.html#amazon-s3

alexott 4y ago
No, it’s not just API layer, but significant addon on top of existing tech… It brings transactions, faster queries, time travel, etc.
- riku_iki 4y ago
  Maybe, but "infinitely scalable storage" part is because of lower lever layer.
  - Epa095 4y ago
    If course, but there are many ways one could make the storage format such that it would not scale nicely with the underlying layer.
    - riku_iki 4y ago
      Like how?
- FridgeSeal 4y ago
  It’s a parquet file, with some json as metadata to define the version boundaries. That’s all. You too can have transactions if you implement your access layer appropriately. It’s a useful format, but let’s note pretend it’s any more magic than it actually is. I’ve also not noticed any improved performance above what partitioning would give you.
  - Epa095 4y ago
    >You too can have transactions if you implement your access layer appropriately. Yes, we can all "just" make it ourself, still nice when others make it. Personally I very much like that it's conseptually easy and that it builds on an open source format (parquet). I also expect there to be a dragons, both small and large, in actually getting the acid stuff right,so I am happy we can fight them together.
  - alexott 4y ago
    Regarding performance- try version 1.2.x or higher - it includes data skipping, etc. from what was Databricks-only implementation before. Regarding “that’s all” - can you give an estimate how much time it will take to reimplement it from scratch on multiple cloud platforms?
  - manish_gill 4y ago
    Delta metadata based min/max statistics, Parquet row groups and min/max in-file, combined with the right combination of bucketing and sorting, allowed us to actually ignore all partitioning and drastically improve performance on PB scale. The metadata can be a very useful way to do indexing/allows you to skip a lot of things. It's not magic, and you can understand what the _delta_log is doing fairly easily, but I can testify that the performance improvement over partitioning can be achieved.
    - alexott 4y ago
      And some queries, like, max(col) could be answered without touching the data…

majormajor 4y ago

Delta is a fairly recent entrant, and when some of my coworkers evaluated it, it didn't really seem as compelling as Snowflake or BigQuery. Databricks was something our data science department liked, but it didn't have a compelling sell to our (much larger) analytics org.

mr_toad 4y ago
> Databricks was something our data science department liked, but it didn't have a compelling sell to our (much larger) analytics org. So they’ll just have to suffer running their non-SQL workloads locally on corporate issued laptops? Or they’re not really going to do data science at all?
- majormajor 4y ago
  Who's they here? The non-data-science analysts? They were running SQL in the cloud on other products. Non-sql workloads were the domain of data engineers and data science in that org. The data scientists were using Databricks, but data eng was instructed to try to replicate a few core pieces because it was $$$$ as far as "a way to run notebooks and such" went.
ramraj07 4y ago
Yeah, we moved away from spark for a reason (you kinda have to babysit it or it’ll crash on anything actually big data). Snowflake took care of things much better than that. Their lake offering is like the worst of both worlds.
- disgruntledphd2 4y ago
  Spark allows you to distributed model training though, which is really nice when you need it.
zjaffee 4y ago
I don't think Delta is meant to compete with Big Query, although it's been a long time since I've used Big Query, I recall upserts not being something you were able to do within BQ and that's essentially the main selling point of delta.

bayan1234 4y ago

I think the author isn't talking about changing the product, but the market positioning and how it's communicated to customers.

Rimbo 4y ago
Yeah. The clickbait title suggests that big data tools themselves are going away; the actual contents are that the needless hype for their companies, folded into their sales pitches, is going away.