9 ms·

Databricks is an RDBMS

148 points by georgewfraser 6y ago

anonu 6y ago
So is Databricks like AWS RDS with some Lambda functions built around it?
- pbourke 6y ago
  No, Databricks is a distribution of Apache Spark with some value-added features such as the Delta Lake data format and a clean UI for hosting notebooks, doing cluster admin, etc
  - fs111 6y ago
    delta lake is open source too btw
    - rymurr 6y ago
      only in name. check out the lack of traction for a lot of PRs on https://github.com/delta/delta-oss https://github.com/delta/delta-oss . Iceberg (https://iceberg.apache.org https://iceberg.apache.org) is comparable but actually OSS.
      - snidane 6y ago
        https://github.com/delta/delta-oss https://github.com/delta/delta-oss link seems to be broken.
        ywywyw 6y ago
        this is the right url: https://github.com/delta-io/delta https://github.com/delta-io/delta
bpodgursky 6y ago
Hmm, I agree there's a market here, but I don't know why I wouldn't just use Snowflake or Bigquery if what I really wanted was a big-data RDBMS. Everyone I know who uses Databricks (and they all like it) use it as hosted Spark with S3 integrations, or write... directly to Snowflake. I'm a little skeptical they're going to get traction as a true data lake model
- MikeDelta 6y ago
  Their Delta also has SSD caching, which turns out to be logic that stores a local copy of the file you queried for faster re-query. Going to call my lru cache function like that as well... My company loves them, I think they only do a few things good of which marketing the best, and are not worth the money for data science teams with devops skills. Happy to hear from others if I am wrong.
  - pram 6y ago
    Depends. If you only have a couple workspaces, then no. It's worth the money for a company with lots of data science teams, and one team who is janitoring all the workspace infra. Our company has 20 workspaces for 10 teams already and it would be a nightmare if we expected everyone to fix and manage their own AWS stuff.
- georgewfraser 6y ago
  It's a lot simpler to use a single system as both your data lake, and your data warehouse. As Databricks gets better and better at the core data warehouse features, it becomes feasible to use it for both. Meanwhile, Snowflake and BQ are coming from the other direction, implementing data lake features. AWS strategy seems to be, just make it easier to have 2 systems and move data back and forth.
- willvarfar 6y ago
  The reason snowflake has such a high market cap is because it’s customers aren’t paying much now, but they’ll be paying and paying monthly forever. It’s lock-in on a massive scale. Delta lake is something you can run on data you feel you still have some semblance of control over.
ACow_Adonis 6y ago
data lake, delta lake, lake house. snowflake, snowpark. cloud, data warehouse. I just want to take a second to thank these companies for trying to turn my profession (data science + analytics) into the living hell of your every day enterprise and tech culture cluster-fuck (pun not intended). I'm still coming to terms with the fact that there's actually a technology called Kafka... /sorry, I'm just particularly bitter after having to do some databricks training yesterday... it was basically 80% trying to rote the sales literature about why they're so great/enterprisy and parroting company and platform specific jargon. I don't like seeing my profession turn into an obsession with tech and platforms when 99.9% of companies and people can't reason properly or operate their current tools/resources efficiently. obviously just my own opinion.
- arafa 6y ago
  The training is quite heavy on the sales pitch, I agree (and expensive). There were some useful bits if you dig around though.
- MikeDelta 6y ago
  I read a wonderful comment in another thread the other day. It was about the obsession of devs to collect a whole range of technologies on their resume. It went something like: "When I hire a carpenter I won't hire him for what he has in his toolbox. I want to know what he can do with only a hammer and chisel (= Linux machine). The rest he can learn." I will try to look up the link and obviously 'he' can be 'she' as well.
  - dominotw 6y ago
    > I want to know what he can do with only a hammer and chisel (= Linux machine). The rest he can learn. I don't know about this, I would be a bit hesitant if the carpenter to build my kitchen cabinets only knows how to use hammer and a chisel.
- kthejoker2 6y ago
  I reformatted this comment as found poetry on LinkedIn https://www.linkedin.com/pulse/overheard-hacker-news-kafkaesque-kyle-hale/?published=t https://www.linkedin.com/pulse/overheard-hacker-news-kafkaes... I hope you don't mind, it was, to my eye and ear, an instant classic.
willvarfar 6y ago
The key gap as I see it is that databricks doesn’t support multi-table transactions. And that is why you can’t treat it as though it as a drop in functional replacement for an rdbms. (Why no vector clocks in the manifest files or something?)
- georgewfraser 6y ago
  Interestingly, BigQuery is also missing multi-table transactions. There are ways to live without this feature, but I agree it's a gap.
- rymurr 6y ago
  Check out https://projectnessie.org https://projectnessie.org it adds multi-table transactions to Databricks Delta. disclaimer: an author of Nessie
kfk 6y ago
These are great innovations but can we please take a moment to realize 99% of companies are still stuck with a blend of excels, access and sql servers? Why is adoption of this new tech so poor? Maybe it has something to do with the amount of confusion all the sales pitches about data lakes, big data, ai and company are generating
- t0mas88 6y ago
  Because most data isn't "big" data. If it fits in Excel on a laptop, why bother to roll out a distributed data lake system like Spark with all the associated ops work.
  - noxvilleza 6y ago
    ... and if it doesn't fit on Excel on a laptop, the natural path is just some database running on a desktop, then a server, then a beefier server. It's actually amazing how far just something like postgres will scale for your data needs.
- phoe-krk 6y ago
  > Why is adoption of this new tech so poor? Because it's unnecessary to those 99% of companies. If company data fits in an Excel spreadsheet, Access database, or a single MySQL/Postgres instance, then introducing all this new tech with all of the associated costs and little return gain is a net loss.
  - baskire 6y ago
    Might even see negative gain. Often time the query perf per machine goes down as you move from a single node, indexed, postgresql To a multi node unstructured data. If you don’t have so much data or unstructured data. Why use the more complex and generally slower per FLOP solution.
  - kfk 6y ago
    Some of this new tech is simpler to run than MySQL/Postgres like S3/Spectrum. Excel is ok for last mile type of analytics but it doesn't work if you are looking for serious insights. Excel also generates a significant amount of unnecessary manual work and data quality issues. It's not so black and white, what I am saying is that the analytics tech industry direction doesn't seem to address the real issues.
rymurr 6y ago
Check out https://searchdatamanagement.techtarget.com/news/252495619/Apache-Iceberg-rising-for-new-cloud-data-lake-platforms https://searchdatamanagement.techtarget.com/news/252495619/A... the convergence between data lake and data warehouse idea is starting to spread rapidly.
jpau 6y ago
I'm deeply disappointed in Databricks as an RDBMS. As a DS/DE, there's a lot to love (not all, but a lot). The easy provision of Spark clusters. The jobs API. DeltaLake (mostly). Easy notebooks (please don't create a prod system from these..). And Spark itself continues to improve, albeit in an increasingly crowded field. But I've worked closely with BigCo SQL analysts on Azure Databricks, and their experience was terrible. For example: - You cannot browse the data structure without an active cluster - Starting a cluster can take ~5 minutes and, since you missed that moment, you may not submit your first query until 10-15 minutes. - The SQL error messages are often (perhaps usually?) nonsense, so you have to operate without them. - An unfortunate amount of downtime, followed by bizarre excuses. - It's so darn slow, relative to equivalent queries on BigQuery or Snowflake. - Even submitting a query can take a weird amount of time. If Databricks-as-an-RDBMS were competing against Teradata, sure, let's have a chat. But we're in 2021, and there's just no comparing the experience of the SQL analyst on Databricks-as-an-RDBMS vs. Snowflake/BigQuery. I'm excited for the potential of Snowflake's SnowPark (though know little about it). Calling UDFs from SQL means you can create great features for SQL analysts, provided that they can build the momentum to need it.
- nattaylor 6y ago
  I got excellent performance in Databricks with well partitioned Parquet and Spark 2.4. What is making the queries slow? Data scanning?
  - jpau 6y ago
    They use DeltaLake + Spark 3.0, and are mostly careful to partition well. Their datasets are small. Most tables are ~50GB, the odd table up to ~2TB. The clusters typically are nothing shabby for this size, defaults to ~[4-12]x32GB. The queries that I have seen are typically not written well. Think view-on-view-on-view (there's a BigCo policy against them materialising data..), and where the filter is applied in the last step. The stuff of horrors, but something I've seen in more-than-one-BigCo. But we have compared some of those same queries on BigQuery vs. Databricks, and, I don't know if BigQuery's execution optimiser is better? Or if the BigQuery storage is better organising the data? Or if BigQuery is simply throwing more resource their way?
drej 6y ago
I have seen several deployments of Databricks (including first hand experience) and... most use cases could be better served by Postgres (or Redshift, Athena, Snowflake for larger scale). It has honestly been such an overkill for so many workloads, it was quite astonishing. I've seen people move from Excel to Spark... to handle the same volume of data. That's obviously not Databricks' fault, but their PR is pretty much "please do all your data work in our product, it's well suited for it". Yes, it's very good if you don't like setting up clusters (few do/can) and the UI is rather useful for getting up and running (not so much for writing code though). But you need to really understand the platform before adopting it. Please, please don't just adopt it because it's popular.
wokwokwok 6y ago
This 100 word article says everything about databricks and the whole ecosystem of content free data engineering blog posts. It’s all buzz words, no content and please signup now. How so this link so we’ll upvoted? Did people just see “databricks” and think ooo... I have an opinion on that? Read the article before you upvote folks; this one doesn’t deserve your time.
bertomart 6y ago
Databricks is not an RDBMS...it's a Unified Analytics Platform (their words i know), but that really describes it well. It's been really liberating to having easy access to versioned data at your fingertips when iterating on models...and also being able to track models and share work. It does have some warts but it's much better than any other single platform I've used for data science/ML
LukeEF 6y ago
Were there some serious mental gymnastics coming from Fivetran yesterday, or am I missing something? The CEO had a data lake hit piece in Forbes ('The Business Case For Ditching Your Data Lake'[1]) and almost simultaneously authored this 'Databricks' Delta Lake is not a data lake' blog. Felt like the CEO wanted to make a strong point, but has a relationship with Databricks that he didn't want to undermine, so did some reclassification gymnastics to make it all fit. But I am probably too cynical. Finally, I beg for mercy from the 'lakehouse'. It is a bridge too far ;-) [1] https://www.forbes.com/sites/forbestechcouncil/2021/02/01/the-business-case-for-ditching-your-data-lake/ https://www.forbes.com/sites/forbestechcouncil/2021/02/01/th...

148 points by georgewfraser 6y ago

anonu 6y ago

So is Databricks like AWS RDS with some Lambda functions built around it?

pbourke 6y ago
No, Databricks is a distribution of Apache Spark with some value-added features such as the Delta Lake data format and a clean UI for hosting notebooks, doing cluster admin, etc
- fs111 6y ago
  delta lake is open source too btw
  - rymurr 6y ago
    only in name. check out the lack of traction for a lot of PRs on https://github.com/delta/delta-oss https://github.com/delta/delta-oss . Iceberg (https://iceberg.apache.org https://iceberg.apache.org) is comparable but actually OSS.
    - snidane 6y ago
      https://github.com/delta/delta-oss https://github.com/delta/delta-oss link seems to be broken.
      - ywywyw 6y ago
        this is the right url: https://github.com/delta-io/delta https://github.com/delta-io/delta

bpodgursky 6y ago

Hmm, I agree there's a market here, but I don't know why I wouldn't just use Snowflake or Bigquery if what I really wanted was a big-data RDBMS. Everyone I know who uses Databricks (and they all like it) use it as hosted Spark with S3 integrations, or write... directly to Snowflake. I'm a little skeptical they're going to get traction as a true data lake model

MikeDelta 6y ago
Their Delta also has SSD caching, which turns out to be logic that stores a local copy of the file you queried for faster re-query. Going to call my lru cache function like that as well... My company loves them, I think they only do a few things good of which marketing the best, and are not worth the money for data science teams with devops skills. Happy to hear from others if I am wrong.
- pram 6y ago
  Depends. If you only have a couple workspaces, then no. It's worth the money for a company with lots of data science teams, and one team who is janitoring all the workspace infra. Our company has 20 workspaces for 10 teams already and it would be a nightmare if we expected everyone to fix and manage their own AWS stuff.
georgewfraser 6y ago
It's a lot simpler to use a single system as both your data lake, and your data warehouse. As Databricks gets better and better at the core data warehouse features, it becomes feasible to use it for both. Meanwhile, Snowflake and BQ are coming from the other direction, implementing data lake features. AWS strategy seems to be, just make it easier to have 2 systems and move data back and forth.
willvarfar 6y ago
The reason snowflake has such a high market cap is because it’s customers aren’t paying much now, but they’ll be paying and paying monthly forever. It’s lock-in on a massive scale. Delta lake is something you can run on data you feel you still have some semblance of control over.

ACow_Adonis 6y ago

data lake, delta lake, lake house. snowflake, snowpark. cloud, data warehouse. I just want to take a second to thank these companies for trying to turn my profession (data science + analytics) into the living hell of your every day enterprise and tech culture cluster-fuck (pun not intended). I'm still coming to terms with the fact that there's actually a technology called Kafka... /sorry, I'm just particularly bitter after having to do some databricks training yesterday... it was basically 80% trying to rote the sales literature about why they're so great/enterprisy and parroting company and platform specific jargon. I don't like seeing my profession turn into an obsession with tech and platforms when 99.9% of companies and people can't reason properly or operate their current tools/resources efficiently. obviously just my own opinion.

arafa 6y ago
The training is quite heavy on the sales pitch, I agree (and expensive). There were some useful bits if you dig around though.
MikeDelta 6y ago
I read a wonderful comment in another thread the other day. It was about the obsession of devs to collect a whole range of technologies on their resume. It went something like: "When I hire a carpenter I won't hire him for what he has in his toolbox. I want to know what he can do with only a hammer and chisel (= Linux machine). The rest he can learn." I will try to look up the link and obviously 'he' can be 'she' as well.
- dominotw 6y ago
  > I want to know what he can do with only a hammer and chisel (= Linux machine). The rest he can learn. I don't know about this, I would be a bit hesitant if the carpenter to build my kitchen cabinets only knows how to use hammer and a chisel.
kthejoker2 6y ago
I reformatted this comment as found poetry on LinkedIn https://www.linkedin.com/pulse/overheard-hacker-news-kafkaesque-kyle-hale/?published=t https://www.linkedin.com/pulse/overheard-hacker-news-kafkaes... I hope you don't mind, it was, to my eye and ear, an instant classic.

willvarfar 6y ago

The key gap as I see it is that databricks doesn’t support multi-table transactions. And that is why you can’t treat it as though it as a drop in functional replacement for an rdbms. (Why no vector clocks in the manifest files or something?)

georgewfraser 6y ago
Interestingly, BigQuery is also missing multi-table transactions. There are ways to live without this feature, but I agree it's a gap.
rymurr 6y ago
Check out https://projectnessie.org https://projectnessie.org it adds multi-table transactions to Databricks Delta. disclaimer: an author of Nessie

kfk 6y ago

These are great innovations but can we please take a moment to realize 99% of companies are still stuck with a blend of excels, access and sql servers? Why is adoption of this new tech so poor? Maybe it has something to do with the amount of confusion all the sales pitches about data lakes, big data, ai and company are generating

t0mas88 6y ago
Because most data isn't "big" data. If it fits in Excel on a laptop, why bother to roll out a distributed data lake system like Spark with all the associated ops work.
- noxvilleza 6y ago
  ... and if it doesn't fit on Excel on a laptop, the natural path is just some database running on a desktop, then a server, then a beefier server. It's actually amazing how far just something like postgres will scale for your data needs.
phoe-krk 6y ago
> Why is adoption of this new tech so poor? Because it's unnecessary to those 99% of companies. If company data fits in an Excel spreadsheet, Access database, or a single MySQL/Postgres instance, then introducing all this new tech with all of the associated costs and little return gain is a net loss.
- baskire 6y ago
  Might even see negative gain. Often time the query perf per machine goes down as you move from a single node, indexed, postgresql To a multi node unstructured data. If you don’t have so much data or unstructured data. Why use the more complex and generally slower per FLOP solution.
- kfk 6y ago
  Some of this new tech is simpler to run than MySQL/Postgres like S3/Spectrum. Excel is ok for last mile type of analytics but it doesn't work if you are looking for serious insights. Excel also generates a significant amount of unnecessary manual work and data quality issues. It's not so black and white, what I am saying is that the analytics tech industry direction doesn't seem to address the real issues.

rymurr 6y ago

Check out https://searchdatamanagement.techtarget.com/news/252495619/Apache-Iceberg-rising-for-new-cloud-data-lake-platforms https://searchdatamanagement.techtarget.com/news/252495619/A... the convergence between data lake and data warehouse idea is starting to spread rapidly.

jpau 6y ago

I'm deeply disappointed in Databricks as an RDBMS. As a DS/DE, there's a lot to love (not all, but a lot). The easy provision of Spark clusters. The jobs API. DeltaLake (mostly). Easy notebooks (please don't create a prod system from these..). And Spark itself continues to improve, albeit in an increasingly crowded field. But I've worked closely with BigCo SQL analysts on Azure Databricks, and their experience was terrible. For example: - You cannot browse the data structure without an active cluster - Starting a cluster can take ~5 minutes and, since you missed that moment, you may not submit your first query until 10-15 minutes. - The SQL error messages are often (perhaps usually?) nonsense, so you have to operate without them. - An unfortunate amount of downtime, followed by bizarre excuses. - It's so darn slow, relative to equivalent queries on BigQuery or Snowflake. - Even submitting a query can take a weird amount of time. If Databricks-as-an-RDBMS were competing against Teradata, sure, let's have a chat. But we're in 2021, and there's just no comparing the experience of the SQL analyst on Databricks-as-an-RDBMS vs. Snowflake/BigQuery. I'm excited for the potential of Snowflake's SnowPark (though know little about it). Calling UDFs from SQL means you can create great features for SQL analysts, provided that they can build the momentum to need it.

nattaylor 6y ago
I got excellent performance in Databricks with well partitioned Parquet and Spark 2.4. What is making the queries slow? Data scanning?
- jpau 6y ago
  They use DeltaLake + Spark 3.0, and are mostly careful to partition well. Their datasets are small. Most tables are ~50GB, the odd table up to ~2TB. The clusters typically are nothing shabby for this size, defaults to ~[4-12]x32GB. The queries that I have seen are typically not written well. Think view-on-view-on-view (there's a BigCo policy against them materialising data..), and where the filter is applied in the last step. The stuff of horrors, but something I've seen in more-than-one-BigCo. But we have compared some of those same queries on BigQuery vs. Databricks, and, I don't know if BigQuery's execution optimiser is better? Or if the BigQuery storage is better organising the data? Or if BigQuery is simply throwing more resource their way?

drej 6y ago

I have seen several deployments of Databricks (including first hand experience) and... most use cases could be better served by Postgres (or Redshift, Athena, Snowflake for larger scale). It has honestly been such an overkill for so many workloads, it was quite astonishing. I've seen people move from Excel to Spark... to handle the same volume of data. That's obviously not Databricks' fault, but their PR is pretty much "please do all your data work in our product, it's well suited for it". Yes, it's very good if you don't like setting up clusters (few do/can) and the UI is rather useful for getting up and running (not so much for writing code though). But you need to really understand the platform before adopting it. Please, please don't just adopt it because it's popular.

wokwokwok 6y ago

This 100 word article says everything about databricks and the whole ecosystem of content free data engineering blog posts. It’s all buzz words, no content and please signup now. How so this link so we’ll upvoted? Did people just see “databricks” and think ooo... I have an opinion on that? Read the article before you upvote folks; this one doesn’t deserve your time.

bertomart 6y ago

Databricks is not an RDBMS...it's a Unified Analytics Platform (their words i know), but that really describes it well. It's been really liberating to having easy access to versioned data at your fingertips when iterating on models...and also being able to track models and share work. It does have some warts but it's much better than any other single platform I've used for data science/ML

LukeEF 6y ago

Were there some serious mental gymnastics coming from Fivetran yesterday, or am I missing something? The CEO had a data lake hit piece in Forbes ('The Business Case For Ditching Your Data Lake'[1]) and almost simultaneously authored this 'Databricks' Delta Lake is not a data lake' blog. Felt like the CEO wanted to make a strong point, but has a relationship with Databricks that he didn't want to undermine, so did some reclassification gymnastics to make it all fit. But I am probably too cynical. Finally, I beg for mercy from the 'lakehouse'. It is a bridge too far ;-) [1] https://www.forbes.com/sites/forbestechcouncil/2021/02/01/the-business-case-for-ditching-your-data-lake/ https://www.forbes.com/sites/forbestechcouncil/2021/02/01/th...