5 ms·

Hmm, I agree there's a market here, but I don't know why I wouldn't just use Snowflake or Bigquery if what I really wanted was a big-data RDBMS. Everyone I kno

Hmm, I agree there's a market here, but I don't know why I wouldn't just use Snowflake or Bigquery if what I really wanted was a big-data RDBMS. Everyone I know who uses Databricks (and they all like it) use it as hosted Spark with S3 integrations, or write... directly to Snowflake. I'm a little skeptical they're going to get traction as a true data lake model

MikeDelta 6y ago
Their Delta also has SSD caching, which turns out to be logic that stores a local copy of the file you queried for faster re-query. Going to call my lru cache function like that as well... My company loves them, I think they only do a few things good of which marketing the best, and are not worth the money for data science teams with devops skills. Happy to hear from others if I am wrong.
- pram 6y ago
  Depends. If you only have a couple workspaces, then no. It's worth the money for a company with lots of data science teams, and one team who is janitoring all the workspace infra. Our company has 20 workspaces for 10 teams already and it would be a nightmare if we expected everyone to fix and manage their own AWS stuff.
georgewfraser 6y ago
It's a lot simpler to use a single system as both your data lake, and your data warehouse. As Databricks gets better and better at the core data warehouse features, it becomes feasible to use it for both. Meanwhile, Snowflake and BQ are coming from the other direction, implementing data lake features. AWS strategy seems to be, just make it easier to have 2 systems and move data back and forth.
willvarfar 6y ago
The reason snowflake has such a high market cap is because it’s customers aren’t paying much now, but they’ll be paying and paying monthly forever. It’s lock-in on a massive scale. Delta lake is something you can run on data you feel you still have some semblance of control over.
- dominotw 6y ago
  > but they’ll be paying and paying monthly forever. It’s lock-in on a massive scale. Is making money by tricking customers into product lock in really a good business strategy. Wouldn't the word get out sooner or later? dont you have to pay aws for s3,emr /data engineers monthly and forever too.
  - fphhotchips 6y ago
    FWIW the only organisation I know that's ever truly locked in data was SAP, through their byzantine licensing and horrific API. Snowflake is no more locked in than anything else with a standard SQL interface.
- fphhotchips 6y ago
  I work for Snowflake and I'm always curious about this way of thinking. How is putting the data in Snowflake any different to any other RDBMS, or even S3, in this regard? You don't own the underlying storage, true, but there's a defined method of getting data in and a defined method of getting that data out. The API is different to S3, sure, but it's an API all the same.
  - snidane 6y ago
    When I want to process something in Snowflake I would have to copy the data 1. from snowflake storage to s3 2. from s3 to my compute node 3. do the processing 4. store result in s3 5. copy result from s3 to snowflake storage If I only use s3, my data gies like this 1. s3 to compute node 2. run my script 3. result to s3 Perhaps I could use bunch if insert statement and load data into Snowflake through sql straight feom compute node, but then at a great cost. I don't really see much benefit of using Snowflake for not-yet structured data. For structured data it works wonders. Only downsides there: 1. Expensive as f when you scale up 2. can't control cost in pay-per-query model.
snidane 6y ago
Snowflake and Bigquery will bite you in the ass later on. You can do 80% of the things you will need - which is great for some newbie stuff or for sales presentations. Once you need something complicated, you're on your own, while being stuck in a proprietary environment that you cannot extend. You will have to develop some kind of data lake to store unstructured data anyway. You will end up with a Snowflake data warehouse and a data lake. Why not just go with data lake first then. Databricks/Spark are just good platforms to help you do something with structured data in your lake. With the recent additions to its execution engine and Delta (strange naming tbh) it will be pretty much the same as Snowflake for you.
- mrbungie 6y ago
  BigQuery/Snowflake can process Parquet and multiple other formats in Object Storage. You can use them more "freely" if you keep your raw data in open formats. You need something more complicated than what can be done using BigQuery/Snowflake (that remaining 20%, though I would say 10%)? Export the dataset to CSV/Parquet/Avro/ORC/whatever and process it with anything, including Dataproc/HDInsight/EMR or even Databricks. That's actually a common pattern.
  - snidane 6y ago
    While both BQ and Snowflake are adopting lake features, they still only support parquet and other file formats for loading, not for querying. Can't do a simple select * from s3://file.parquet which you can do in Spark. Having to load it into the data warehouse means that you duplicate your data two times and it is stupidly annoying. Many times the data doesn't even resemble anything tabular before I structure it in python scripts. Why would I load it inside a warehouse only to then pull it down to do some python processing and loading it back. Which makes the data travel from DWH storage to a data lake and then to my compute cluster and then the same cumbersome roadtrip back. Pretty wasteful. Spark at least allows me to schedule a python function across a cluster while copying only from lake to my compute node and back. Data warehouses like BQ and Snowflake are great for data scientists after a bunch of engineers slice and dice raw data into clean tables. For anyone working with not yet structured data, data lake wins hands down.
opportune 6y ago
I’m one of the apparently odd few who doesn’t like it. It’s slow, overkill for many tasks, and a Swiss Army knife that’s ok at everything but not very good at anything. Debugging it can be a very frustrating experience. They’re trying to turn it into an everything platform so it checks every box in the procurement process and don’t care about actually making it good. I have a feeling legacy databricks installations will be much derided in 5-10 years time

MikeDelta 6y ago

Their Delta also has SSD caching, which turns out to be logic that stores a local copy of the file you queried for faster re-query. Going to call my lru cache function like that as well... My company loves them, I think they only do a few things good of which marketing the best, and are not worth the money for data science teams with devops skills. Happy to hear from others if I am wrong.

pram 6y ago
Depends. If you only have a couple workspaces, then no. It's worth the money for a company with lots of data science teams, and one team who is janitoring all the workspace infra. Our company has 20 workspaces for 10 teams already and it would be a nightmare if we expected everyone to fix and manage their own AWS stuff.

georgewfraser 6y ago

It's a lot simpler to use a single system as both your data lake, and your data warehouse. As Databricks gets better and better at the core data warehouse features, it becomes feasible to use it for both. Meanwhile, Snowflake and BQ are coming from the other direction, implementing data lake features. AWS strategy seems to be, just make it easier to have 2 systems and move data back and forth.

willvarfar 6y ago

The reason snowflake has such a high market cap is because it’s customers aren’t paying much now, but they’ll be paying and paying monthly forever. It’s lock-in on a massive scale. Delta lake is something you can run on data you feel you still have some semblance of control over.

dominotw 6y ago
> but they’ll be paying and paying monthly forever. It’s lock-in on a massive scale. Is making money by tricking customers into product lock in really a good business strategy. Wouldn't the word get out sooner or later? dont you have to pay aws for s3,emr /data engineers monthly and forever too.
- fphhotchips 6y ago
  FWIW the only organisation I know that's ever truly locked in data was SAP, through their byzantine licensing and horrific API. Snowflake is no more locked in than anything else with a standard SQL interface.
fphhotchips 6y ago
I work for Snowflake and I'm always curious about this way of thinking. How is putting the data in Snowflake any different to any other RDBMS, or even S3, in this regard? You don't own the underlying storage, true, but there's a defined method of getting data in and a defined method of getting that data out. The API is different to S3, sure, but it's an API all the same.
- snidane 6y ago
  When I want to process something in Snowflake I would have to copy the data 1. from snowflake storage to s3 2. from s3 to my compute node 3. do the processing 4. store result in s3 5. copy result from s3 to snowflake storage If I only use s3, my data gies like this 1. s3 to compute node 2. run my script 3. result to s3 Perhaps I could use bunch if insert statement and load data into Snowflake through sql straight feom compute node, but then at a great cost. I don't really see much benefit of using Snowflake for not-yet structured data. For structured data it works wonders. Only downsides there: 1. Expensive as f when you scale up 2. can't control cost in pay-per-query model.

snidane 6y ago

Snowflake and Bigquery will bite you in the ass later on. You can do 80% of the things you will need - which is great for some newbie stuff or for sales presentations. Once you need something complicated, you're on your own, while being stuck in a proprietary environment that you cannot extend. You will have to develop some kind of data lake to store unstructured data anyway. You will end up with a Snowflake data warehouse and a data lake. Why not just go with data lake first then. Databricks/Spark are just good platforms to help you do something with structured data in your lake. With the recent additions to its execution engine and Delta (strange naming tbh) it will be pretty much the same as Snowflake for you.

mrbungie 6y ago
BigQuery/Snowflake can process Parquet and multiple other formats in Object Storage. You can use them more "freely" if you keep your raw data in open formats. You need something more complicated than what can be done using BigQuery/Snowflake (that remaining 20%, though I would say 10%)? Export the dataset to CSV/Parquet/Avro/ORC/whatever and process it with anything, including Dataproc/HDInsight/EMR or even Databricks. That's actually a common pattern.
- snidane 6y ago
  While both BQ and Snowflake are adopting lake features, they still only support parquet and other file formats for loading, not for querying. Can't do a simple select * from s3://file.parquet which you can do in Spark. Having to load it into the data warehouse means that you duplicate your data two times and it is stupidly annoying. Many times the data doesn't even resemble anything tabular before I structure it in python scripts. Why would I load it inside a warehouse only to then pull it down to do some python processing and loading it back. Which makes the data travel from DWH storage to a data lake and then to my compute cluster and then the same cumbersome roadtrip back. Pretty wasteful. Spark at least allows me to schedule a python function across a cluster while copying only from lake to my compute node and back. Data warehouses like BQ and Snowflake are great for data scientists after a bunch of engineers slice and dice raw data into clean tables. For anyone working with not yet structured data, data lake wins hands down.

opportune 6y ago

I’m one of the apparently odd few who doesn’t like it. It’s slow, overkill for many tasks, and a Swiss Army knife that’s ok at everything but not very good at anything. Debugging it can be a very frustrating experience. They’re trying to turn it into an everything platform so it checks every box in the procurement process and don’t care about actually making it good. I have a feeling legacy databricks installations will be much derided in 5-10 years time