5 ms·

As someone new to the data ops platform world.. what does Databricks even do? They get so much hype as a great employer, but why are they attracting people if t

As someone new to the data ops platform world.. what does Databricks even do? They get so much hype as a great employer, but why are they attracting people if their secret sauce is irrelevant?

moneywoes 3y ago
Further, what is it different from snowflake that they do?
- pram 3y ago
  The biggest difference to consider is storage, I guess. Databricks clusters can store in S3 or wherever you want really. Snowflake has their own storage layer and it always lives in their cloud. This precludes any on-prem deployments.
  - ramraj07 3y ago
    You can’t be more wrong? Snowflake also stores in s3 or elsewhere, in fact it’s like the only common feature. Snowflake uses proprietary formats mostly though and only gives a sql interface. Think of it as Apple vs android.
    - cfeduke 3y ago
      > You can’t be more wrong? Snowflake also stores in s3 or elsewhere Snowflake stores data in S3 in Snowflake's various AWS accounts. Egress fees are necessary when performing anything outside of Snowflake. Databricks operates on data stored in your S3 on your AWS account(s). Databricks also runs on your compute contained within your AWS account(s). Both approaches have valid use cases.
      - chrisjc 3y ago
        > Snowflake stores data in S3 in Snowflake's various AWS accounts. Egress fees are necessary when performing anything outside of Snowflake. Egress fees are only if you are unloading into a different region/cloud. If your data is in SF AWS us-east-1, and you unload to your S3 in us-east-1, there is no egress fee. > Snowflake charges a per-byte fee for data egress when users transfer data from a Snowflake account into a different region on the same cloud platform or into a completely different cloud platform. Data transfers within the same region are free. https://docs.snowflake.com/en/user-guide/cost-understanding-data-transfer https://docs.snowflake.com/en/user-guide/cost-understanding-...
    - pram 3y ago
      Not in your account.
      - jrott 3y ago
        The compute itself can be in your account. The control plane is usually managed by databricks
  - jamesblonde 3y ago
    Databricks are not on-premises either.
    - cfeduke 3y ago
      > Databricks are not on-premises either. Usually, no. But in some cases, yes.
      - jamesblonde 3y ago
        I must have missed the product announcement. Do you have a link, because everywhere I look I can't find it.
        forrest2 3y ago
        You can get most products on prem at a certain price & size. A lot of companies will apply resistance though unless the contract size is right b/c on-prem contracts tend to be less unit-profitable, risky, + unique annoying terms or constraints. If you * need * it, you find a human to talk to (sales or connect from your network).
- kentm 3y ago
  Snowflake only really offers SQL (with UDFs) to do data transforms. Spark offers SQL, but also code-based solutions (Python, Scala, Java) that can be used to scale things like ML pipelines and non-SQL transforms.
bryan0 3y ago
Databricks was founded by the creators of Apache Spark, and their platform is a fully hosted spark solution. Spark is a flexible distributed computing framework which came as a successor to Hadoop and map/reduce. Spark processes big data at scale on clusters, this includes ML training and inference pipelines. Now though, their value prop is less clear since you can implement these pipelines (for example using AWS sagemaker) without having to learn spark.
enumjorge 3y ago
As far as being an attractive employer they pay really well. Not sure about other dimensions. This is anecdotal but a couple of people in my network had bad things to say about the work life balance and culture there.
bobbruno 3y ago
I’m a solution architect at Databricks, so feel free to skip if you think I’m too biased - just know that I worked with Databricks before working for them, and I liked the platform then as well. While Databricks definitely started with Spark, and it’s still a significant foundation, there’s done much more on top and around it. For instance: - MLFlow for ML lifecycle management from experiments to real-time ML serving; - Delta Storage format, extending parquet to leverage cloud storage and enable efficient updates and very fast access; - SQL Warehouses, which expose Databricks as a SQL engine for Analytics; - Jobs/Workflows, which is one of the most used orchestration engines in the world; - Unity Catalog, which will replace (at least in Databricks) Hive Metastore for metadata, access control, lineage and data governance tooling in general. And now LLMs, on top of the data and ML capabilities mentioned above, much extended by the Mosaic deal (still to be approved). The interesting thing is that yes, Databricks can do the whole “Data Warehousing” thing, but it can also do very large scale streaming, machine learning, process unstructured data like text, audio or images, support BI applications, etc - all accessing the same data with compatible tooling. So, it’s a full blown multi-workload data platform for any kind of use case and company size. One can argue that most components are open-source and can be deployed independently - and Databricks has open-sourced Spark, MLFlow and Delta. It’s just that most companies simply don’t have enough (if any) staff with skills to deploy and operate all these things, let alone as one integrated platform. With Databricks, I’m used to deliver a demo where it takes me about 20 minutes from having a new cloud account to be running data workloads against a cluster or SQL, with all the functionality above.

As someone new to the data ops platform world.. what does Databricks even do? They get so much hype as a great employer, but why are they attracting people if their secret sauce is irrelevant?

moneywoes 3y ago

Further, what is it different from snowflake that they do?

pram 3y ago
The biggest difference to consider is storage, I guess. Databricks clusters can store in S3 or wherever you want really. Snowflake has their own storage layer and it always lives in their cloud. This precludes any on-prem deployments.
- ramraj07 3y ago
  You can’t be more wrong? Snowflake also stores in s3 or elsewhere, in fact it’s like the only common feature. Snowflake uses proprietary formats mostly though and only gives a sql interface. Think of it as Apple vs android.
  - cfeduke 3y ago
    > You can’t be more wrong? Snowflake also stores in s3 or elsewhere Snowflake stores data in S3 in Snowflake's various AWS accounts. Egress fees are necessary when performing anything outside of Snowflake. Databricks operates on data stored in your S3 on your AWS account(s). Databricks also runs on your compute contained within your AWS account(s). Both approaches have valid use cases.
    - chrisjc 3y ago
      > Snowflake stores data in S3 in Snowflake's various AWS accounts. Egress fees are necessary when performing anything outside of Snowflake. Egress fees are only if you are unloading into a different region/cloud. If your data is in SF AWS us-east-1, and you unload to your S3 in us-east-1, there is no egress fee. > Snowflake charges a per-byte fee for data egress when users transfer data from a Snowflake account into a different region on the same cloud platform or into a completely different cloud platform. Data transfers within the same region are free. https://docs.snowflake.com/en/user-guide/cost-understanding-data-transfer https://docs.snowflake.com/en/user-guide/cost-understanding-...
  - pram 3y ago
    Not in your account.
    - jrott 3y ago
      The compute itself can be in your account. The control plane is usually managed by databricks
- jamesblonde 3y ago
  Databricks are not on-premises either.
  - cfeduke 3y ago
    > Databricks are not on-premises either. Usually, no. But in some cases, yes.
    - jamesblonde 3y ago
      I must have missed the product announcement. Do you have a link, because everywhere I look I can't find it.
      - forrest2 3y ago
        You can get most products on prem at a certain price & size. A lot of companies will apply resistance though unless the contract size is right b/c on-prem contracts tend to be less unit-profitable, risky, + unique annoying terms or constraints. If you * need * it, you find a human to talk to (sales or connect from your network).
kentm 3y ago
Snowflake only really offers SQL (with UDFs) to do data transforms. Spark offers SQL, but also code-based solutions (Python, Scala, Java) that can be used to scale things like ML pipelines and non-SQL transforms.

bryan0 3y ago

Databricks was founded by the creators of Apache Spark, and their platform is a fully hosted spark solution. Spark is a flexible distributed computing framework which came as a successor to Hadoop and map/reduce. Spark processes big data at scale on clusters, this includes ML training and inference pipelines. Now though, their value prop is less clear since you can implement these pipelines (for example using AWS sagemaker) without having to learn spark.

enumjorge 3y ago

As far as being an attractive employer they pay really well. Not sure about other dimensions. This is anecdotal but a couple of people in my network had bad things to say about the work life balance and culture there.

bobbruno 3y ago

I’m a solution architect at Databricks, so feel free to skip if you think I’m too biased - just know that I worked with Databricks before working for them, and I liked the platform then as well. While Databricks definitely started with Spark, and it’s still a significant foundation, there’s done much more on top and around it. For instance: - MLFlow for ML lifecycle management from experiments to real-time ML serving; - Delta Storage format, extending parquet to leverage cloud storage and enable efficient updates and very fast access; - SQL Warehouses, which expose Databricks as a SQL engine for Analytics; - Jobs/Workflows, which is one of the most used orchestration engines in the world; - Unity Catalog, which will replace (at least in Databricks) Hive Metastore for metadata, access control, lineage and data governance tooling in general. And now LLMs, on top of the data and ML capabilities mentioned above, much extended by the Mosaic deal (still to be approved). The interesting thing is that yes, Databricks can do the whole “Data Warehousing” thing, but it can also do very large scale streaming, machine learning, process unstructured data like text, audio or images, support BI applications, etc - all accessing the same data with compatible tooling. So, it’s a full blown multi-workload data platform for any kind of use case and company size. One can argue that most components are open-source and can be deployed independently - and Databricks has open-sourced Spark, MLFlow and Delta. It’s just that most companies simply don’t have enough (if any) staff with skills to deploy and operate all these things, let alone as one integrated platform. With Databricks, I’m used to deliver a demo where it takes me about 20 minutes from having a new cloud account to be running data workloads against a cluster or SQL, with all the functionality above.