4 ms·

Great post but it seems like you still rely on Fabric to run Spark NEE. If you're on AWS or GCP, you should probably not ditch Spark but combine both. DuckDB's

Great post but it seems like you still rely on Fabric to run Spark NEE. If you're on AWS or GCP, you should probably not ditch Spark but combine both. DuckDB's gotcha is that it can't scale horizontally (multi-node), unlike Databricks. A single node can get you as far as you or can rent 2TB memory + 20TB NVME in AWS, and if you use PySpark, you can run DuckDB until it doesn't scale with its Spark integration (https://duckdb.org/docs/api/python/spark_api.html https://duckdb.org/docs/api/python/spark_api.html) and switch to Databricks if you need to scale out. That way, you get the best of the two worlds. DuckDB on AWS EC2's price performance rate is 10x that of Databricks and Snowflake with its native file format, so it's a better deal if you're not processing petabyte-level data. That's unsurprising, given that DuckDB operates in a single node (no need for distributed shuffles) and works primarily with NVME (no use of object stores such as S3 for intermediate data). Thus, it can optimize the workloads much better than the other data warehouses. If you use SQL, another gotcha is that DuckDB doesn't have advanced catalog features in cloud data warehouses. Still, it's possible to combine DuckDB compute and Snowflake Horizon / Databricks Unity Catalog thanks to Apache Iceberg, which enables multi-engine support in the same catalog. I'm experimenting this multi-stack idea with DuckDB <> Snowflake, and it works well so far: https://github.com/buremba/universql https://github.com/buremba/universql

hipadev23 2y ago
Or just use Clickhouse.
- buremba 2y ago
  Clickhouse has excellent performance and it scales out indeed. Unfortunately, it's hard to deploy and maintain a Clickhouse cluster and its catalog features are pretty basic compared to Databricks and Snowflake.
  - hipadev23 2y ago
    I’m sorry what? Clickhouse is crazy easy to deploy. Single node is just a binary. “Cluster” is single binaries on individual nodes plus clickhouse-keeper (the one extra layer of complexity). Altinity and others offer k8s operators or managed solutions to do all this on-prem to make it even easier. Your data catalog point remains, it doesn’t offer anything other basic SQL describe functionality.
    - buremba 2y ago
      The comparison is with Snowflake and Databricks, which takes a button or a query (https://docs.snowflake.com/en/sql-reference/sql/create-warehouse https://docs.snowflake.com/en/sql-reference/sql/create-wareh...) to deploy new clusters. No disk, network, or permission setup is needed for each cluster. I have rarely seen data people familiar with K8s as they mostly use managed services, but feel free to prove me wrong!
      - hipadev23 2y ago
        I fully disagree with suggesting ridiculously overpriced for-profit SaaS solutions in the context of choosing among open-source tech.
        buremba 2y ago
        I don't recommend overpriced SaaS solutions; I highlight that there is a lesson to be learned on why they're so popular, and open-source tech could be as simple as them. That's why I started Universql in the first place; use their interface/protocol as an open-source tech and reduce the compute cost (often ~90% of the total cost of DWHs) by using DuckDB. You still get all the catalog features without paying the premium price.
        datadeft 2y ago
        They are so popular because the decisions are made by managament in many companies and they have a good sales team. Using popularity as justification of anything is the definition of slippery slope.
        buremba 2y ago
        I agree but I also don't think you can have $3B ARR with only good sales.
      - hogu 2y ago
        Clickhouse also has managed service (https://clickhouse.com/ https://clickhouse.com/)
- brunoqc 2y ago
  Clickhouse is open core. Which is a dealbreaker for me.
- riku_iki 2y ago
  in my testing, clickhouse dies on various queries with OOM on large datasets, because algos are subefficient.
hamandcheese 2y ago
> A single node can get you as far as you or can rent 2TB memory + 20TB NVME in AWS What I'm a little curious about with these "single node" solutions - is redundancy not a concern with setups like this? Is it assumed that you can just rebuild your data warehouse from some form of "cold" storage if you lose your nvme data?
- buremba 2y ago
  Exactly. Let's say you have a data pipeline in dbt or your favourite transformation tool. All the source data will be in your cold storage, AWS S3, all the intermediate data (TEMP tables) is written to NVME, and then the final tables are created in S3 & registered in your catalog. AWS recently released S3 Tables, which aims to make the maintenance easier with Apache Iceberg as well: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tab... You can use spot instances if you can afford more latency or might prefer on-demand instances and keep them warm if you need low latency. Databricks (compute) and Snowflake (warehouse) do that automatically for you for the premium price.
- mrbungie 2y ago
  In these setups compute is ephemeral and decoupled from storage (you would normally use an object storage offering that is actually redundant out-of-the-box), so the 2TB is for working memory + OS + whatever and the 20TB NVMe is purely for maybe local spilling and a local cache so you can save on storage reads. If a node fails when running a process (e.g. for an external reason not related to your own code or data: like your spot EC2 instance terminating due to high demand), you just run it again. When you're done running your processes, normally the processing node is completely terminated. tl;dr: you treat them like cattle with a very short lifecycle around data processes. The specifics of resource/process scheduling being dependent on your data needs.
  - hamandcheese 2y ago
    But duckdb has its own on-disk format, and it generally is the best performing thing to scan (as opposed to always just querying, say, parquet files in S3). So if you want to use the duckdb native format, and you have a lot of data... what do you do? How do you keep your duckdb file up to date with incoming data to your data lake? Maybe you just... don't? Have a daily rebuild of your ephemeral-ish duckdb node? It seems like it would be kind of a pain to manage once you get to moderate scale (say, TB+ datasets).

hipadev23 2y ago

Or just use Clickhouse.

buremba 2y ago
Clickhouse has excellent performance and it scales out indeed. Unfortunately, it's hard to deploy and maintain a Clickhouse cluster and its catalog features are pretty basic compared to Databricks and Snowflake.
- hipadev23 2y ago
  I’m sorry what? Clickhouse is crazy easy to deploy. Single node is just a binary. “Cluster” is single binaries on individual nodes plus clickhouse-keeper (the one extra layer of complexity). Altinity and others offer k8s operators or managed solutions to do all this on-prem to make it even easier. Your data catalog point remains, it doesn’t offer anything other basic SQL describe functionality.
  - buremba 2y ago
    The comparison is with Snowflake and Databricks, which takes a button or a query (https://docs.snowflake.com/en/sql-reference/sql/create-warehouse https://docs.snowflake.com/en/sql-reference/sql/create-wareh...) to deploy new clusters. No disk, network, or permission setup is needed for each cluster. I have rarely seen data people familiar with K8s as they mostly use managed services, but feel free to prove me wrong!
    - hipadev23 2y ago
      I fully disagree with suggesting ridiculously overpriced for-profit SaaS solutions in the context of choosing among open-source tech.
      - buremba 2y ago
        I don't recommend overpriced SaaS solutions; I highlight that there is a lesson to be learned on why they're so popular, and open-source tech could be as simple as them. That's why I started Universql in the first place; use their interface/protocol as an open-source tech and reduce the compute cost (often ~90% of the total cost of DWHs) by using DuckDB. You still get all the catalog features without paying the premium price.
        datadeft 2y ago
        They are so popular because the decisions are made by managament in many companies and they have a good sales team. Using popularity as justification of anything is the definition of slippery slope.
        buremba 2y ago
        I agree but I also don't think you can have $3B ARR with only good sales.
    - hogu 2y ago
      Clickhouse also has managed service (https://clickhouse.com/ https://clickhouse.com/)
brunoqc 2y ago
Clickhouse is open core. Which is a dealbreaker for me.
riku_iki 2y ago
in my testing, clickhouse dies on various queries with OOM on large datasets, because algos are subefficient.

hamandcheese 2y ago

> A single node can get you as far as you or can rent 2TB memory + 20TB NVME in AWS What I'm a little curious about with these "single node" solutions - is redundancy not a concern with setups like this? Is it assumed that you can just rebuild your data warehouse from some form of "cold" storage if you lose your nvme data?

buremba 2y ago
Exactly. Let's say you have a data pipeline in dbt or your favourite transformation tool. All the source data will be in your cold storage, AWS S3, all the intermediate data (TEMP tables) is written to NVME, and then the final tables are created in S3 & registered in your catalog. AWS recently released S3 Tables, which aims to make the maintenance easier with Apache Iceberg as well: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tab... You can use spot instances if you can afford more latency or might prefer on-demand instances and keep them warm if you need low latency. Databricks (compute) and Snowflake (warehouse) do that automatically for you for the premium price.
mrbungie 2y ago
In these setups compute is ephemeral and decoupled from storage (you would normally use an object storage offering that is actually redundant out-of-the-box), so the 2TB is for working memory + OS + whatever and the 20TB NVMe is purely for maybe local spilling and a local cache so you can save on storage reads. If a node fails when running a process (e.g. for an external reason not related to your own code or data: like your spot EC2 instance terminating due to high demand), you just run it again. When you're done running your processes, normally the processing node is completely terminated. tl;dr: you treat them like cattle with a very short lifecycle around data processes. The specifics of resource/process scheduling being dependent on your data needs.
- hamandcheese 2y ago
  But duckdb has its own on-disk format, and it generally is the best performing thing to scan (as opposed to always just querying, say, parquet files in S3). So if you want to use the duckdb native format, and you have a lot of data... what do you do? How do you keep your duckdb file up to date with incoming data to your data lake? Maybe you just... don't? Have a daily rebuild of your ephemeral-ish duckdb node? It seems like it would be kind of a pain to manage once you get to moderate scale (say, TB+ datasets).