3 ms·

I still don’t understand DuckDB use case. To handle huge amount of data, you need a large database that’s connected to the network. DuckDB will eventually hit a

by didip 3y ago

I still don’t understand DuckDB use case. To handle huge amount of data, you need a large database that’s connected to the network. DuckDB will eventually hit a ceiling.

icyfox 3y ago
It fits a pretty nice niche between small data (32GB in-memory, pandas level) and truly huge data where the disk files can't even fit on one machine (2TB+). Most data that people want to process on a daily cadence fit somewhere in between those ranges. As for whether to use a database or not, I think that's a more fundamental question about where your different read and write loads are coming from. Scaling a database to support billions of rows and have high query performance is not trivial; if your use-case fits more in a write-infrequently, read-often then OLAP is a pretty good choice of index structure.
- riku_iki 3y ago
  > It fits a pretty nice niche between small data (32GB in-memory, pandas level) and truly huge data where the disk files can't even fit on one machine (2TB+) you can build/rent server with say 50TB nvme raid and run duckdb on it?..
simonw 3y ago
It's for analytics. It's a tool that can run on my laptop that means I can run analytical queries against potentially TBs of data hosted in an S3 bucket, without needing to spin up an expensive data warehouse.
swasheck 3y ago
similar use case to sqlite as an embedded database system with a columnar/analytics focus.
isoprophlex 3y ago
No no it is insanely useful and I wish I had a job where I could use it more! Noone really has big big data, they only think they do. You can do the analytics that Databricks and Snowflake are trying to sell your boss for $100.000 a month, only then on an 80$ vps and a couple of S3 buckets full of parquet files. It's a fantastic tool to run analytics. Lean, purposeful. I hope they can manage to stay out of the clutches of corporate enshittification for a while longer.
- ramraj07 3y ago
  You seem to be quite sure of how useful duckdb is even though you haven’t found a use case for it. This is actually a common pattern I’ve seen everywhere - the people super excited about duckdb are people who never use it in their job and at best for toy use cases. If your data is truly small enough to run on a vps with duckdb, your monthly snowflake bill will not break a few hundred dollars by any stretch. A terabyte stored on snowflake costs 23 bucks a month and running a query on it depending on complexity will cost no more than a dollar. And you don’t pay any cost other than these two.
  - pradeepchhetri 3y ago
    Wow I didn't know that snowflake charges on query level. Very good business idea to first create dependency and rip them off once their business grows.
    - ramraj07 3y ago
      Not disagreeing, but pick a lane - is your data too small or too big? The discussion here was about data thats allegedly too small to be called "big data"..
    - hodgesrm 3y ago
      > Wow I didn't know that snowflake charges on query level. It doesn't. You get charged "credits" for virtual data warehouse operation plus storage. A credit corresponds (approximately) to running a VM of a certain size for an hour. You can turn VDWs on and off more or less at will. Not taking a position on whether that's good for users or not. Just pointing out how it works.
chrisjc 3y ago
DuckDB use case? I remember being in the same spot. Then I realized "the" use case for DuckDB didn't really exist. There were dozens of use-cases. No wonder I couldn't figure out "the" use case from reading all the articles/blogs/hype people were creating about what they were doing with DuckDB. It seemed like everyone was using it in different ways. It's a data engineering/analytics Swiss army knife. ETL/ELT, ad-hoc/exploratory, munger/wrangler, edge-analytics, and so on... > DuckDB will eventually hit a ceiling. Yeah, probably. Just about every database has hit a ceiling in the past. But then someone comes out with some fantastic new idea to overcome the challenges to some degree. Map-reduce, moving the query to the data instead of the data to the query, serverless, separating storage/compute... However, what if we start thinking along these lines with DuckDB? Reading parquet files addresses separating storage and compute. Parquet also provides columnar/row-grouped data giving us push-down predicates (so kinda moving part of the query closer to the data). We can run a DuckDb instance (EC2/S3) closer to the data so that sorta helps too. What I'm really excited about using DuckDB in a similar way to map-reduce. What if there was a way to take some SQL's logical plan and turn it into a physical plan that uses compute resources from a pool of (serverless) DuckDB instances. Starting at the leaves of the graph (physical plan) pulling/filtering data from the source (parquet files), and returning their completed work up the branches until it is completed and ready to be served up as results. I've seen a few examples of this already, but nothing that I would consider production ready. I have a hunch that someone is going to drop such a project on us shortly, and it's going to change a lot of things we have become use to in the data world. https://github.com/BauplanLabs/quack-reduce https://github.com/BauplanLabs/quack-reduce https://www.boilingdata.com/ https://www.boilingdata.com/
- zX41ZdbW 3y ago
  > What if there was a way to take some SQL's logical plan and turn it into a physical plan that uses compute resources from a pool This is how ClickHouse works on a cluster. For example, writing the same query as SELECT sum(size) FROM urlCluster('default', 'https://huggingface.co/datasets/vivym/midjourney-messages/resolve/main/data/0000{01..55}.parquet') (note the urlCluster usage) will give the result in 0.3 seconds on a cluster: https://pastila.nl/?00ef0aac/a54918ef6d3536fad34a5eca0e1157f3#4/SenMvm+mngkbiCGnQIXg== https://pastila.nl/?00ef0aac/a54918ef6d3536fad34a5eca0e1157f...

by didip 3y ago

I still don’t understand DuckDB use case. To handle huge amount of data, you need a large database that’s connected to the network. DuckDB will eventually hit a ceiling.

icyfox 3y ago

It fits a pretty nice niche between small data (32GB in-memory, pandas level) and truly huge data where the disk files can't even fit on one machine (2TB+). Most data that people want to process on a daily cadence fit somewhere in between those ranges. As for whether to use a database or not, I think that's a more fundamental question about where your different read and write loads are coming from. Scaling a database to support billions of rows and have high query performance is not trivial; if your use-case fits more in a write-infrequently, read-often then OLAP is a pretty good choice of index structure.

riku_iki 3y ago
> It fits a pretty nice niche between small data (32GB in-memory, pandas level) and truly huge data where the disk files can't even fit on one machine (2TB+) you can build/rent server with say 50TB nvme raid and run duckdb on it?..

simonw 3y ago

It's for analytics. It's a tool that can run on my laptop that means I can run analytical queries against potentially TBs of data hosted in an S3 bucket, without needing to spin up an expensive data warehouse.

swasheck 3y ago

similar use case to sqlite as an embedded database system with a columnar/analytics focus.

isoprophlex 3y ago

No no it is insanely useful and I wish I had a job where I could use it more! Noone really has big big data, they only think they do. You can do the analytics that Databricks and Snowflake are trying to sell your boss for $100.000 a month, only then on an 80$ vps and a couple of S3 buckets full of parquet files. It's a fantastic tool to run analytics. Lean, purposeful. I hope they can manage to stay out of the clutches of corporate enshittification for a while longer.

ramraj07 3y ago
You seem to be quite sure of how useful duckdb is even though you haven’t found a use case for it. This is actually a common pattern I’ve seen everywhere - the people super excited about duckdb are people who never use it in their job and at best for toy use cases. If your data is truly small enough to run on a vps with duckdb, your monthly snowflake bill will not break a few hundred dollars by any stretch. A terabyte stored on snowflake costs 23 bucks a month and running a query on it depending on complexity will cost no more than a dollar. And you don’t pay any cost other than these two.
- pradeepchhetri 3y ago
  Wow I didn't know that snowflake charges on query level. Very good business idea to first create dependency and rip them off once their business grows.
  - ramraj07 3y ago
    Not disagreeing, but pick a lane - is your data too small or too big? The discussion here was about data thats allegedly too small to be called "big data"..
  - hodgesrm 3y ago
    > Wow I didn't know that snowflake charges on query level. It doesn't. You get charged "credits" for virtual data warehouse operation plus storage. A credit corresponds (approximately) to running a VM of a certain size for an hour. You can turn VDWs on and off more or less at will. Not taking a position on whether that's good for users or not. Just pointing out how it works.

chrisjc 3y ago

DuckDB use case? I remember being in the same spot. Then I realized "the" use case for DuckDB didn't really exist. There were dozens of use-cases. No wonder I couldn't figure out "the" use case from reading all the articles/blogs/hype people were creating about what they were doing with DuckDB. It seemed like everyone was using it in different ways. It's a data engineering/analytics Swiss army knife. ETL/ELT, ad-hoc/exploratory, munger/wrangler, edge-analytics, and so on... > DuckDB will eventually hit a ceiling. Yeah, probably. Just about every database has hit a ceiling in the past. But then someone comes out with some fantastic new idea to overcome the challenges to some degree. Map-reduce, moving the query to the data instead of the data to the query, serverless, separating storage/compute... However, what if we start thinking along these lines with DuckDB? Reading parquet files addresses separating storage and compute. Parquet also provides columnar/row-grouped data giving us push-down predicates (so kinda moving part of the query closer to the data). We can run a DuckDb instance (EC2/S3) closer to the data so that sorta helps too. What I'm really excited about using DuckDB in a similar way to map-reduce. What if there was a way to take some SQL's logical plan and turn it into a physical plan that uses compute resources from a pool of (serverless) DuckDB instances. Starting at the leaves of the graph (physical plan) pulling/filtering data from the source (parquet files), and returning their completed work up the branches until it is completed and ready to be served up as results. I've seen a few examples of this already, but nothing that I would consider production ready. I have a hunch that someone is going to drop such a project on us shortly, and it's going to change a lot of things we have become use to in the data world. https://github.com/BauplanLabs/quack-reduce https://github.com/BauplanLabs/quack-reduce https://www.boilingdata.com/ https://www.boilingdata.com/

zX41ZdbW 3y ago
> What if there was a way to take some SQL's logical plan and turn it into a physical plan that uses compute resources from a pool This is how ClickHouse works on a cluster. For example, writing the same query as SELECT sum(size) FROM urlCluster('default', 'https://huggingface.co/datasets/vivym/midjourney-messages/resolve/main/data/0000{01..55}.parquet') (note the urlCluster usage) will give the result in 0.3 seconds on a cluster: https://pastila.nl/?00ef0aac/a54918ef6d3536fad34a5eca0e1157f3#4/SenMvm+mngkbiCGnQIXg== https://pastila.nl/?00ef0aac/a54918ef6d3536fad34a5eca0e1157f...