4 ms·
we regularly run audits on over 12 years of customer order histories. This requires scanning of about 40TB of data and growing. They used to jump through hoop
by cmollis 4y ago
we regularly run audits on over 12 years of customer order histories. This requires scanning of about 40TB of data and growing. They used to jump through hoops on the Oracle cluster just to get data out for one customer. We pushed all of the order history into s3 parquet using Spark and I can query this in about 20 seconds using Spark or Presto. It's now streamed through kafka and Spark structured streaming so it's up to date in about 3 minutes. The click-bait-y title notwithstanding, I get that not all data is 'big' and duckdb (and datafusion, polars, etc) is probably great for certain use-cases but what I work on every day can't be done on a single machine.
- greggyb 4y agoI mean, you can almost fit all that data on one SSD. Micron's latest are 30TB. Commodity servers are available with 24 NVMe drive bays. At 7 GB/s read, across 24 drives, you can scan 40 TB in 240 seconds. Such a server could readily fit 384 threads with dual EPYC and would be available with enough RAM to keep more than 10% of that data in cache. Your workload absolutely fits the definition of "fits on one machine". I am not suggesting here that you should put it on one machine. You definitely could, though.