6 ms·

I don’t totally understand the fascination with storing analytical data on S3. It’s not fast, and if you’re in a write heavy environment it’s definitely not che

I don’t totally understand the fascination with storing analytical data on S3. It’s not fast, and if you’re in a write heavy environment it’s definitely not cheap either. What’s with the avoidance of clickhouse or duckdb paired with insanely fast EBS or even physically attached storage? You can still backup to s3, but using s3 for live analytics queries is missing out on so much of the speed.

nine_k 2y ago
S3 is a protocol understood by "everyone". If you're on AWS, as many are, it's basically the only natural choice. But a number of cloud providers, and a bunch of self-hostable software offer an S3 interface. Clickhouse on local NVMe is one possible solution, but then you are married to that solution. An S3 interface is more universal and allows you to mix and match your tools, even though this comes at some expense.
exAspArk 2y ago
My few cents: - Compute and storage separation simplifies managing a system making compute "ephemeral" - Compute resources can be scaled separately without worrying about scaling storage - Object storage provides much higher durability (99.999999999% on S3) compared to disks - Open table formats on S3 become a universal interface in the data space allowing to bring many other data tools if necessary - Costs at scale can actually be lower since there is no data transfer cost within the same region. For example, you can check out WarpStream (Kafka on object storage) case studies that claim saving 5-10x
potamic 2y ago
Would love the authors to pitch in with their use cases, but I think most people simply do not need sub millisecond analytics. This is mostly replacing typical spark pipelines where you're okay with sub second latencies. S3 is the cheapest, fully managed storage you can get that can scale infinitely. When you're already archiving to S3, doubling it for analytics saves cost and simplifies data management.
- hipadev23 2y ago
  Sub-millisecond? Literally nobody in large data analysis is doing that. Can you cite some sources or show me what sort of setups (with any DB tech) that you're able to run meaningful queries on S3 sources with even sub-second latency?
  - broner 2y ago
    I'm doing this with clickhouse querying parquet files on S3 from an EC2 instance in the same region as the S3 bucket (yes DuckDB pretty similar). S3 time to first byte within AWS is 50ms and I get close to saturating an big EC2 instance's 100Gb link doing reads. For OLTP type queries fetching under 1 MB you'll see ~4 round trips + transfer time of compressed data so 150-200 ms latency.
    - hipadev23 2y ago
      Are you using s3 local cache? Do you have heavy writes? What type of s3 disk type, if any, are you using? (s3, s3_plain, s3_plain_rewritable)? Or are you just using the s3 functions. Clickhouse is amazing but I still struggle getting it working efficiently on s3, especially writes.
      - broner 2y ago
        My workload is 100% read. Querying zstd parquet on s3 standard. Neither clickhouse nor duckdb has a great s3 driver, which is why smart people like https://www.boilingdata.com/ https://www.boilingdata.com/ wrote their own. I compared a handful of queries and found DuckDB makes a lot of round trips and Clickhouse takes the opposite approach and just reads the entire parquet file.
ozgrakkurt 2y ago
It is immense amounts of marketing and being lazy to implement actual local storage with replication etc. It just makes everything easier and is marketed a lot.
deleted 2y ago
[deleted]

nine_k 2y ago

S3 is a protocol understood by "everyone". If you're on AWS, as many are, it's basically the only natural choice. But a number of cloud providers, and a bunch of self-hostable software offer an S3 interface. Clickhouse on local NVMe is one possible solution, but then you are married to that solution. An S3 interface is more universal and allows you to mix and match your tools, even though this comes at some expense.

exAspArk 2y ago

My few cents: - Compute and storage separation simplifies managing a system making compute "ephemeral" - Compute resources can be scaled separately without worrying about scaling storage - Object storage provides much higher durability (99.999999999% on S3) compared to disks - Open table formats on S3 become a universal interface in the data space allowing to bring many other data tools if necessary - Costs at scale can actually be lower since there is no data transfer cost within the same region. For example, you can check out WarpStream (Kafka on object storage) case studies that claim saving 5-10x

potamic 2y ago

Would love the authors to pitch in with their use cases, but I think most people simply do not need sub millisecond analytics. This is mostly replacing typical spark pipelines where you're okay with sub second latencies. S3 is the cheapest, fully managed storage you can get that can scale infinitely. When you're already archiving to S3, doubling it for analytics saves cost and simplifies data management.

hipadev23 2y ago
Sub-millisecond? Literally nobody in large data analysis is doing that. Can you cite some sources or show me what sort of setups (with any DB tech) that you're able to run meaningful queries on S3 sources with even sub-second latency?
- broner 2y ago
  I'm doing this with clickhouse querying parquet files on S3 from an EC2 instance in the same region as the S3 bucket (yes DuckDB pretty similar). S3 time to first byte within AWS is 50ms and I get close to saturating an big EC2 instance's 100Gb link doing reads. For OLTP type queries fetching under 1 MB you'll see ~4 round trips + transfer time of compressed data so 150-200 ms latency.
  - hipadev23 2y ago
    Are you using s3 local cache? Do you have heavy writes? What type of s3 disk type, if any, are you using? (s3, s3_plain, s3_plain_rewritable)? Or are you just using the s3 functions. Clickhouse is amazing but I still struggle getting it working efficiently on s3, especially writes.
    - broner 2y ago
      My workload is 100% read. Querying zstd parquet on s3 standard. Neither clickhouse nor duckdb has a great s3 driver, which is why smart people like https://www.boilingdata.com/ https://www.boilingdata.com/ wrote their own. I compared a handful of queries and found DuckDB makes a lot of round trips and Clickhouse takes the opposite approach and just reads the entire parquet file.

ozgrakkurt 2y ago

It is immense amounts of marketing and being lazy to implement actual local storage with replication etc. It just makes everything easier and is marketed a lot.

deleted 2y ago

[deleted]