3 ms·

I'm very curious about the convergence of the best in class fast OLAP products (StarRocks, ClickHouse) with Trino. It sounds like everybody is going for decoupl

by a34729t 2mo ago

I'm very curious about the convergence of the best in class fast OLAP products (StarRocks, ClickHouse) with Trino. It sounds like everybody is going for decoupled compute/storage, using S3 or similar as the storage layer, and thus forgoing colocated joins (ok I know ClickHouse joins suck)... So what does this mean for ingestion (and indexing)? Iceberg V3? Paimon? Bespoke ingestion through the DB engine to do the indexing?

mrits 2mo ago
Clickhouse joins have been improving almost every month for the last couple of years. Maybe they still suck but a lot less
jimmyl02 2mo ago
It's also interesting how Clickhouse / Starrocks can now also act as a query planner and executor on top of non-native formats (ex. Iceberg). I assume the native formats will always be faster / more optimized but the need for Trino as a separate executor while running either of these databases seems to be close to gone.
- a34729t 2mo ago
  The benefit of StarRocks/ClickHouse over Trino is that you get secondary indices, but that means you have to do the indexing somehow. Native format is faster (especially for colocated joins), but it's way more expensive if you have to run a bunch of separate storage nodes vs just using S3, especially your query volume isn't that high. I liken it to the BigQuery cost model, where storage is effectively free.
- hodgesrm 2mo ago
  > I assume the native formats will always be faster / more optimized Actually query on Iceberg is end-to-end faster on ClickHouse than MergeTree native format on disk. It's mostly a matter of how much compute you throw at it. Native MergeTree still wins in cases that depend on use of indexes to reduce I/O but scan speed is no longer an issue.
efromvt 2mo ago
I’ve historically read this as ‘open format compatible’ but ‘native preferred’ - where this opens up market space and dev velocity - but it’ll be interesting to see if native storage differentiation gets dumped entirely. It just seems like ‘fork and optimize for our engine’ would always be tempting enough that you’d want a native play for when you don’t need the decoupling.
mrlongroots 2mo ago
Opinionated take: ingestion should be treated as a streaming reorganization workload, separate from whatever a "database" is. You do not even need to change the Iceberg spec, although you could. Just rejig things a bit to produce better statistics (think liquid clustering but on the write path). You still write the same dataset queried the same way, but automatically get better query performance. Workload owners should be able to decide the tier at which this streaming reorganization happens: they know their data staging and bandwidth constraints the best.

by a34729t 2mo ago

mrits 2mo ago

Clickhouse joins have been improving almost every month for the last couple of years. Maybe they still suck but a lot less

jimmyl02 2mo ago

It's also interesting how Clickhouse / Starrocks can now also act as a query planner and executor on top of non-native formats (ex. Iceberg). I assume the native formats will always be faster / more optimized but the need for Trino as a separate executor while running either of these databases seems to be close to gone.

a34729t 2mo ago
The benefit of StarRocks/ClickHouse over Trino is that you get secondary indices, but that means you have to do the indexing somehow. Native format is faster (especially for colocated joins), but it's way more expensive if you have to run a bunch of separate storage nodes vs just using S3, especially your query volume isn't that high. I liken it to the BigQuery cost model, where storage is effectively free.
hodgesrm 2mo ago
> I assume the native formats will always be faster / more optimized Actually query on Iceberg is end-to-end faster on ClickHouse than MergeTree native format on disk. It's mostly a matter of how much compute you throw at it. Native MergeTree still wins in cases that depend on use of indexes to reduce I/O but scan speed is no longer an issue.

efromvt 2mo ago

I’ve historically read this as ‘open format compatible’ but ‘native preferred’ - where this opens up market space and dev velocity - but it’ll be interesting to see if native storage differentiation gets dumped entirely. It just seems like ‘fork and optimize for our engine’ would always be tempting enough that you’d want a native play for when you don’t need the decoupling.

mrlongroots 2mo ago

Opinionated take: ingestion should be treated as a streaming reorganization workload, separate from whatever a "database" is. You do not even need to change the Iceberg spec, although you could. Just rejig things a bit to produce better statistics (think liquid clustering but on the write path). You still write the same dataset queried the same way, but automatically get better query performance. Workload owners should be able to decide the tier at which this streaming reorganization happens: they know their data staging and bandwidth constraints the best.