4 ms·
Opinionated take: ingestion should be treated as a streaming reorganization workload, separate from whatever a "database" is. You do not even need to change th
by mrlongroots 2mo ago
Opinionated take: ingestion should be treated as a streaming reorganization workload, separate from whatever a "database" is.
You do not even need to change the Iceberg spec, although you could. Just rejig things a bit to produce better statistics (think liquid clustering but on the write path). You still write the same dataset queried the same way, but automatically get better query performance. Workload owners should be able to decide the tier at which this streaming reorganization happens: they know their data staging and bandwidth constraints the best.