3 ms·
> Reductions in generality are a major mechanism for reducing engineering cost in database engines Yeah, just look at snowflake for a great example of this. T
by AdamProut 5y ago
> Reductions in generality are a major mechanism for reducing engineering cost in database engines
Yeah, just look at snowflake for a great example of this. They outsource the entire bottom half of the database (making data durable) to a blob store (S3, GCS, etc.). That is a huge simplification for them. They don't need to write their own durability and high availability components (no write ahead log, no recovery or replication protocols, etc.). It also makes it completely impractical to run lower latency small write transactions vs snowflake because the data needs to hit blob storage before the transaction can commit (and blob store writes are 10 to 100s of ms of high variability latency). Being a more specific case database (a data warehouse) let them make this trade-off. More general purpose systems can't make this type of trade-off. I wrote a bit on this topic a few months ago if anyone is interested [1]
[1] https://www.singlestore.com/blog/separating-storage-and-compute-for-transaction-and-analytics/ https://www.singlestore.com/blog/separating-storage-and-comp...