4 ms·
I would love to learn from database experts here: Why some databases choose to use segment files as persistence and why some other databases choose to use key-
by didip 5y ago
I would love to learn from database experts here:
Why some databases choose to use segment files as persistence and why some other databases choose to use key-value as persistence?
From my perspective, using key-value is very convenient because you can then outsource persistence to a distributed, fault tolerant, key-value storage.
- Something1234 5y agoThere was another article posted today about LSM trees, and how they use segment files. I think at the end of the day the "distributed, fault tolerant, key-value storage" is using segment files too. By having your database use segment files directly you're just skipping a middleman.
- binarray2000 5y agoAdding for (future) reference https://news.ycombinator.com/item?id=30269286 https://news.ycombinator.com/item?id=30269286
- radiowave 5y agoNot sure that I qualify as an expert, but in the case of Postgres the choice was probably influenced by the fact that readily available, distributed, fault tolerant, key-value storage wouldn't exist for at least another decade.
- dikei 5y agoThere are no conflict: key-value is an interface, segment files is an implementation. You can implement a key-value using segment files too.
- inopinatus 5y agoIt's all very well until you discover the distributed, fault-tolerant, key-value storage is implemented using a database.
- jimbokun 5y agoBut what if that database is using a key value store?
- inopinatus 5y agoit's turtles all the way down
- kklisura 5y agoit's tuples all the way down
- inopinatus 5y agoperfect, thank you
- mdavidn 5y agoMany query plans (e.g. sorts, joins, aggregation) involve scanning large swaths of data. The importance of sequential access for such operations cannot be overstated. Storage, processors, and networks are designed to make sequential access performant. Some SQL databases are backed by key-values stores, but they generally store large segments in the values rather than individual records. (e.g. Snowflake, Athena)
- polskibus 5y agoSegments also allow you to store some info about segment statistics, for example min and max for each numeric column which allow optimiser to skip entire segments when filtering. More tricks can be found in Snowflake paper.