4 ms·
Appending log based file system has a lot of appeals. A lot of hard problems, like atomic write, become trivial. A log based FS shares similar properties as t
by ww520 7y ago
Appending log based file system has a lot of appeals. A lot of hard problems, like atomic write, become trivial. A log based FS shares similar properties as the transaction log in RDBMS, making consistency and recovery easy. Write performance is fantastic. With enough cache memory, read performance should be good, too. The only down-size is the performance hit when doing garbage collection. More research in this area would be helpful.
- futureastronaut 7y agohttps://en.wikipedia.org/wiki/Journaling_file_system https://en.wikipedia.org/wiki/Journaling_file_system https://en.wikipedia.org/wiki/Page_cache https://en.wikipedia.org/wiki/Page_cache
- ww520 7y agoJournaling FS only puts metadata and FS commands in log, not structuring the FS itself as an appending log.
- Scriptor 7y agoHaven't log-based file systems been a thing for a while now? https://en.wikipedia.org/wiki/Log-structured_file_system https://en.wikipedia.org/wiki/Log-structured_file_system
- notacoward 7y agoThey have, but they had fallen way out of favor for a couple of reasons - write amplification, cost/complexity of segment cleaning, poor performance for some workloads, etc. They've experienced a bit of a resurgence thanks to flash and SMR, though. Personally, I've always thought log-structured filesystems were elegant and wish they'd been more actively developed during the long "winter" during which their COW/write-anywhere cousins stole all the limelight.
- ww520 7y agoWas hoping there are more research attempt at the Log Structured FS. For example, - Borrow some ideas from generational garbage collection. Young generation in SSD (or mirrored in RAM) with copying GC to get rid of old versions of fast changing data blocks. - Utilize some deduplication techniques with content based signature.
- kjeetgill 7y agoI think elements of that generationality are the foundations of the Log Structured Merge Trees used by KV Stores like LevelDB and RocksDB. Atleast I think that's the same concept, I'm not well read on filesystems.
- ww520 7y agoYes, LSMT is a good example of pushing the idea of a hybrid append log and in memory data structure further. However, LSMT is for relatively smaller data set, i.e. ordered key-value. It has worse write amplification than a simple append log. The level-0 memtable flushed to the write-ahead-log counts as one write. Writing to the level-1 sorted files counts as 2nd write. Merging the sorted files counts as 3rd write. There're 2~3 writes per change. Also it doesn't offer help to address the frequent update block problem. All versions of a data change are written to disk. A merge is needed to get rid of the old versions. But it has a number of good implementation ideas that can be borrowed.
- burmecia 7y agoTotally agree. Log based file system makes things much easier, such as atomic write, data consistency and failure recovery. Along with COW semantics, transaction isolation also becomes easier. By its append-only writing nature, file versioning is trivial as well. Garbage collection is performance downside indeed, but it can be mitigated by COW switching and delayed bulk collection IMHO. Overall, the cost is worthwhile and that is what I am currently doing in ZboxFS: https://github.com/zboxfs/zbox https://github.com/zboxfs/zbox.
- ww520 7y agoThat looks promising. Worth taking a look.