13 ms·
DwarFS – Deduplicating Warp-Speed Advanced Read-Only File System
- nrdvana 3y agoThis looks really cool! though its a bit limited since it is a FUSE module and not a kernel driver, and unlikely to become a kernel module since it is written in C++ with large dependencies :-\ Would it be possible to take the core design changes here and apply them to squashfs, and maybe propose a next major version of the squashfs internal format to make all these things possible?
- dgfitz 3y agoGiven the prose in the README I don’t get the impression the author has much interest in that.
- ramses0 3y agoI've heard that SMB-fs is "better" than FUSE. More cross-platform, can also be implemented in user-space, and less likely to get jammed up/deadlocked by slow network calls inside the kernel.
- pxx 3y agoYou've heard from where? Are there any extant specialty fs that expose an smb interface?
- ramses0 3y agoWhoops: WebDAV: https://news.ycombinator.com/item?id=39417503 https://news.ycombinator.com/item?id=39417503 SeaweedFS supports WebDAV. https://github.com/seaweedfs/seaweedfs/wiki/WebDAV https://github.com/seaweedfs/seaweedfs/wiki/WebDAV I'm not able to find if both/restic supports mounting backups as WebDAV, but in theory there's nothing stopping you. It's 100% user space (expose a rest service) and supported by a bunch of file-browsers with a bit of a network aware component to it as well.
- pxx 3y agoI feel like 9p is a better competitor in this space, and I've actually heard of people using it (e.g. the Windows Subsystem for Linux mounts host filesystems this way and so does Crostini, the ChromeOS equivalent).
- qwerty456127 3y agoHow do you put the files in it if it's read-only? To me it seems such an fs should be immutable-first rather than read-only, i.e. let you create (by copying from another fs) files which just can't be changed (can be moved though) ever after. And such would be what I actually need as I have a lot of big and relatively redundant files which aren't meant to change (e.g. video files, picture originals, distro and backup disk images etc.).
- Cloudef 3y agoIt's meant as squashfs replacement. That is image used for rootfs on live-boot system with overlayfs on top for writable tempfs for example. But I imagine this being useful for many embedded applications in general where you need to save space.
- claudex 3y agoIt's in the man page[1], you put the file at the filesystem creation: $ mkdwarfs -i /path/dir -o image.dwarfs [1]: https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md
- thesnide 3y agoAppend only files might be enough to be fair. And still very useful. As then, you can dedup per block, even with a sliding window. Nicest is that you can do it fully asynchronously, so real time appends are not slowed down.
- Neikius 3y agoThen you add delete and you have a cow fs. Which are many. Some with deduplicarion built in.
- pdimitar 3y agoIt's read-only after you create the image. And then you can use overlayfs on top of it as others mentioned -- useful for playing with live Linux distributions or embedded systems where you never want to change the root bootable image except when you are actually updating the firmware. ...Or, in the author's case, when you have hundreds of directories that have only small differences compared to each other. As other posters in the bigger thread mentioned, this is also very useful for big arcade game collections.
- m463 3y agoI always thought a deduplicating squashfs might be really cool for a read-only MAME system. Since most of the rom files seem to be variants of each other, instead of having zip archives for each game, just put all the raw files in a directory and create a deduplicated squashfs-type filesystem with all of them.
- flomo 3y agoYeah, the original rationale is kinda perversely interesting: > my main use case and major motivation was that I had several hundred different versions of Perl okay... The question is if you want to trust a github filesystem or just spend the $dough to deal with it.
- smueller1234 3y agoIt's not an idle use case by the way. mhx wrote and maintains a library that provides important backwards compatibility for native (typically C based) extensions for Perl across decades of language releases.
- flomo 3y agoNo real doubt here this guy isn't 100% serious, so thanks for the explanation.
- dang 3y agoRelated: DwarFS: A fast high compression read-only file system - https://news.ycombinator.com/item?id=32216275 https://news.ycombinator.com/item?id=32216275 - July 2022 (64 comments) DwarFS: A fast high compression read-only file system - https://news.ycombinator.com/item?id=25246050 https://news.ycombinator.com/item?id=25246050 - Nov 2020 (111 comments)
- donatj 3y agoCan someone explain what this is and when I would use it in layman’s terms? I understand it’s a fuse filesystem but don’t understand quite what it does?
- taneliv 3y agoIt fills a similar niche as https://en.wikipedia.org/wiki/SquashFS https://en.wikipedia.org/wiki/SquashFS but has different data compression characteristics. However, since it's a FUSE only file system, it's difficult to see how it would be used on embedded system firmware, so it could perhaps see use as a distribution mechanism. Similar to tar or zip files, but possibly with (much) better performance for random access, should you need only smaller portion of the whole archive. The author indicates need for keeping multiple similar copies of sets of unchanging files on their computer, and made this to reduce the space needed for them, while retaining the access through the file system. So that is also a use case.
- tambourine_man 3y agoThis is quite interesting for the similarity hash function alone. But the self-congratulatory tone from description is… unusual, to say the least.
- deleted 3y ago[deleted]
- naikrovek 3y agoWell it performs extremely well, and if I wrote it, I’d be proud of it, too.
- tambourine_man 3y agoSo would I, but I’m uncomfortable with that tone even in reading, let alone writing. Just an observation, something that jumped out.
- genewitch 3y agocan you give an example of "self-congratulatory tone"? is it things like, "this is still 10 times larger than dwarfs", because a statement of fact can not be insolent. I actually only went through a significant portion of the readme (through the CromFS part) because of your comment, and i just don't see it. I see a person who wrote actually useful software that is multi-platform and gives the positives and negatives of the software they wrote compared to alternatives available today. In every test DwarFS compared favorably, and on tests where one aspect was marginal, the DwarFS code was better in other regards: power at the wall, extract/read times, etc. How would it be better presented by a solo developer?
- tambourine_man 3y ago“While this is already impressive, it gets even better.” I can almost hear “and if you call now, you get this amazing towel for free!” But judging by the comments here, it seems I’m more sensitive to this tone than most.
- eigengrau 3y agoNot to be confused with dwarffs – https://github.com/edolstra/dwarffs https://github.com/edolstra/dwarffs.
- emayljames 3y agothere are maybe gonna be a lot of mis-spelled pull requests and confusion.
- kseistrup 3y agoSo how much space would the entire english wikipedia take up on this filesystem, I wonder.
- hnarn 3y agoOnly one way to find out :-)
- mhx77 3y agoI've got a (relatively old) snapshot of the English Wikipedia that I'm using for testing. The snapshot is around 200 GiB in 14,000,000 files and compresses down to an 11 GiB DwarFS image.
- kseistrup 3y agoThat's a substantial reduction. Thanks for testing this!
- mhx77 3y agoIt's ever so slightly smaller than a .tar.xz of the same data. The main difference being that you don't have to fully extract it in order to access the data.
- karteum 3y agoI guess it's without the pictures then ? Because if I compare with the zim file format (which is optimized for this use-case) https://kiwix.org/en/what-is-the-size-of-wikipedia/ https://kiwix.org/en/what-is-the-size-of-wikipedia/ I read "As of October 2022, the Full English Wikipedia (ca. 6.5 million articles), with images will use up 91GB of storage space (German and French, the second-largest: 36 GB). (...) If you can do without the images (what we call the nopic version), then you are down to 46 GB."
- mhx77 3y agoCorrect, there are no images in the data except for 68 PNGs. It's just HTML files.
- mnw21cam 3y agoThis is really neat. Feature request: Add a "library" option to give mkdwarfs a list of files that should be loaded into the dedup mechanism first, but not stored, allowing the image to be even smaller if the contents of the file can be retrieved from that library instead. Bonus points if you can specify a dwarfs image as a library and have it sensibly use the files contained in it. Then you have the basis for a deduplicating incremental backup system. Currently, I have a system I wrote that will take a single file and a list of library files and produce a compressed deduplicated file that can re-create that single file using the library, which is great if you use tar to create that single file, but a little unwieldy when coming to decompress and restore everything. The bonus of making it a proper mountable filesystem instead is that then it's a proper mountable filesystem and retrieving single files is a doddle. My use case is that I have students, and I have given them coursework, which involves them logging in to a Linux machine and hacking away. I want to store regular snapshots of their work so that I can keep a backup for their sake but also so I can see a progression of development to try to work out if they are cheating (yes, I have had to deal with this), but I don't want to store 100 copies of the same fairly large files.
- nextaccountic 3y agoborg and restic/rustic are alternatives that do this multi layered backup thing. they can be mounted as a filesystem as well (the rustic reimplementation of restic however doesn't support this yet) https://www.borgbackup.org/ https://www.borgbackup.org/ https://restic.net/ https://restic.net/ https://github.com/rustic-rs/rustic https://github.com/rustic-rs/rustic
- jszymborski 3y agoCan't say enough nice things about Borg and it's dedup. Saves me a lot of money in cloud storage.
- chuzz 3y agowould this be doable with overlayFS using a tmpfs to prepare a snapshot?
- 3y ago
- ku1ik 3y agoAll tech aside, this is the best name for a filesystem, period.
- planetis 3y agoNeat. This FS is used predominantly in torrents of pirated steam games by a certain group that advertises their work with "play without extracting".
- nextaccountic 3y ago> "play without extracting". This kind of feature is really important if you to encourage people to seed even after their download is completed.
- xialvjun 3y agoipfs mount. Since ipfs is content-addressable storage, it's readonly. If we make it cow, encourage people to seed is easy.
- pdimitar 3y agoIPFS is slow as molasses and their solution was to introduce paid pinning services. Complete failure. And then they announced they will willingly and proactively delete any hash that any legislative agency tells them to, and it was dead in the next minute.
- cyanydeez 3y agoMost distributed systems simply don't live up to their theory. Having a hub is simply always easier and consumers hopping temporally from hub to hub is just the most efficient mechanism.
- pdimitar 3y agoAgreed. The idea behind IPFS is nice but we need to combine it with all the good lessons from the torrent software and stuff like NNCP in order to have something truly distributed, resilient, fault-tolerant and automatically replicated.
- 7bit 3y agoFor rock and stone!
- emayljames 3y agoThis will have huge benifits for Linux ISO images, in reducing there size by using this instead of squashfs.
- reify 3y agoVery cool for linux gaming. Highly recommended. works great for me. https://gitlab.com/jc141x/setup https://gitlab.com/jc141x/setup
- rcthompson 3y agoWhat does the code in this repo actually do, and how does it use DwarFS? The README just explains how to install it.
- karteum 3y agoI would love to see a comparison with EroFS (https://en.m.wikipedia.org/wiki/EROFS https://en.m.wikipedia.org/wiki/EROFS )
- mhx77 3y agohttps://github.com/mhx/dwarfs?tab=readme-ov-file#with-erofs https://github.com/mhx/dwarfs?tab=readme-ov-file#with-erofs It's been a while since I did that comparison, so the results could be significantly different now.
- mhx77 2y agoFWIW, I've updated the comparison using the latest versions of EROFS and DwarFS. There definitely have been improvements to EROFS in the meantime. DwarFS is still orders of magnitude faster in creating the file system and can achieve significantly better compression, but throughput is very similar unless you start optimizing the EROFS image for better compression.
- karteum 3y agoThanks ! One question though : why do you say "as it's pretty obvious that the domains in which EROFS and DwarFS are being used have extremely little overlap. DwarFS will likely never be able to run on embedded devices" ? (I mean : as far as I know EroFS was mostly used in smartphones, which are actually powerful devices !)
- mhx77 3y agoThat's definitely a fair point! DwarFS actually works perfectly fine on 64-bit ARM and would likely work fine on a smartphone as well. Still (and I might be completely wrong about this), I think the primary goal of EROFS is to consume as little resources as possible when the file system is accessed and to be able to run on much less capable hardware, including 32-bit systems. DwarFS primarily cares about maximizing compression as long as it doesn't negatively impact performance (access times and throughput). This involves a certain amount of caching / pre-fetching and assumes there's plenty of memory available. This can be configured to a certain extent, but my pessimistic assumption is that DwarFS would use significantly more memory than EROFS. Might be worth actually backing this by numbers! :)
- hnarn 3y agoThis sounds like something that could be useful for immutable distros like Vanilla and others, but I wonder how practical it is if it’s not part of the kernel.
- aloer 3y agoI would like to archive every web page I ever visit. Something like this sounds very useful but it would need to be updated periodically. I’m thinking a hybrid approach could make sense here. Historical data is thrown into a big archive dwarFS while new data of e.g. the current day is kept in a simple normal folder. Every now and then this would be merged together. Skimming through the docs I can’t tell if it will be possible to recreate a new dwarFS by taking an existing one and adding just a few more files, or if I would need to create a completely new one which also means I need to temporarily have enough space for twice the archive size
- pyinstallwoes 3y agoI want exactly the same.
- mgaunard 3y agoToday I discovered _mm512_mask_compressstoreu_epi64. Neat instruction.
- eigenvalue 3y agoVery cool idea. I wonder how it would work to first turn every file you want to store into a stream of symbol files using a fountain code like RaptorQ, and then trying to compress all of the symbol files together using something like Z-standard with a big dictionary. It would probably be too slow to use for many file system applications, but I would think it would be pretty good at exploiting redundancy across files and getting good compression ratios.
- sydbarrett74 3y agoKudos to the developer, and thanks for sharing this.
- mrbluecoat 3y agoOpenWrt would be a nice use case: https://forum.openwrt.org/t/openwrt-21-02-x-ath79-tiny-luci/105312/27 https://forum.openwrt.org/t/openwrt-21-02-x-ath79-tiny-luci/...
- ysofunny 3y agoyes, now let's put all public government data on one of this so that we can all inspect the public institutions which we use to rule over ourselves and achieve the next level of real digital democracy: full institutional public transparency ...as if... now, let me spit out all this purple democratic kool aid
- 20after4 3y agoI'm tempted to try using this for container images. I would imagine the savings could be pretty significant.
- pdimitar 3y agoIf you do, you definitely should blog about it and promote it. I'm very interested in knowing the results.
- all2 3y agoIt's a userspace only FS. Can we use that for container images?
- sonicanatidae 3y agoI'm curious to know, but I believe a major portion of this idea comes into play when the data is massively redundant. I typical container (say, a vhdx file) normally wouldn't apply. Did I misunderstand?
- 20after4 3y agoNo you didn't misunderstand, however, I do believe there would be quite a bit of redundant data in the layers of a docker image, especially when the Dockerfile hasn't been optimized and was just haphazardly constructed.
- sonicanatidae 2y ago
- vendiddy 3y agoPretty neat! "Clustering of files by similarity using a similarity hash function" does anyone have an intuition of how similarity hashes work?
- mhx77 3y agohttps://en.wikipedia.org/wiki/Locality-sensitive_hashing https://en.wikipedia.org/wiki/Locality-sensitive_hashing https://en.wikipedia.org/wiki/Nilsimsa_Hash https://en.wikipedia.org/wiki/Nilsimsa_Hash https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md#nilsimsa-ordering https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md#nils...
- all2 3y agoFor the curious arch users: https://aur.archlinux.org/packages?O=0&K=dwarfs https://aur.archlinux.org/packages?O=0&K=dwarfs
- rezaprima 3y agoDo I miss ZipMagic that makes us use zipfile like folder in Windows explorer, including read write operation. It is diacontinued and its name is reused by another tool so I can only find its description in Amazon [1] [1] https://www.amazon.com/Ontrack-Data-International-99-00030-000-ZipMagic/dp/B00002S9TW https://www.amazon.com/Ontrack-Data-International-99-00030-0...
- rezaprima 3y ago... and that's because my immediate usecase thought is compressing node_modules