6 ms·
DwarFS: A fast high compression read-only file system
- metadat 4y ago"DwarFS compression is an order of magnitude better than SquashFS compression, it's 6 times faster to build the file system, it's typically faster to access files on DwarFS and it uses less CPU resources." Credit and thanks to coldblues for alerting me about this! https://news.ycombinator.com/context?id=32212870 https://news.ycombinator.com/context?id=32212870
- pdimitar 4y agoI thought I was the one mentioning it? https://news.ycombinator.com/item?id=32211651 https://news.ycombinator.com/item?id=32211651
- metadat 4y agoYou did mention it, although with less context and somehow it didn't stick in my brain. Regardless, it wasn't intentional to not give you credit, and I am happy to give you all the credit sir, madame, or they!
- pdimitar 4y agoWell, don't get me wrong -- not that I care that much but I found it impossible to find the other reference to it that you mentioned so I was wondering if you (or me) made a mistake. Thanks for clarifying!
- karteum 4y ago"DwarFS compression is an order of magnitude better than SquashFS compression, it's 6 times faster" : I suspect this is on a highly specific test-case and no generalities can be made... N.B. there is also EroFS https://www.kernel.org/doc/html/latest/filesystems/erofs.html https://www.kernel.org/doc/html/latest/filesystems/erofs.htm...
- dumbfounder 4y agoFavorite part: “I had several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space”. That’s a lot of Perl.
- simcop2387 4y agoI have this same kind of thing, for running https://perl.bot/ https://perl.bot/ and related services. I'm using BTRFS however (due to a long story and actual reasons) and use it to dedupe, compress and discard a file backed filesystem for them. I'm at a logical store of about 46GB with an on disk size of 25GB. Most of that is the fact that I have several hundred (550ish last i counted) libraries and modules installed into each install of perl. This means that they dedupe incredibly well since they're basically perfect copies, and most of it is text based so it compresses well too. I've looked at DwarFS before for this same use case but the fact that it's read only makes it more difficult to handle since I'd have to have an uncompressed version sitting out elsewhere. Though I've now got the hardware to actually put that all into a CI/CD that generates the image. I might actually work on that once I get my turing pi 2 boards since I want to port this whole setup to ARM as well as x86_64. I might also put it on my RISC-V board too but it's too slow so I think it won't be as useful. EDIT: fixed the tbd size, it was taking a while to calculate.
- liuliu 4y agoCan you have OverlayFS on top of it to make it writable and periodically generate new base DwarFS from that overlay?
- simcop2387 4y agoI'd imagine so, that might be a good strategy for doing it and you could measure the newly setup stuff fairly easily.
- pdimitar 4y agoPeople do something similar with e.g. Raspberry Pi machines with a microSD card; they make an overlayfs setup where the OS logs et. al. get written to memory and are only being synced to the main microSD card once an hour or so -- to avoid amortizing it too quickly. So I'd imagine making something like that but with DwarFS below would be quite easy although it'd require you to set it up by hand. Still, once done it'll likely be a rock-solid setup for a long time.
- rmatt2000 4y agoA few years ago, I designed an incredibly fast write-only file system, but for some reason couldn't leverage it into a commercial product that anyone was willing to buy.
- notamy 4y agohttps://devnull-as-a-service.com/ https://devnull-as-a-service.com/ (:
- osigurdson 4y agoI didn’t dig into the details, but unless DwarfFS is a joke, I assume they mean the system supports create, read and delete operations.
- askvictor 4y agoI'd guess you build a fs image from an existing filesystem, so you'd only have read operations.
- klyrs 4y agoThere's plenty of read-only media out there. I can see this being useful for container deployment -- you've got a database somewhere that you have read/write access to, but the deployed code doesn't need to be updated in the container. A readonly filesystem is a great way to enforce principle of least privilege.
- gabagool 4y agohttp://www.supersimplestorageservice.com/ http://www.supersimplestorageservice.com/ > The Super Simple Storage Service (S4) is a new innovation in cloud storage. Our advanced write-only storage provides the highest security, lowest cost, and simplest management available.
- vardump 4y agoSeems like this a pretty nice read-only filesystem. Shame that it can't be used much commercially in many cases due to GPLv3 license. This could improve so many embedded systems currently using SquashFS.
- isr 4y agoWell, if you want to build a commercial product using dwarfs, you could always (shock, horror) contact the dev and *pay* him to license it to you under a bsd-like license (or whatever). (the dwarfs dev would have to work it out amongst all who contributed to his repo)
- vardump 4y agoWho said I'd want (or need) to use it in a product? What I'm saying is that due to the license fewer companies want to use and thus we're getting a bit worse products as a consequence. Of course I completely agree the developer should get licensing income (heck, that's in my self interest as well!), but what's going to really happen is that the cheapo companies just make do with less and use SquashFS or whatever instead. It's a tragedy of commons there's so little will to donate for or crowdsource open source projects and that they're taken for granted.
- fps_doug 4y ago> It's a tragedy of commons there's so little will to donate for or crowdsource open source projects and that they're taken for granted. Which is not made better by your sentiment? I mean GPL at least forces you to either try negotiating a different license with the author (for money ideally), or suck it and use something inferior. Using a permissive license that allows multi-billion dollar companies to use your code for free certainly doesn't help changing the mind set of "open source is other people working for me for free".
- vardump 4y ago"I mean GPL at least forces you to either try negotiating a different license with the author (for money ideally)" Sounds great in theory, but even for a popular open source library developer this happens so rarely that it rarely pays the bills regardless of license. "Using a permissive license that allows multi-billion dollar companies" Most embedded devices by far are not developed by multi-billion dollar companies. Their legal departments are also actively steering away from GPL licenses. "try negotiating a different license" In my experience, by far the most companies don't want to negotiate anything. If there's a product with a set price, yes, then it might be purchased. They typically also want some kind of product support with it. "use your code for free certainly doesn't help changing the mind set of "open source is other people working for me for free"." How to solve this? Ideally the library developer would get paid AND the consumers get more value for their money. This would encourage more developers to write useful libraries that provide great value in the big picture, but are uneconomical or otherwise too much trouble to deal with on an individual company or product level. As it is, everyone seems to lose.
- staticassertion 4y agoMaybe this is a really dumb question but how does one use a read-only filesystem? Can you mount it as writable temporarily or something? Or is it that you create a compressed 'file' that you can mount as a file system? Like a zip file kinda I guess?
- koolba 4y agoYou mount it as a read only file system to a mount point. It’s like putting in a CD (remember those?) into a drive. The source data can be a raw block device or more likely a local file. It doesn’t matter as either way the kernel is just reading blocks of bytes.
- staticassertion 4y ago> remember those? Not really tbh. I haven't used a CD in my adult life. I remember "burning a CD" was a thing and that's about it.
- dataflow 4y agoHow does this compare to WoF compression in NTFS (which uses LZX)?
- lathiat 4y agoSounds like the main benefit is hash-based dedup. You can do this with ZFS and borgbackup - they'd likely be a better comparison - but most other compressed filesystems won't do it. Most compressed filesystems will fail to dedup stuff because for various performance related reasons the compression window is usually quite small maybe up to 128kb and the files are often not sorted in the archive such that small files would manage to compress anyway in that window. EDIT: Apparently squashfs does include file de-dupe however DwarFS is fixing what I basically said above. It sorts files in order of similarity so that you can compress accross file boundaries as well as just removing complete duplicates. That's pretty cool. "Clustering of files by similarity using a similarity hash function. This makes it easier to exploit the redundancy across file boundaries." This kindof thing is more difficult to do on a writable filesystem though not impossible.
- kretaceous 4y agoSlightly OT. What resources would people here suggest for learning about file systems? I see a lot of new file systems like zfs, btrfs, etc. I looked up for resources but couldn't find anything substantial. I want to learn how they work so that I can appreciate projects like this and compare them. I looked into the Build Your Own X repo but didn't find anything. I found a book called Practical File System Design: The BeOS file system but it's apparently dated, and I'm not sure I want too much of a deep dive.
- adastra22 4y agoThe BeFS book is seriously good. I would start there.
- 8n4vidtmkvmk 4y agois zfs still considered new?
- mprovost 4y agoI remember learning about filesystem design from the classic "demon" book (The Design and Implementation of the 4.4 BSD Operating System by Marshall Kirk McKusick) which has a lot of details about the original Fast File System.
- dcj4 4y agoI think the author overlooked that a read-only file system is of absolutely no practical use. If you can't write to it, then there will never be any data in it to be read.
- unsafecast 4y agoRead-only means you can't change it _after_ you initially put all your files in it.
- pdimitar 4y agoThere are plenty of data archival projects that can benefit from DwarFS. Stuff like e.g. old magazines or ROM collections or scanned books/comics that don't have copyright attached etc., don't need modification and if the collection is big enough then the deduplication can reduce the final size of the archive by a lot.
- hawk_ 4y ago> Clustering of files by similarity using a similarity hash function. Does that mean that I can store all those 100s of photos taken at slightly different angles, in an efficient way?
- thewakalix 4y agoThat depends on whether they are bytewise similar, which in turn depends on the details of the image encoding of the format you’re using. I would expect that, by default, no.
- lvncelot 4y agoThat does lead to the question whether, with enough files, storing a lot of similar photos in uncompressed png is more space efficient in DwarFS than it is storing them in jpg.
- jcelerier 4y agoTangentially related - what's a good option for a cross-platform (portable to all platforms with a filesystem) read-only virtual file-system today like e.g. quake's pk3 file format ? e.g. let's say I want to access a few ten of thousand small files fast, much faster than what e.g. NTFS allows since I know that I'll likely have to read more-or-less all the files and I can mmap the whole thing, what are my options? My prime concern is having an api such as handle = vfs_fopen("/my/file1.txt") pointer_to_the_file_bytes = vfs_map(handle, <start offset>) which would be as fast as possible. compression, encryption aren't needed.
- jcelerier 4y agoended up biting the bullet and started https://github.com/celtera/uvfs https://github.com/celtera/uvfs
- dspillett 4y agoWould a RAM disk fit the bill? Just read all the contents from the copy in non-volatile storage at boot. Cross-platform then by virtue of using what-ever RAM-based filesystem or block-device options are commonly available on the target OS. For “as fast as possible” you'll need to experiment and benchmark with your workload. Which filesystem is optimal may depend on how you are laying out the data and where the latency/throughput sensitivities are in your use case and the given filesystems. > My prime concern is having an api such as... Having a different API other than it looking like a filesystem would make cross-platform more of a concern, as you then have a data access library not a general filesystem. It will likely to be necessary for best performance though: any filesystem is going to have significant overheads (orders of magnitude) compared to being able to map chunks of the data directly into your process' address space. If abandoning a generic filesystem, perhaps something like sqlite with an in-memory table/db (https://www.sqlite.org/inmemorydb.html https://www.sqlite.org/inmemorydb.html)? Again like the ramdisk option just load up the content from permanent storage on first use.
- jcelerier 4y ago> Would a RAM disk fit the bill? A normal unprivileged app cannot create RAM disks easily on any OS as far as I know ; also it wouldn't really work on e.g. WASM > as you then have a data access library not a general filesystem. That's fine for me - although I don't see a particular difference between either, a filesystem is just a system to access files, whatever that means
- nora-puchreiner 4y agoIt looks good for NixOS (or Guix) store.