5 ms·
My understanding is that deduplication has generally been not worth the effort for a general use filesystem. The cost in memory, cpu, and sheer complexity does
by jandrese 7y ago
My understanding is that deduplication has generally been not worth the effort for a general use filesystem. The cost in memory, cpu, and sheer complexity doesn't pay off in enough storage savings. Disk space is relatively cheap. Filesystem crashes/corruption are not.
- vardump 7y agoYeah, block level deduplication is not useful for files on a typical consumer laptop. I can imagine it's even less useful for files on iOS devices. However it is often great for virtual machines and other software development related things, as long as all the underlying block sizes and alignments match. You really do want strong integrity guarantees though. One corrupted block can cause a lot of damage.
- Scaevolus 7y agoThe filesystem encryption used on iOS probably makes deduplication even less effective.
- vardump 7y agoDepends on the layering order. You can do both dedup and encryption in the same time, if the encryption occurs on a lower layer.
- eridius 7y agoEncryption in iOS is per-file. I'm not sure the OS really wants to go ahead and individual decrypt large numbers of files just to check if there are any duplicated blocks.
- FPGAhacker 7y agoDisk space isn’t cheap on a Mac laptop.
- mikepurvis 7y agoYeah, but that's a customer cost, not an Apple cost. In any case, dedup just makes things more confusing to the customer— like, "you have X GB in files on your computer, but more than N-X GB free space because of magical deduplication." Or worse, "You deleted all those files, but didn't actually free up any space because they were dupes. So sad."
- Wowfunhappy 7y ago> In any case, dedup just makes things more confusing to the customer— like, "you have X GB in files on your computer, but more than N-X GB free space because of magical deduplication." Or worse, "You deleted all those files, but didn't actually free up any space because they were dupes. So sad." For what it's worth, this can already happen in APFS. While they don't have real deduplication, if you copy a file within Finder, that copy won't take up extra space.
- jjoonathan 7y agoHow granular is the COW? If you touch a byte, does just a block get copied or the whole file?
- mindajar 7y agoJust a block
- cmurf 7y agoIt depends on the application. Often applications do a kind of COW on their own, behind the scene they write out a new file with a randomized naming scheme, when that succeeds, do a rename/replace. So in this case you're getting a new file, as in, a file with a new inode. If your application is smarter, then it could just COW one block, whatever the minimum block size is for that file system. e.g. for Btrfs it's page size, so x86 that's 4KiB, ppc64le it's 64KiB. I'm not sure what it is for APFS, I'd guess 4KiB. It could actually be inefficient depending on the use case, e.g. a 1MiB file with a single byte change in the middle, if it were a single 1MiB extent, becomes three extents with COW: 0-500KiB - extent 1 500-504KiB - hole where stale data is 504KiB-1MiB - extent 3 4KiB - extent 2, with new data, a copy of 4095 bytes from the above hole, with 1 bytes of new data. Edit: 2nd paragraph suggests the application would COW if it's smarter; that's not correct. If it's smarter, it would request an overwrite of some portion of the file and the file system is what would COW (if it's a COW file system). Granularity depends on a combination of application and file system. Edit 2: More detailed answer on Btrfs. The whole file system on a zram device is about 260 lines using 'btrfs inspect-internal dump-tree' command. I'll excerpt just the file extents. 1. 'dd if=/dev/zero of=/mnt/test/bunchofzeros.txt bs=M count=1' results in this: item 7 key (257 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 7 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 0 nr 1048576 ram 1048576 extent compression 0 (none) 2. Using vi to change one character and save as the same file, ':wq', I get: item 7 key (262 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 9 type 1 (regular) extent data disk byte 7401472 nr 1052672 extent data offset 0 nr 1052672 ram 1052672 extent compression 0 (none) So you can see it's a new file, new inode, and looks like length is changed too, maybe something to do with encoding. 3. Delete that file and create a new one with the same 'dd' command as in 1. item 7 key (266 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 11 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 0 nr 1048576 ram 1048576 extent compression 0 (none) [root@flap ~]# dd conv=notrunc if=/dev/urandom bs=1 seek=524290 count=1 of=/mnt/test/bunchofzeros.txt [root@flap ~]# hexdump -C /mnt/test/bunchofzeros.txt | more 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00080000 00 00 5d 00 00 00 00 00 00 00 00 00 00 00 00 00 |..].............| 00080010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00100000 item 7 key (266 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 11 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 0 nr 524288 ram 1048576 extent compression 0 (none) item 8 key (266 EXTENT_DATA 524288) itemoff 3356 itemsize 53 generation 14 type 1 (regular) extent data disk byte 5242880 nr 4096 extent data offset 0 nr 4096 ram 4096 extent compression 0 (none) item 9 key (266 EXTENT_DATA 528384) itemoff 3303 itemsize 53 generation 11 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 528384 nr 520192 ram 1048576 extent compression 0 (none) Clever. item7 and item9 actually reference the same original 1MiB data extent, but use offsets to point to the unchanged portion "creating" a 4KiB hole, and that hole is filed by item 8, a 4KiB extent containing the change. So it really did only COW one block.
- Damogran6 7y agoNot only that, but a Server with 100 VM's is going to have a substantial percentage of each OS image as identical...not so much on an average consumer laptop.
- deleted 7y ago[deleted]
- api 7y agoSame with transparent compression in most cases.