5 ms·

Disk space isn’t cheap on a Mac laptop.

mikepurvis 7y ago
Yeah, but that's a customer cost, not an Apple cost. In any case, dedup just makes things more confusing to the customer— like, "you have X GB in files on your computer, but more than N-X GB free space because of magical deduplication." Or worse, "You deleted all those files, but didn't actually free up any space because they were dupes. So sad."
- Wowfunhappy 7y ago
  > In any case, dedup just makes things more confusing to the customer— like, "you have X GB in files on your computer, but more than N-X GB free space because of magical deduplication." Or worse, "You deleted all those files, but didn't actually free up any space because they were dupes. So sad." For what it's worth, this can already happen in APFS. While they don't have real deduplication, if you copy a file within Finder, that copy won't take up extra space.
  - jjoonathan 7y ago
    How granular is the COW? If you touch a byte, does just a block get copied or the whole file?
    - mindajar 7y ago
      Just a block
    - cmurf 7y ago
      It depends on the application. Often applications do a kind of COW on their own, behind the scene they write out a new file with a randomized naming scheme, when that succeeds, do a rename/replace. So in this case you're getting a new file, as in, a file with a new inode. If your application is smarter, then it could just COW one block, whatever the minimum block size is for that file system. e.g. for Btrfs it's page size, so x86 that's 4KiB, ppc64le it's 64KiB. I'm not sure what it is for APFS, I'd guess 4KiB. It could actually be inefficient depending on the use case, e.g. a 1MiB file with a single byte change in the middle, if it were a single 1MiB extent, becomes three extents with COW: 0-500KiB - extent 1 500-504KiB - hole where stale data is 504KiB-1MiB - extent 3 4KiB - extent 2, with new data, a copy of 4095 bytes from the above hole, with 1 bytes of new data. Edit: 2nd paragraph suggests the application would COW if it's smarter; that's not correct. If it's smarter, it would request an overwrite of some portion of the file and the file system is what would COW (if it's a COW file system). Granularity depends on a combination of application and file system. Edit 2: More detailed answer on Btrfs. The whole file system on a zram device is about 260 lines using 'btrfs inspect-internal dump-tree' command. I'll excerpt just the file extents. 1. 'dd if=/dev/zero of=/mnt/test/bunchofzeros.txt bs=M count=1' results in this: item 7 key (257 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 7 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 0 nr 1048576 ram 1048576 extent compression 0 (none) 2. Using vi to change one character and save as the same file, ':wq', I get: item 7 key (262 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 9 type 1 (regular) extent data disk byte 7401472 nr 1052672 extent data offset 0 nr 1052672 ram 1052672 extent compression 0 (none) So you can see it's a new file, new inode, and looks like length is changed too, maybe something to do with encoding. 3. Delete that file and create a new one with the same 'dd' command as in 1. item 7 key (266 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 11 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 0 nr 1048576 ram 1048576 extent compression 0 (none) [root@flap ~]# dd conv=notrunc if=/dev/urandom bs=1 seek=524290 count=1 of=/mnt/test/bunchofzeros.txt [root@flap ~]# hexdump -C /mnt/test/bunchofzeros.txt | more 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00080000 00 00 5d 00 00 00 00 00 00 00 00 00 00 00 00 00 |..].............| 00080010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00100000 item 7 key (266 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 11 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 0 nr 524288 ram 1048576 extent compression 0 (none) item 8 key (266 EXTENT_DATA 524288) itemoff 3356 itemsize 53 generation 14 type 1 (regular) extent data disk byte 5242880 nr 4096 extent data offset 0 nr 4096 ram 4096 extent compression 0 (none) item 9 key (266 EXTENT_DATA 528384) itemoff 3303 itemsize 53 generation 11 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 528384 nr 520192 ram 1048576 extent compression 0 (none) Clever. item7 and item9 actually reference the same original 1MiB data extent, but use offsets to point to the unchanged portion "creating" a 4KiB hole, and that hole is filed by item 8, a 4KiB extent containing the change. So it really did only COW one block.
      - Wowfunhappy 7y ago
        Sorry, I can’t tell, are you still talking about APFS?
        cmurf 7y ago
        No the example under Edit 2 is Btrfs. On the details I'd expect APFS to differ. But the gist is that how changes manifest on disk do also depend on the application being used. On any file system, an edit in vi means a whole new file is written out, so it's effectively COW at an application level. Whereas an edit with an application that does byte level overwrites, would get translated into COW on a COW file system like Btrfs, APFS or ZFS. And I'd expect this COW operation to operate at the file system's minimum block level size.
- dictum 7y ago
  In my opinion storing, not deleting, is the most important job of a file system. Increased efficiency and free space is a better proposition than replicating the average user's mental model of free space in the volume. (If my opinion holds true, then deduplicating is a problem when it comes to storage: a copy made to prevent bitrot/other damage to the original data doesn't work as such.)
  - voxadam 7y ago
    Storing vs deleting is one of the reasons that I'd love to see more research done on log based filesystems.
  - Wowfunhappy 7y ago
    > Increased efficiency and free space is a better proposition than replicating the average user's mental model of free space in the volume. I'm not sure if I agree. In an ideal world, it makes sense that performance should be paramount. In reality, I think it's often more important that computers do what we expect them to, so we can predict when they'll break and know how to resolve the problem when they do. This is all the more important in consumer software. "My hard drive is out of space" is a very basic computer problem that most users will run into at some point. The intuitive solution is to delete stuff. If that solution doesn't work, it could be a major problem.
- duskwuff 7y ago
  Or even worse: "You no longer have any free space on your hard disk because you made a change to a large file that had a deduplicated copy."
  - voxadam 7y ago
    Block level deduplication should limit the impact of this case.
    - duskwuff 7y ago
      Only if the change is localized to specific blocks. Besides, block-level dedup is expensive to maintain. Unless you're expecting to find a lot of duplicates -- which probably isn't the case on most end-user devices -- it's a net loss.
      - avani 7y ago
        Expect about 30%, most of it in smaller system files and libraries. Block-level dedup is great for pretty much anything where you have a consistent workflow.
        macintux 7y ago
        A consistent workflow sounds like something you’re going to find more regularly on a server than a mobile device or home computer.
ghaff 7y ago
Relative to what? I've been around long enough it's hard for me to see just about any disk storage as expensive even if it costs more than whatever is on sale at Newegg.
- FPGAhacker 7y ago
  Mac laptops aren’t upgradable without goofy adapters to convert nvme ssd m.2 to apple’s form factor, at which point your case won’t close up without flexing the aluminum or main board or both. For many MacBook Pro users, upgrading storage is buying a new laptop. That’s expensive in my opinion. OWC has made drives that fit Apple’s form factor (assuming your laptop is old enough) for a nice premium. Depending on which laptop you have, this can still be a science experiment.
bluedino 7y ago
They do RAM compression, I thought disk compression was soon to follow after Apple went all-flash storage
- twoodfin 7y ago
  Presumably not particularly valuable since most of what takes up consumer storage these days (pictures, videos, app assets) is already in some highly compressed format.
- yakkers 7y ago
  APFS does indeed do compression, the rationale being that compute is cheaper than I/O. Even if most files on a typical personal computer are already inherently compressed (JPEG photos, lossy audio, etc.), you're not losing any performance, and are gaining some extra space where possible.
jandrese 7y ago
It's cheaper than the memory upgrade you need to support dedup properly.
- leiroigh 7y ago
  Not really. ZFS dedup is just brain-dead stupid. Rabin-Karp chunking plus spatial locality / opportunistic dedup allow you to gracefully scale down: you keep a complete table of successfully deduped blocks, plus a small fraction of other blocks, like e.g. smallest 1% of hashes, first blocks in files and final blocks in files. If you have a 12 MB (100 blocks) stretch that matches, you have 63% (1-1/e) chance of discovering one of these blocks as a match, and then use spatial locality to dedup adjacent blocks (duplicate blocks tend to come in bunches).
  - jandrese 7y ago
    Then why does turning on dedup on ZFS suddenly make it chomp through 4GB of memory? Is there some tuneable you're supposed to be setting to make it less insane?

Yeah, but that's a customer cost, not an Apple cost. In any case, dedup just makes things more confusing to the customer— like, "you have X GB in files on your computer, but more than N-X GB free space because of magical deduplication." Or worse, "You deleted all those files, but didn't actually free up any space because they were dupes. So sad."

Wowfunhappy 7y ago
> In any case, dedup just makes things more confusing to the customer— like, "you have X GB in files on your computer, but more than N-X GB free space because of magical deduplication." Or worse, "You deleted all those files, but didn't actually free up any space because they were dupes. So sad." For what it's worth, this can already happen in APFS. While they don't have real deduplication, if you copy a file within Finder, that copy won't take up extra space.
- jjoonathan 7y ago
  How granular is the COW? If you touch a byte, does just a block get copied or the whole file?
  - mindajar 7y ago
    Just a block
  - cmurf 7y ago
    It depends on the application. Often applications do a kind of COW on their own, behind the scene they write out a new file with a randomized naming scheme, when that succeeds, do a rename/replace. So in this case you're getting a new file, as in, a file with a new inode. If your application is smarter, then it could just COW one block, whatever the minimum block size is for that file system. e.g. for Btrfs it's page size, so x86 that's 4KiB, ppc64le it's 64KiB. I'm not sure what it is for APFS, I'd guess 4KiB. It could actually be inefficient depending on the use case, e.g. a 1MiB file with a single byte change in the middle, if it were a single 1MiB extent, becomes three extents with COW: 0-500KiB - extent 1 500-504KiB - hole where stale data is 504KiB-1MiB - extent 3 4KiB - extent 2, with new data, a copy of 4095 bytes from the above hole, with 1 bytes of new data. Edit: 2nd paragraph suggests the application would COW if it's smarter; that's not correct. If it's smarter, it would request an overwrite of some portion of the file and the file system is what would COW (if it's a COW file system). Granularity depends on a combination of application and file system. Edit 2: More detailed answer on Btrfs. The whole file system on a zram device is about 260 lines using 'btrfs inspect-internal dump-tree' command. I'll excerpt just the file extents. 1. 'dd if=/dev/zero of=/mnt/test/bunchofzeros.txt bs=M count=1' results in this: item 7 key (257 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 7 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 0 nr 1048576 ram 1048576 extent compression 0 (none) 2. Using vi to change one character and save as the same file, ':wq', I get: item 7 key (262 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 9 type 1 (regular) extent data disk byte 7401472 nr 1052672 extent data offset 0 nr 1052672 ram 1052672 extent compression 0 (none) So you can see it's a new file, new inode, and looks like length is changed too, maybe something to do with encoding. 3. Delete that file and create a new one with the same 'dd' command as in 1. item 7 key (266 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 11 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 0 nr 1048576 ram 1048576 extent compression 0 (none) [root@flap ~]# dd conv=notrunc if=/dev/urandom bs=1 seek=524290 count=1 of=/mnt/test/bunchofzeros.txt [root@flap ~]# hexdump -C /mnt/test/bunchofzeros.txt | more 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00080000 00 00 5d 00 00 00 00 00 00 00 00 00 00 00 00 00 |..].............| 00080010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00100000 item 7 key (266 EXTENT_DATA 0) itemoff 3409 itemsize 53 generation 11 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 0 nr 524288 ram 1048576 extent compression 0 (none) item 8 key (266 EXTENT_DATA 524288) itemoff 3356 itemsize 53 generation 14 type 1 (regular) extent data disk byte 5242880 nr 4096 extent data offset 0 nr 4096 ram 4096 extent compression 0 (none) item 9 key (266 EXTENT_DATA 528384) itemoff 3303 itemsize 53 generation 11 type 1 (regular) extent data disk byte 5283840 nr 1048576 extent data offset 528384 nr 520192 ram 1048576 extent compression 0 (none) Clever. item7 and item9 actually reference the same original 1MiB data extent, but use offsets to point to the unchanged portion "creating" a 4KiB hole, and that hole is filed by item 8, a 4KiB extent containing the change. So it really did only COW one block.
    - Wowfunhappy 7y ago
      Sorry, I can’t tell, are you still talking about APFS?
      - cmurf 7y ago
        No the example under Edit 2 is Btrfs. On the details I'd expect APFS to differ. But the gist is that how changes manifest on disk do also depend on the application being used. On any file system, an edit in vi means a whole new file is written out, so it's effectively COW at an application level. Whereas an edit with an application that does byte level overwrites, would get translated into COW on a COW file system like Btrfs, APFS or ZFS. And I'd expect this COW operation to operate at the file system's minimum block level size.
dictum 7y ago
In my opinion storing, not deleting, is the most important job of a file system. Increased efficiency and free space is a better proposition than replicating the average user's mental model of free space in the volume. (If my opinion holds true, then deduplicating is a problem when it comes to storage: a copy made to prevent bitrot/other damage to the original data doesn't work as such.)
- voxadam 7y ago
  Storing vs deleting is one of the reasons that I'd love to see more research done on log based filesystems.
- Wowfunhappy 7y ago
  > Increased efficiency and free space is a better proposition than replicating the average user's mental model of free space in the volume. I'm not sure if I agree. In an ideal world, it makes sense that performance should be paramount. In reality, I think it's often more important that computers do what we expect them to, so we can predict when they'll break and know how to resolve the problem when they do. This is all the more important in consumer software. "My hard drive is out of space" is a very basic computer problem that most users will run into at some point. The intuitive solution is to delete stuff. If that solution doesn't work, it could be a major problem.
duskwuff 7y ago
Or even worse: "You no longer have any free space on your hard disk because you made a change to a large file that had a deduplicated copy."
- voxadam 7y ago
  Block level deduplication should limit the impact of this case.
  - duskwuff 7y ago
    Only if the change is localized to specific blocks. Besides, block-level dedup is expensive to maintain. Unless you're expecting to find a lot of duplicates -- which probably isn't the case on most end-user devices -- it's a net loss.
    - avani 7y ago
      Expect about 30%, most of it in smaller system files and libraries. Block-level dedup is great for pretty much anything where you have a consistent workflow.
      - macintux 7y ago
        A consistent workflow sounds like something you’re going to find more regularly on a server than a mobile device or home computer.

ghaff 7y ago

Relative to what? I've been around long enough it's hard for me to see just about any disk storage as expensive even if it costs more than whatever is on sale at Newegg.

FPGAhacker 7y ago
Mac laptops aren’t upgradable without goofy adapters to convert nvme ssd m.2 to apple’s form factor, at which point your case won’t close up without flexing the aluminum or main board or both. For many MacBook Pro users, upgrading storage is buying a new laptop. That’s expensive in my opinion. OWC has made drives that fit Apple’s form factor (assuming your laptop is old enough) for a nice premium. Depending on which laptop you have, this can still be a science experiment.

bluedino 7y ago

They do RAM compression, I thought disk compression was soon to follow after Apple went all-flash storage

twoodfin 7y ago
Presumably not particularly valuable since most of what takes up consumer storage these days (pictures, videos, app assets) is already in some highly compressed format.
yakkers 7y ago
APFS does indeed do compression, the rationale being that compute is cheaper than I/O. Even if most files on a typical personal computer are already inherently compressed (JPEG photos, lossy audio, etc.), you're not losing any performance, and are gaining some extra space where possible.

jandrese 7y ago

It's cheaper than the memory upgrade you need to support dedup properly.

leiroigh 7y ago
Not really. ZFS dedup is just brain-dead stupid. Rabin-Karp chunking plus spatial locality / opportunistic dedup allow you to gracefully scale down: you keep a complete table of successfully deduped blocks, plus a small fraction of other blocks, like e.g. smallest 1% of hashes, first blocks in files and final blocks in files. If you have a 12 MB (100 blocks) stretch that matches, you have 63% (1-1/e) chance of discovering one of these blocks as a match, and then use spatial locality to dedup adjacent blocks (duplicate blocks tend to come in bunches).
- jandrese 7y ago
  Then why does turning on dedup on ZFS suddenly make it chomp through 4GB of memory? Is there some tuneable you're supposed to be setting to make it less insane?