8 ms·

> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is th

> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safely manage these concerns. What could be the downsides of having this happen automatically within APFS?

pizzafeelsright 2y ago
data loss is the largest concern I still do not trust de-duplication software.
- dylan604 2y ago
  Even using sha-256 or greater type of hashing, I'd still have concerns about letting a system make deletion decisions without my involvement. I've even been part of de-dupe efforts, so maybe my hesitation is just because I wrote some of the code and I know I'm not perfect in my coding or even my algo decision trees. I know that any mistake I made would not be of malice but just ignorance or other stupid mistake. I've done the entire compare every file via hashing and then log each of the matches for humans to compare, but never has any of that ever been allowed to mv/rm/link -s anything. I feel my imposter syndrome in this regard is not a bad thing.
  - borland 2y ago
    Now you understand why this app costs more than 2x the price of alternatives such as diskDedupe. Any halfway-competent developer can write some code that does a SHA256 hash of all your files and uses the Apple filesystem API's to replace duplicates with shared-clones. I know swift, I could probably do it in an hour or two. Should you trust my bodgy quick script? Heck no. The author - John Siracusa - has been a professional programmer for decades and is an exceedingly meticulous kind of person. I've been listening to the ATP podcast where they've talked about it, and the app has undergone an absolute ton of testing. Look at the guardrails on the FAQ page https://hypercritical.co/hyperspace/ https://hypercritical.co/hyperspace/ for an example of some of the extra steps the app takes to keep things safe. Plus you can review all the proposed file changes before you touch anything. You're not paying for the functionality, but rather the care and safety that goes around it. Personally, I would trust this app over just about any other on the mac.
    - btilly 2y ago
      More than TeX or SQLite?
    - lukeholder 2y ago
      Best course of action is to not trust John, and just wait for a year of the app out the wild, until everyone else trusts John . I have enough hard drive space in the meantime to not rush into trusting John.
      - xp84 2y ago
        Having listened to John for 10 years, he’d be the first to encourage you to wait around to trust his app.
  - criddell 2y ago
    > I'd still have concerns about letting a system make deletion decisions without my involvement You are involved. You see the list of duplicates and can review them as carefully as you'd like before hitting the button to write the changes.
    - dylan604 2y ago
      Yeah, the lack of involvement was more in response to ZFS doing this not this app. I could have crossed the streams with other threads about ZFS if it's not directly in this thread
- axus 2y ago
  Question for the developer: what's your liability if user files are corrupted?
  - codazoda 2y ago
    Most EULA’s would disclaim liability for data loss and suggest users keep good backups. I haven’t read a EULA for a long time, but I think most of them do so.
    - borland 2y ago
      I can't find a specific EULA or disclaimer for the Hyperspace app, but given that the EULA's for major things like Microsoft Office basically say "we offer you no warranty or recourse no matter what this software does" I would hardly expect an indie app to offer anything like that
- czk 2y ago
  Dedupe seemed more interesting when storage was expensive, but nowadays it feels like the overhead you get from running dedupe, in most cases, is priced-in. At least with software like CommVault for backups, dedupe requires beefy hardware and low-latency SSDs for the database, If there is even a few extra milliseconds of latency or the server can’t handle requests fast enough, your backup throughput absolutely tanks. Depending on your data though you could see some ridiculous savings here that make it worth the trouble. I’ve heard many horror stories of dedupe related corruption or restoration woes though, especially after a ransomware attack.
- throwaway439080 2y ago
  Agreed, "I made a deduplication software in my garage! Do you want to try it?" is a terrifying pitch. I've been writing a similar thing to dedupe my photo collection and I'm so paranoid of pulling the trigger I just keep writing more tests.
albertzeyer 2y ago
> This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. I think that ZFS actually does this. https://www.truenas.com/docs/references/zfsdeduplication/ https://www.truenas.com/docs/references/zfsdeduplication/
- pmarreck 2y ago
  It's considered an "expensive" configuration that is only good for certain use-cases, though, due to its memory requirements.
  - abrookewood 2y ago
    Yes true, but that page also covers some recent improvements to de-duplication that might assist.
    - pmarreck 2y ago
      Really? I haven't looked at this ZFS feature in a few years so I will take a look EDIT: Is this referring to the "fast" dedup feature?
      - abrookewood 2y ago
        Yes. It doesn't solve everything, but it looks promising
p_ing 2y ago
Windows Server does this for NTFS and ReFS volumes. I used it quite a bit on ReFS w/ Hyper-V VMs and it worked wonders. Cut my storage usage down by ~45% with a majority of Windows Server VMs running a mix of 2016/2019 at the time.
- borland 2y ago
  Yep. At a previous job we had a file server that we published Windows build output to. There were about 1000 copies of the same pre-requisite .NET and VC++ runtimes (each build had one) and we only paid for the cost of storing it once. It was great. It is worth pointing out though, that on Windows Server this deduplication is a background process; When new duplicate files are created, they genuinely are duplicates and take up extra space, but once in a while the background process comes along and "reclaims" them, much like the Hyperspace app here does. Because of this (the background sweep process is expensive), it doesn't run all the time and you have to tell it which directories to scan. If you want "real" de-duplication, where a duplicate file will never get written in the first place, then you need something like ZFS
  - sterlind 2y ago
    hey, it's defrag all over again! (not really, since it's not fragmentation, but conceptually similar)
  - p_ing 2y ago
    Both ZFS and WinSvr offer "real" dedupe. One is on-write, which requires a significant amount of available memory, the other is on a defined schedule, which uses considerably less memory (300MB + 10MB/TB). ZFS is great if you believe you'll exceed some threshold of space while writing. I don't personally plan my volumes with that in mind but rather make sure I have some amount of excess free space. WinSvr allows you to disable dedupe if you want (don't know why you would) where as ZFS is a one-way street without exporting the data. Both have pros and cons. I can live with the WinSvr cons while ZFS cons (memory) would be outside of my budget, or would have been at the particular time with the particular system.
taneliv 2y ago
On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot disk activity, as the file system would need to duplicate the file again.
- gmueckl 2y ago
  Files are always represented as lists of blocks or block spans within a file system. Individual blocks could in theory be partially shared between files at the complexity cost of a reference counter for each block. So changing a single byte in a copy on write file could take the same time regardless of file size because only the affected bock would have to be duplicated. I don't know at all how MacOS implements this copynon write scheme, though.
  - MBCook 2y ago
    APFS is a copy on write filesystem if you use the right APIs, so it does what you describe but only for entire files. I believe as soon as you change a single bite you get a complete copy that’s your own. And that’s how this program works. It finds perfect duplicates and then effectively deletes and replaces them with a copy of the existing file so in the background there’s only one copy of the bits on the disk.
    - mintplant 2y ago
      I suppose this means that you could find yourself unexpectedly out of disk space in unintuitive ways, if you're only trying to change one byte in a cloned file but there isn't enough space to copy its entire contents?
      - pansa777 2y ago
        It doesn't work like you think. If you change one byte of duplicated file - the only "byte" will be changed on disk (a "byte", because, technically is not a byte, but a block). As far as I understand, it works like a reflink feature in the modern linux FSs. If so, thats really cool, and thats also a bit better than the zfs's snapshots. Iam newbie on macos, but it looks amazing
asdfman123 2y ago
If Apple is anything like where I work, there's probably a three-year-old bug ticket in their system about it and no real mandate from upper management to allocate resources for it.
- deleted 2y ago
  [deleted]
ted_dunning 2y ago
This is commonly done with compression on block storage devices. That fails, of course, if the file system is encrypting the blocks it sends down to the device. Doing deduplication at this level is nice because you can dedupe across file systems. If you have, say, a thousand systems that all have the same OS files you can save vats of storage. Many times, the only differences will be system specific configurations like host keys and hostnames. No single filesystem could recognize this commonality. This fails when the deduplication causes you to have fewer replicas of files with intense usage. To take the previous example, if you boot all thousand machines at the same time, you will have a prodigious I/O load on the kernel images.
UltraSane 2y ago
NTFS supports deduplication but it is only available on Server versions which is very annoying.
nielsbot 2y ago
Disk Utility.app manages to keep the OS running while make the disk exclusive-access.. I wonder how it does that.
- nsonha 2y ago
  It doesn't? Freeze when commiting operations.
  - nielsbot 2y ago
    I was musing about this comment: > There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. Maybe not solving the same problem.
mentalgear 2y ago
what's the source of that quote, does it mean it's not safe to use hyperspace ?
a-dub 2y ago
a content addressed block store with pointers and skiplists for file continuity would be kinda neat.
Sesse__ 2y ago
This is the standard API for deduplication on Linux (used for btrfs and XFS); you ask the OS nicely to deduplicate a given set of ranges, and it responds by locking the ranges, verifying that they are indeed identical and only then deduplicates for you (you get a field back saying how many bytes were deduplicated from each range). So there's no way a userspace program can mess up your files.
- kccqzy 2y ago
  Yup. This is the ioctl BTRFS_IOC_FILE_EXTENT_SAME.
alwillis 2y ago
Probably because APFS runs on everything from the Apple Watch to the Mac Pro and everything in between. You probably don’t want your phone or watch de-duping stuff. There are tons of knobs you can tweak in macOS but Apple has always been pretty conservative when it comes to what should be default behavior for the vast majority of their users. Certainly when you duplicate a file using the Finder or use cp -c at the command line, the Copy-on-Write functionality is being used; most users don’t need to know that.

pizzafeelsright 2y ago

data loss is the largest concern I still do not trust de-duplication software.

dylan604 2y ago
Even using sha-256 or greater type of hashing, I'd still have concerns about letting a system make deletion decisions without my involvement. I've even been part of de-dupe efforts, so maybe my hesitation is just because I wrote some of the code and I know I'm not perfect in my coding or even my algo decision trees. I know that any mistake I made would not be of malice but just ignorance or other stupid mistake. I've done the entire compare every file via hashing and then log each of the matches for humans to compare, but never has any of that ever been allowed to mv/rm/link -s anything. I feel my imposter syndrome in this regard is not a bad thing.
- borland 2y ago
  Now you understand why this app costs more than 2x the price of alternatives such as diskDedupe. Any halfway-competent developer can write some code that does a SHA256 hash of all your files and uses the Apple filesystem API's to replace duplicates with shared-clones. I know swift, I could probably do it in an hour or two. Should you trust my bodgy quick script? Heck no. The author - John Siracusa - has been a professional programmer for decades and is an exceedingly meticulous kind of person. I've been listening to the ATP podcast where they've talked about it, and the app has undergone an absolute ton of testing. Look at the guardrails on the FAQ page https://hypercritical.co/hyperspace/ https://hypercritical.co/hyperspace/ for an example of some of the extra steps the app takes to keep things safe. Plus you can review all the proposed file changes before you touch anything. You're not paying for the functionality, but rather the care and safety that goes around it. Personally, I would trust this app over just about any other on the mac.
  - btilly 2y ago
    More than TeX or SQLite?
  - lukeholder 2y ago
    Best course of action is to not trust John, and just wait for a year of the app out the wild, until everyone else trusts John . I have enough hard drive space in the meantime to not rush into trusting John.
    - xp84 2y ago
      Having listened to John for 10 years, he’d be the first to encourage you to wait around to trust his app.
- criddell 2y ago
  > I'd still have concerns about letting a system make deletion decisions without my involvement You are involved. You see the list of duplicates and can review them as carefully as you'd like before hitting the button to write the changes.
  - dylan604 2y ago
    Yeah, the lack of involvement was more in response to ZFS doing this not this app. I could have crossed the streams with other threads about ZFS if it's not directly in this thread
axus 2y ago
Question for the developer: what's your liability if user files are corrupted?
- codazoda 2y ago
  Most EULA’s would disclaim liability for data loss and suggest users keep good backups. I haven’t read a EULA for a long time, but I think most of them do so.
  - borland 2y ago
    I can't find a specific EULA or disclaimer for the Hyperspace app, but given that the EULA's for major things like Microsoft Office basically say "we offer you no warranty or recourse no matter what this software does" I would hardly expect an indie app to offer anything like that
czk 2y ago
Dedupe seemed more interesting when storage was expensive, but nowadays it feels like the overhead you get from running dedupe, in most cases, is priced-in. At least with software like CommVault for backups, dedupe requires beefy hardware and low-latency SSDs for the database, If there is even a few extra milliseconds of latency or the server can’t handle requests fast enough, your backup throughput absolutely tanks. Depending on your data though you could see some ridiculous savings here that make it worth the trouble. I’ve heard many horror stories of dedupe related corruption or restoration woes though, especially after a ransomware attack.
throwaway439080 2y ago
Agreed, "I made a deduplication software in my garage! Do you want to try it?" is a terrifying pitch. I've been writing a similar thing to dedupe my photo collection and I'm so paranoid of pulling the trigger I just keep writing more tests.

albertzeyer 2y ago

> This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. I think that ZFS actually does this. https://www.truenas.com/docs/references/zfsdeduplication/ https://www.truenas.com/docs/references/zfsdeduplication/

pmarreck 2y ago
It's considered an "expensive" configuration that is only good for certain use-cases, though, due to its memory requirements.
- abrookewood 2y ago
  Yes true, but that page also covers some recent improvements to de-duplication that might assist.
  - pmarreck 2y ago
    Really? I haven't looked at this ZFS feature in a few years so I will take a look EDIT: Is this referring to the "fast" dedup feature?
    - abrookewood 2y ago
      Yes. It doesn't solve everything, but it looks promising

p_ing 2y ago

Windows Server does this for NTFS and ReFS volumes. I used it quite a bit on ReFS w/ Hyper-V VMs and it worked wonders. Cut my storage usage down by ~45% with a majority of Windows Server VMs running a mix of 2016/2019 at the time.

borland 2y ago
Yep. At a previous job we had a file server that we published Windows build output to. There were about 1000 copies of the same pre-requisite .NET and VC++ runtimes (each build had one) and we only paid for the cost of storing it once. It was great. It is worth pointing out though, that on Windows Server this deduplication is a background process; When new duplicate files are created, they genuinely are duplicates and take up extra space, but once in a while the background process comes along and "reclaims" them, much like the Hyperspace app here does. Because of this (the background sweep process is expensive), it doesn't run all the time and you have to tell it which directories to scan. If you want "real" de-duplication, where a duplicate file will never get written in the first place, then you need something like ZFS
- sterlind 2y ago
  hey, it's defrag all over again! (not really, since it's not fragmentation, but conceptually similar)
- p_ing 2y ago
  Both ZFS and WinSvr offer "real" dedupe. One is on-write, which requires a significant amount of available memory, the other is on a defined schedule, which uses considerably less memory (300MB + 10MB/TB). ZFS is great if you believe you'll exceed some threshold of space while writing. I don't personally plan my volumes with that in mind but rather make sure I have some amount of excess free space. WinSvr allows you to disable dedupe if you want (don't know why you would) where as ZFS is a one-way street without exporting the data. Both have pros and cons. I can live with the WinSvr cons while ZFS cons (memory) would be outside of my budget, or would have been at the particular time with the particular system.

taneliv 2y ago

On ZFS it consumes a lot of RAM. In part I think this is because ZFS does it on the block level, and has to keep track of a lot of blocks to compare against when a new one is written out. It might be easier on resources if implemented on the file level. Not sure if the implementation would be simpler or more complex. It might also be a little unintuitive that modifying one byte of a large file would result in a lot disk activity, as the file system would need to duplicate the file again.

gmueckl 2y ago
Files are always represented as lists of blocks or block spans within a file system. Individual blocks could in theory be partially shared between files at the complexity cost of a reference counter for each block. So changing a single byte in a copy on write file could take the same time regardless of file size because only the affected bock would have to be duplicated. I don't know at all how MacOS implements this copynon write scheme, though.
- MBCook 2y ago
  APFS is a copy on write filesystem if you use the right APIs, so it does what you describe but only for entire files. I believe as soon as you change a single bite you get a complete copy that’s your own. And that’s how this program works. It finds perfect duplicates and then effectively deletes and replaces them with a copy of the existing file so in the background there’s only one copy of the bits on the disk.
  - mintplant 2y ago
    I suppose this means that you could find yourself unexpectedly out of disk space in unintuitive ways, if you're only trying to change one byte in a cloned file but there isn't enough space to copy its entire contents?
    - pansa777 2y ago
      It doesn't work like you think. If you change one byte of duplicated file - the only "byte" will be changed on disk (a "byte", because, technically is not a byte, but a block). As far as I understand, it works like a reflink feature in the modern linux FSs. If so, thats really cool, and thats also a bit better than the zfs's snapshots. Iam newbie on macos, but it looks amazing

asdfman123 2y ago

If Apple is anything like where I work, there's probably a three-year-old bug ticket in their system about it and no real mandate from upper management to allocate resources for it.

deleted 2y ago
[deleted]

ted_dunning 2y ago

This is commonly done with compression on block storage devices. That fails, of course, if the file system is encrypting the blocks it sends down to the device. Doing deduplication at this level is nice because you can dedupe across file systems. If you have, say, a thousand systems that all have the same OS files you can save vats of storage. Many times, the only differences will be system specific configurations like host keys and hostnames. No single filesystem could recognize this commonality. This fails when the deduplication causes you to have fewer replicas of files with intense usage. To take the previous example, if you boot all thousand machines at the same time, you will have a prodigious I/O load on the kernel images.

UltraSane 2y ago

NTFS supports deduplication but it is only available on Server versions which is very annoying.

nielsbot 2y ago

Disk Utility.app manages to keep the OS running while make the disk exclusive-access.. I wonder how it does that.

nsonha 2y ago
It doesn't? Freeze when commiting operations.
- nielsbot 2y ago
  I was musing about this comment: > There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. Maybe not solving the same problem.

mentalgear 2y ago

what's the source of that quote, does it mean it's not safe to use hyperspace ?

a-dub 2y ago

a content addressed block store with pointers and skiplists for file continuity would be kinda neat.

Sesse__ 2y ago

This is the standard API for deduplication on Linux (used for btrfs and XFS); you ask the OS nicely to deduplicate a given set of ranges, and it responds by locking the ranges, verifying that they are indeed identical and only then deduplicates for you (you get a field back saying how many bytes were deduplicated from each range). So there's no way a userspace program can mess up your files.

kccqzy 2y ago
Yup. This is the ioctl BTRFS_IOC_FILE_EXTENT_SAME.

alwillis 2y ago

Probably because APFS runs on everything from the Apple Watch to the Mac Pro and everything in between. You probably don’t want your phone or watch de-duping stuff. There are tons of knobs you can tweak in macOS but Apple has always been pretty conservative when it comes to what should be default behavior for the vast majority of their users. Certainly when you duplicate a file using the Finder or use cp -c at the command line, the Copy-on-Write functionality is being used; most users don’t need to know that.