38 ms·
Hyperspace
- gnomesteel 2y agoI don’t need this,storage is cheap, but I’m glad it exists.
- ttoinou 2y agoStorage isnt cheap on macs though. One has to pay 2k USD to get 8 TB SSD
- NoToP 2y agoThe fact that copying doesn't copy seems dangerous. Like what if I wanted to copy for the purpose of modifying the file while retaining the original. A trivial example of this might be I have a meme template and I want to write text in it while still keeping a blank copy of the template. There's a place for alias file pointers, but lying to the user and pretending like an alias is a copy is bound to lead to unintended and confusing results
- deleted 2y ago[deleted]
- timabdulla 2y agoIt's copy on write.
- herrkanin 2y agoIt’s not a symbolic link - it copies on modification. No need to worry!
- hutattedonmyarm 2y agoIt‘s Copy On Write. When you modify either one it does get turned into an actual copy
- IsTom 2y agoCopy-on-write means that it performs copy only when you make the first change (and only copies part that changes, rest is used from the original file), until then copying is free.
- mlhpdx 2y agoIs it file level or block level copy? The latter, I hope. Update: whoops, missed it in your comment. Block (changed bytes) level.
- pca006132 2y agoCoW is not aliasing. It will perform the actual copying when you modify the file content.
- parwej 2y ago[flagged]
- petercooper 2y agoI love the model of it being free to scan and see if you'd get any benefit, then paying for the actual results. I, too, am a packrat, ran it, and got 7GB to reclaim. Not quite worth the squeeze for me, but I appreciate it existing!
- sejje 2y agoI also really like this pricing model. I wish it were more obvious how to do it with other software. Often there's a learning curve in the way before you can see the value.
- MBCook 2y agoHe’s talked about it on the podcast he was on. So many users would buy this, run it once, then save a few gigs and be done. So a subscription didn’t make a ton of sense. After all how many perfect duplicate files do you probably create a month accidentally? There’s a subscription or buy forever option for people who think that would actually be quite useful to them. But for a ton of people a one time IAP that gives them a limited amount of time to use the program really does make a lot of sense. And you can always rerun it for free to see if you have enough stuff worth paying for again.
- deleted 2y ago[deleted]
- jedbrooke 2y agoit’s very refreshing compared to those “free trials” you have to remember to cancel (pro tip: use virtual credit cards which you can lock for those so if you forget to cancel the charges are blocked) however has anyone been able to find out from the website how much the license actually costs?
- xp84 2y agoDoesn’t the Mac App Store listing list the IAP SKUs like it does on iOS?
- Analemma_ 2y agoIn earlier episodes of ATP when they were musing on possible names, one listener suggested the frankly amazing "Dupe Nukem". I get that this is a potential IP problem, which is why John didn't use it, but surely Duke Nukem is not a zealously-defended brand in 2025. I think interest in that particular name has been stone dead for a while now.
- InsideOutSanta 2y agoIt's a genius name, but Gearbox owns Duke Nukem. They're not exactly dormant. Duke Nukem as a franchise made over a billion in revenue. In 2023, Zen released a licensed Duke Nukem pinball table, so there is at least some ongoing interest in the franchise. I probably wouldn't have risked it, either.
- mzajc 2y agoReminds me of Avira's Luke Filewalker - I wonder if they needed any special agreement with Lucasfilm/Disney. I couldn't find any info on it, and their website doesn't mention Star Wars at all.
- siranachronist 2y agohttps://github.com/pkolaczk/fclones https://github.com/pkolaczk/fclones can do the same thing, and it's perfectly free and open source. terminal based though
- PenguinRevolver 2y agobrew install fclones Thanks for the recommendation! Just installed it via homebrew.
- deleted 2y ago[deleted]
- CharlesW 2y ago[I was wrong, see below.—cw] It doesn't do the same thing. An APFS clone/copy-on-write clone is not the same as a hard or soft link. https://eclecticlight.co/2019/01/05/aliases-hard-links-symlinks-and-copies-in-mojaves-apfs/ https://eclecticlight.co/2019/01/05/aliases-hard-links-symli...
- PenguinRevolver 2y agoYour source points out that: < You can also create [APFS (copy on write) clones] in Terminal using the command `cp -c oldfilename newfilename` where the c option requires cloning rather than a regular copy. `fclones dedupe` uses the same command[1]: if cfg!(target_os = "macos") { result.push(format!("cp -c {target} {link}")); [1] https://github.com/pkolaczk/fclones/blob/555cde08fde4e700b2559bf93044bb17755321cf/fclones/src/dedupe.rs#L406 https://github.com/pkolaczk/fclones/blob/555cde08fde4e700b25...
- CharlesW 2y agoI stand corrected, thank you!
- rahimnathwani 2y agoHyperspace said I can save 10GB. But then I ran this command and saved over 20GB: brew install fclones cd ~ fclones group . | fclones dedupe I've used fclones before in the default mode (create hard links) but this is the first time I've run it at the top level of my home folder, in dedupe mode (i.e. using APFS clones). Fingers crossed it didn't wreck anything.
- herrkanin 2y agoAs a web dev, it’s been fun listening to Accidental Tech Podcast where Siracusa has been talking (or ranting) about the ins and outs of developing modern mac apps in Swift and SwiftUI.
- Analemma_ 2y agoThe part where he said making a large table in HTML and rendering it with a web view was orders of magnitude faster than using the SwiftUI native platform controls made me bash my head against my desk a couple times. What are we doing here, Apple.
- mohsen1 2y agoHacker News loves to hate Electron apps. In my experience ChatGPT on Mac (which I assume is fully native) is nearly impossible to use because I have a lot of large chats in my history but the website works much better and faster. ChatGPT website packed in Electron would've been much better. In fact, I am using a Chrome "PWA App" for ChatGPT now instead of the native app.
- RandomDistort 2y agoSomeone more experienced that me could probably comment on this more, but theoretically is it possible for Electron production builds to become more efficient by having a much longer build process and stripping out all the unnecessary parts of Chromium?
- wat10000 2y agoIt's possible to make bad apps with anything. The difference is that, as far as I can tell, it's not possible to make good apps with Electron.
- avtar 2y ago> In my experience ChatGPT on Mac (which I assume is fully native) If we are to believe ChatGPT itself: "The ChatGPT macOS desktop app is built using Electron, which means it is primarily written in JavaScript, HTML, and CSS"
- divan 2y agoWhat are the potential risks or problems of such conversion of duplicates into APFS clones?
- captn3m0 2y agoThe linked docs cover this in detail.
- deleted 2y ago[deleted]
- pca006132 2y agoIs this the dedup function provided by other FS?
- coder543 2y agoI think the term to search for is reflink. Btrfs is one example: https://btrfs.readthedocs.io/en/latest/Reflink.html https://btrfs.readthedocs.io/en/latest/Reflink.html Like with Hyperspace, you would need to use a tool that can identify which files are duplicates, and then convert them into reflinks.
- pca006132 2y agoI thought reflink is provided by the underlying FS, and Hyperspace is a dedup tool that finds the duplicates.
- coder543 2y agoYes. Hyperspace is finding the identical files and then replacing all but one copy with a reflink copy using the filesystem's reflink functionality. When you asked about the filesystem, I assumed you were asking about which filesystem feature was being used, since hyperspace itself is not provided by the filesystem. Someone else mentioned[0] fclones, which can do this task of finding and replacing duplicates with reflinks on more than just macOS, if you were looking for a userspace tool. [0]: https://news.ycombinator.com/item?id=43173713 https://news.ycombinator.com/item?id=43173713
- MBCook 2y agoHyperspace uses built in APFS features, it just applies them to existing files. You only get CoW on APFS if you copy a file with certain APIs or tools. If you have a program that does it manually, you copied a duplicate to somewhere on your desk from some other source, or your files already existed on the file system when you converted to APFS because you’ve been carrying them for a long time then you’d have duplicates. APFS doesn’t look for duplicates at any point. It just keeps track of those that it knows are duplicates because of copy operations.
- exitb 2y agoWhat are examples of files that make up the "dozens of gigabytes" of duplicated data?
- xnx 2y agoThere are some CUDA files that every local AI app install that take multiple GB.
- wruza 2y agoAlso models that various AI libraries and plugins love to autodownload into custom locations. Python folks definitely need to learn caching, symlinks, asking a user where to store data, or at least logging where they actually do it.
- butlike 2y agoaudio files; renders, etc.
- password4321 2y agoiMovie used to copy video files etc. into its "library".
- zerd 2y ago.terraform, rust target directory, node_modules.
- egorfine 2y ago[flagged]
- apercu 2y ago[flagged]
- Henchman21 2y agoSteve died. Edit: Let’s be real here, Tim Cook is keeping the lights on. He isn’t a product guy. They lack leadership and vision at present and are committed to a foolish release cycle based on calendar year not quality of product. These wouldn’t have come to pass had Steve lived till today. Yes its opinion but I doubt its an unpopular one. So I’ll say it again succinctly, to answer “what happened?” Steve died.
- egorfine 2y agoSteve was the one who delivered MobileMe, arguably the most broken service by Apple. So not that he's a saint.
- egorfine 2y agoA ~20 y.o. account with perhaps hundreds of devices in history across different continents and countries along with family sharing. Every time I need to purchase something via Apple, it becomes a quest. Enter password, validate card, welcome to endless login loop. Reboot. Click purchase, enter password, confirm OTP on another device, then nothing happens, purchase button is active, clicks ignored. Reboot. Click "Get", program begins downloading, wait 30s, app cannot be download, go to Settings to verify account. Sure. Account is perfectly fine in Settings. Reboot. Click "Get". Finally program installed. Click in-app purchase. Enter password again. Choose Apple Pay. Proceed with purchase. You need to verify your account. Account is fine in Settings. Reboot. Click purchase. Cannot be completed at this time. Wait couple of hours, try again. Purchase successful. All. The. Time. For years. On almost all of the devices which I upgrade annually.
- 2y ago
- jarbus 2y agoIn my experience, Macs use up a ridiculous amount of "System" storage for no reason that users can't delete. I've grown tired of family members asking me to help them free up storage that I can't even find. That's the major issue from what I've seen; unless this app prevents apple deliberately eating up 50%+ of the storage space of a machine, this doesn't do much for the people I know.
- ezfe 2y agoThere's no magic around it, macOS just doesn't do a good job explaining it using the built in tools. Just use Daisy Disk or something. It's all there and can be examined.
- p_ing 2y agoThese are often Time Machine snapshots. Nuking those can free up quite a bit of space. sudo tmutil listlocalsnapshots / sudo tmutil deletelocalsnapshots <date_value_of_snapshot>
- Jaxan 2y agoEven without time machine there are loads of storage spent on “system”. Especially now with the apple intelligence (even when turned off).
- p_ing 2y agoApple "Intelligence" gets its own category in 15.3.1.
- ezfe 2y agoYou don't need Terminal even, you can view and delete them in Disk Utility.
- sir_eliah 2y agoThere's a cross-platform open-source version of this program: https://github.com/qarmin/czkawka https://github.com/qarmin/czkawka
- deleted 2y ago[deleted]
- spiderfarmer 2y agoThat’s not remotely comparable.
- nulld3v 2y agoI don't think czkawa supports deduplication via reflink so it's not exactly the same thing. fclones as linked by another user is more similar: https://news.ycombinator.com/item?id=43173713 https://news.ycombinator.com/item?id=43173713
- bhouston 2y agoI gave it a try on my massive folder of NodeJS projects but it only found 1GB of savings on a 8.1GB folder. I then tried again including my user home folder (731K files, 127K folders, 2755 eligible files) to hopefully catch more savings and I only ended up at 1.3GB of savings (300MB more than just what was in the NodeJS folders.) I tried to scan System and Library but it refused to do so because of permission issues. I think the fact that I use pnpm for my package manager has made my disk space usage already pretty near optimal. Oh well. Neat idea. But the current price is too high to justify this. Also I would want it as a background process that runs once a month or something.
- lou1306 2y ago> it only found 1GB of savings on a 8.1GB folder. You "only" found that 12% of the space you are using is wasted? Am I reading this right?
- warkdarrior 2y agoThe relevant number (missing from above) is the total amount of space on that storage device. If it saves 1GB on a 8TB drive, it's not a big win.
- jy14898 2y agoIf it saved 8.1GB, by your measure it'd also not be a big win?
- horsawlarway 2y agoThis is basically only a win on macOS, and only because Apple charges through the nose for disk space. Ex - On my non-apple machines, 8GB is trivial. I load them up with the astoundingly cheap NVMe drives in the multiple terabyte range (2TB for ~$100, 4TB for ~$250) and I have a cheap NAS. So that "big win" is roughly 40 cents of hardware costs on the direct laptop hardware. Hardly worth the time and effort involved, even if the risk is zero (and I don't trust it to be zero). If it's just "storage" and I don't need it fast (the perfect case for this type of optimization) I throw it on my NAS where it's cheaper still... Ex - it's not 40 cents saved, it's ~10. --- At least for me, 8GB is no longer much of a win. It's a rounding error on the last LLM model I downloaded. And I'd suggest that basically anyone who has the ability to not buy extortionately priced drives soldered onto a mainboard is not really winning much here either. I picked up a quarter off the ground on my walk last night. That's a bigger win.
- gbvy 2y ago[flagged]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- jbverschoor 2y agoDoes it preserve all metadata, extended attributes, and alternate streams/named forks?
- atommclain 2y agoHe spoke to this on No Longer ery Good, episode 626 of The Accidental Tech Podcast. Time stamp ~1:32:30 It tries, but there are some things it can't perfectly preserve like the last access time. Instances where it can't duplicate certain types of extended attributes or ownership permissions it will not perform the operation. https://podcasts.apple.com/podcast/id617416468?i=1000691959947&r=5550&t=5550 https://podcasts.apple.com/podcast/id617416468?i=10006919599...
- jbverschoor 2y agoWell, the FAQ also states that people should notify if you're missing attributes, so it really sounds like it's a predefined list instead of just enumeration through everything. No word about alternate data streams. I'll pass for now.. Although it's nice to see how much duplicates you have
- criddell 2y agoThe FAQ talks about this a little: Q: Does Hyperspace preserve file metadata during reclamation? A: When Hyperspace replaces a file with a space-saving clone, it attempts to preserve all metadata associated with that file. This includes the creation date, modification date, permissions, ownership, Finder labels, Finder comments, whether or not the file name extension is visible, and even resource forks. If the attempt to preserve any of these piece of metadata fails, then the file is not replaced. If you find some piece of file metadata that is not preserved, please let us know. Q: How does Hyperspace handle resource forks? A: Hyperspace considers the contents of a file’s resource fork to be part of the file’s data. Two files are considered identical only if their data and resource forks are identical to each other. When a file is replaced by a space-saving clone during reclamation, its resource fork is preserved.
- bob1029 2y ago> There is no way for Hyperspace to cooperate with all other applications and macOS itself to coordinate a “safe” time for those files to be replaced, nor is there a way for Hyperspace for forcibly take exclusive control of those files. This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background. Presumably, it is at a level of abstraction where it could safely manage these concerns. What could be the downsides of having this happen automatically within APFS?
- pizzafeelsright 2y agodata loss is the largest concern I still do not trust de-duplication software.
- dylan604 2y agoEven using sha-256 or greater type of hashing, I'd still have concerns about letting a system make deletion decisions without my involvement. I've even been part of de-dupe efforts, so maybe my hesitation is just because I wrote some of the code and I know I'm not perfect in my coding or even my algo decision trees. I know that any mistake I made would not be of malice but just ignorance or other stupid mistake. I've done the entire compare every file via hashing and then log each of the matches for humans to compare, but never has any of that ever been allowed to mv/rm/link -s anything. I feel my imposter syndrome in this regard is not a bad thing.
- borland 2y agoNow you understand why this app costs more than 2x the price of alternatives such as diskDedupe. Any halfway-competent developer can write some code that does a SHA256 hash of all your files and uses the Apple filesystem API's to replace duplicates with shared-clones. I know swift, I could probably do it in an hour or two. Should you trust my bodgy quick script? Heck no. The author - John Siracusa - has been a professional programmer for decades and is an exceedingly meticulous kind of person. I've been listening to the ATP podcast where they've talked about it, and the app has undergone an absolute ton of testing. Look at the guardrails on the FAQ page https://hypercritical.co/hyperspace/ https://hypercritical.co/hyperspace/ for an example of some of the extra steps the app takes to keep things safe. Plus you can review all the proposed file changes before you touch anything. You're not paying for the functionality, but rather the care and safety that goes around it. Personally, I would trust this app over just about any other on the mac.
- 999900000999 2y agoA 20$ 1 year licence for something that probably has a FOSS equivalent on Linux... However, considering Apple will never ever ever allow user replaceable storage on a laptop, this might be worth it.
- ezfe 2y agoThe cost is because of the fact people won't use it regularly. The developer is offering life time unlocks, lower cost levels for shorter timeframes etc.
- p_ing 2y agoThe developer does need to make up for the $100 yearly privilege of publishing the app to the App Store.
- jeroenhd 2y agoI have yet to see a GUI variant of deduplication software for Linux. There are plenty of command line tools, which probably can be ported to macOS, but there's no user friendly tool to just click through as far as I know. There's value in convenience. I wouldn't pay for a yearly license (that price seems more than fair for a "version lifetime" price to me?) but seeing as this tool will probably need constant maintenance as Apple tweaks and changes APFS over time, combined with the mandatory Apple taxes for publishing software like this, it's not too awful.
- 999900000999 2y ago50$ for a lifetime license. Which really means up until the dev gets bored, which can be as short as 18 months. I wouldn't mind something like this versioned to OS. 20$ for the current OS, and ten dollars for every significant update.
- artimaeis 2y agoThe Mac App Store (and all of Apple's App Stores) doesn't enable this sort of licensing. It's exactly the sort of thing that drives a lot of developers to independent distribution. That's why we see so many more subscription-based apps these days, application development is an ongoing process with ongoing costs, so it needs to have ongoing income. But the traditional buy-it-once app pricing doesn't enable that long-term development and support. The app store supports subscriptions though, so now we get way more subscription-based apps. I really think Siracusa came up with a clever pricing scheme here, given his want to use the app store for distribution.
- ZedZark 2y agoI did this with two scripts - one that produces and cached sha1 sums of files, and another that consumes the output of the first (or any of the *sum progs) and produces stats about duplicate files, with options to delete or hard-link them.
- strunz 2y agoI wonder how any comments about hard links will be in these comments by people misunderstanding what this app does.
- theamk 2y agoif file is not going to be modified (in the low-level sense - open("w") on the filename; as opposed to rename-and-create-new), then reflinks (what this app does) and hardlinks act somewhat identically. For example if you have multiple node_modules, or app installs, or source photos/videos (ones you don't edit), or music archives, then hardlinks work just fine.
- deleted 2y ago[deleted]
- anti-soyboy 2y ago[flagged]
- bflesch 2y ago[flagged]
- frizlab 2y agoIt is not a hard link. A clone is an independent file which is backed by the same storage. So far mostly the same as a hard link you’ll say. However if you modify a clone, it will be “uncloned” and will be modified independently of its clones.
- jeroenhd 2y agoDoes APFS have the extent-level deduplication Linux filesystems have or is it only file-level deduplication? While it's not very user-friendly software (command line, after all), duperemove has proven quite useful for my filesystems.
- sharkjacobs 2y agoAnd I believe that only the modified blocks are “uncloned”, the rest of the data is still shared between the two files
- oneeyedpigeon 2y agoAside from the other point, macOS does, of course, have hard link functionality.
- albertzeyer 2y agoI wrote a similar (but simpler) script which would replace a file by a hardlink if it has the same content. My main motivation was for the packages of Python virtual envs, where I often have similar packages installed, and even if versions are different, many files would still match. Some of the packages are quite huge, e.g. Numpy, PyTorch, TensorFlow, etc. I got quite some disk space savings from this. https://github.com/albertz/system-tools/blob/master/bin/merge-identical-files.py https://github.com/albertz/system-tools/blob/master/bin/merg...
- andrewla 2y agoThis does not use hard links or symlinks; this uses a feature of the filesystem that allows the creation of copy-on-write clones. [1] [1] https://en.wikipedia.org/wiki/Apple_File_System#Clones https://en.wikipedia.org/wiki/Apple_File_System#Clones
- notatallshaw 2y agouv does this out of the box, I think other tools (poetry, hatch, pdm, etc.) do as well but I have less experience with the details.
- diggan 2y ago> Like all my apps, Hyperspace is a bit difficult to explain. I’ve attempted to do so, at length, in the Hyperspace documentation. I hope it makes enough sense to enough people that it will be a useful addition to the Mac ecosystem. Am I missing something, or isn't it a "file de-duplicator" with a nice UI/UX? Sounds pretty simple to describe, and tells you why it's useful with just two words.
- protonbob 2y agoNo because it isn't getting rid of the duplicate, it's using a feature of APFS that allows for duplicates to exist separately but share the same internal data.
- yayoohooyahoo 2y agoIs it not the same as a hard link (which I believe are supported on Mac too)?
- andrewla 2y agoMy understanding is that it is a copy-on-write clone, not a hard link. [1] > Q: Are clone files the same thing as symbolic links or hard links? > A: No. Symbolic links ("symlinks") and hard links are ways to make two entries in the file system that share the same data. This might sound like the same thing as the space-saving clones used by Hyperspace, but there’s one important difference. With symlinks and hard links, a change to one of the files affects all the files. > The space-saving clones made by Hyperspace are different. Changes to one clone file do not affect other files. Cloned files should look and behave exactly the same as they did before they were converted into clones. [1] https://hypercritical.co/hyperspace/ https://hypercritical.co/hyperspace/
- dylan604 2y agoWhat kind of changes could you make to one clone that would still qualify it as a clone? If there are changes, it's no longer the same file. Even after reading the How It Works[0] link, I'm not groking how it works. Is it making some sort of delta/diff that is applied to the original file? That's not possible for every file format like large media files. I could see that being interesting for text based files, but that gets complicated for complex files. [0] https://hypercritical.co/hyperspace/#how-it-works https://hypercritical.co/hyperspace/#how-it-works
- svilen_dobrev 2y ago$ rmlint -c sh:link -L -y s -p -T duplicates will produce a script which, if run, will hardlink duplicates
- deleted 2y ago[deleted]
- Analemma_ 2y agoThat's not what this app is doing though. APFS clones are copy-on-write pointers to the same data, not hardlinks.
- phiresky 2y agoIf you replace `sh:link` with `sh:clone` instead, it will. > clone: reflink-capable filesystems only. Try to clone both files with the FIDEDUPERANGE ioctl(3p) (or BTRFS_IOC_FILE_EXTENT_SAME on older kernels). This will free up duplicate extents while preserving the metadata of both. Needs at least kernel 4.2.
- wpm 2y agoOn Linux
- jamesfmilne 2y agoWould be nice if git could make use of this on macOS. Each worktree I usually work on is several gigs of (mostly) identical files. Unfortunately the source files are often deep in a compressed git pack file, so you can't de-duplicate that. (Of course, the bigger problem is the build artefacts on each branch, which are like 12G per debug/release per product, but they often diverge for boring reasons.)
- diggan 2y agoGit is a really poor fit for a project like that since it's snapshot based instead of diff based... Luckily, `git lfs` exists for working around that, I'm assuming you've already investigated that for the large artifacts?
- theamk 2y ago"git worktree" shares a .git folder between multiple checkouts. You'll still have multiple files in working copy, but at least the .pack files would be shared. It is great feature, very robust, I use it all the time. There is also ".git/objects/info/alternates", accessed via "--shared"/"--reference" option of "git clone", that allows only sharing of object storage and not branches etc... but it is has caveats, and I've only used it in some special circumstances.
- globular-toast 2y agoGit de-duplicates everything in its store (in the .git directory) already. That's how it can store thousands of commits which are snapshots of the entire repository without eating up tons of disk space. Why do you have duplicated files in the working directory, though?
- andrewla 2y agoMany comments here offering similar solutions based on hardlinks or symlinks. This uses a specific feature of APFS that allows the creation of copy-on-write clones. [1] If a clone is written to, then it is copied on demand and the original file is unmodified. This is distinct from the behavior of hardlinks or symlinks. [1] https://en.wikipedia.org/wiki/Apple_File_System#Clones https://en.wikipedia.org/wiki/Apple_File_System#Clones
- deleted 2y ago[deleted]
- bombela 2y agoAlso called reflink on Linux. Which are supported by bcachefs, Btrfs, CIFS, NFS 4.2, OCFS2, overlayfs, XFS, and OpenZFS. Sources: https://unix.stackexchange.com/questions/631237/in-linux-which-filesystems-support-reflinks#631238 https://unix.stackexchange.com/questions/631237/in-linux-whi... https://forums.veeam.com/veeam-backup-replication-f2/openzfs-2-2-support-for-reflinks-now-available-t90517.html https://forums.veeam.com/veeam-backup-replication-f2/openzfs...
- lakomen 2y ago[dead]
- radicality 2y agoHopefully doesn’t have similar bug like jdupes did https://web.archive.org/web/20210506130542/https://github.com/jbruchon/jdupes/issues/142 https://web.archive.org/web/20210506130542/https://github.co...
- deleted 2y ago[deleted]
- kalleboo 2y agoFrom the FAQ > Q: Does Hyperspace preserve file metadata during reclamation? > A: When Hyperspace replaces a file with a space-saving clone, it attempts to preserve all metadata associated with that file. This includes the creation date, modification date, permissions, ownership, Finder labels, Finder comments, whether or not the file name extension is visible, and even resource forks. If the attempt to preserve any of these piece of metadata fails, then the file is not replaced.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- david_allison 2y ago> Hyperspace can’t be installed on “Macintosh HD” because macOS version 15 or later is required. macOS 15 was released in September 2024, this feels far too soon to deprecate older versions.
- kstrauser 2y agoHe wanted to write it in Swift 6. Does it support older OS versions?
- jjcob 2y agoSwift 6 is not the problem. It's backward compatible. The problem is SwiftUI. It's very new, still barely usable on the Mac, but they are adding lots of new features every macOS release. If you want to support older versions of macOS you can't use the nice stuff they just released. Eg. pointerStyle() is a brand new macOS 15 API that is very useful.
- MBCook 2y agoI can’t remember for sure but there may also have been a recent file system API he said he needed. Or a bug that he had to wait for a fix on.
- therockhead 2y agoIt's been a while since I last looked at SwiftUI on mac, Is it really still that bad ?
- jjcob 2y agoIt's not bad, just limited. I think it's getting usable, but just barely so. They are working on it, and making it better every year. I've started using it for small projects and it's pretty neat how fast you can work with it -- but not everything can be done yet. Since they are still adding pretty basic stuff every year, it really hurts if you target older versions. AppKit is so mature that for most people it doesn't matter if you can't use new features introduced in the last 3 years. For SwiftUI it still makes a big difference.
- BWStearns 2y agoI have file A that's in two places and I run this. I modify A_0. Does this modify A_1 as well or just kind of reify the new state of A_0 while leaving A_1 untouched?
- madeofpalk 2y agoIt's called copy-on-write because when you modify A_0, it'll make a copy of the file if you write to it but not A_1. https://en.wikipedia.org/wiki/Copy-on-write#In_computer_storage https://en.wikipedia.org/wiki/Copy-on-write#In_computer_stor...
- bsimpson 2y agoWhich means if you actually edited those files, you might fill up your HD much more quickly than you expected. But if you have the same 500MB of node_modules in each of your dozen projects, this might actually durably save some space.
- _rend 2y ago> Which means if you actually edited those files, you might fill up your HD much more quickly than you expected. I'm not sure if this is what you intended, but just to be sure: writing changes to a cloned file doesn't immediately duplicate the entire file again in order to write those changes — they're actually written out-of-line, and the identical blocks are only stored once. From [the docs](^1) posted in a sibling comment: > Modifications to the data are written elsewhere, and both files continue to share the unmodified blocks. You can use this behavior, for example, to reduce storage space required for document revisions and copies. The figure below shows a file named “My file” and its copy “My file copy” that have two blocks in common and one block that varies between them. On file systems like HFS Plus, they’d each need three on-disk blocks, but on an Apple File System volume, the two common blocks are shared. [^1]: https://developer.apple.com/documentation/foundation/file_system/about_apple_file_system https://developer.apple.com/documentation/foundation/file_sy...
- 2y ago
- astennumero 2y agoWhat algorithm does the application use to figure out if two files are identical? There's a lot of interesting algorithms out there. Hashes, bit by bit comparison etc. But these techniques have their own disadvantages. What is the best way to do this for a large amount of files?
- diegs 2y agoThis reminds me of https://en.wikipedia.org/wiki/Venti_(software) https://en.wikipedia.org/wiki/Venti_(software) which was a content-addressible filesystem which used hashes for de-duplication. Since the hashes were computed at write time, the performance penalty is amortized.
- w4yai 2y agoI'd hash the first 1024 bytes of all files, and starts from there is any collision. That way you don't need to hash the whole (large) files, but only those with same hashes.
- kstrauser 2y agoAt that point, why hash them instead of just using the first 1024 bytes as-is?
- ziofill 2y agoLovely idea, but way too expensive for me.
- bsimpson 2y agoInteresting idea, and I like the idea of people getting paid for making useful things. Also, I get a data security itch having a random piece of software from the internet scan every file on an HD, particularly on a work machine where some lawyers might care about what's reading your hard drive. It would be nice if it was open source, so you could see what it's doing.
- Nevermark 2y ago> I like the idea of people getting paid for making useful things > It would be nice if it was open source > I get a data security itch having a random piece of software from the internet scan every file on an HD With the source it would be easy for others to create freebie versions, with or without respecting license restrictions or security. I am not arguing anything, except pondering how software economics and security issues are full of unresolved holes, and the world isn't getting default fairer or safer. -- The app was a great idea, indeed. I am now surprised Apple doesn't automatically reclaim storage like this. Kudos to the author.
- benced 2y agoYou could download the app, disconnect Wifi and Ethernet, run the app and the reclamation process, remove the app (remember, you have the guarantees of the macOS App Store so no kernel extensions etc), and then reconnect. Edit: this might not work with the payment option actually. I don't think you can IAP without the internet.
- diimdeep 2y agoRequires macOS 15.0 or later. – Oh god, this is so stupid and most irritating thing about macOS "Application development". It is really unfair to call it "software" it is more like "glued to recent version of OS ware", meanwhile I can still run .exe compiled in 2006, and with wine even on mac or linux.
- kstrauser 2y agoHowever, you can't run an app targeted for Windows 11 on Windows XP. How unfair is that? Curse you, Microsoft.
- ilumanty 2y agoI would also have appreciated a version that's compatible with a non-latest macOS release. Then again, this app was written with SwiftUI, which hasn't received some handy features before macOS 12 and is still way behind AppKit. When I see an app that's not compatible with the second most recent macOS, I assume the dev either didn't know better or they were too lazy to write workarounds / shims for the latest-and-greatest shiny stuff.
- DontBreakAlex 2y agoNice, but I'm not getting a subscription for a filesystem utility. Had it been a one-time $5 license, I would have bought it. At the current price, it's literally cheaper to put files in a S3 bucket or outright buy an SSD.
- benced 2y ago"I don't value software but that's not a respectable opinion so I'll launder that opinion via subscriptions"
- DontBreakAlex 2y agoWell I do value software, I'm paid $86/h to write some! I just find that for $20/year or $50 one time, you can get way more than 12G of hard drive space. I also don't think that this piece of software requires so much maintenance that it wouldn't be worth making at a lower price. I'm not saying that it's bad software, it's really great, just too expensive... Personally, my gut feeling is that the dev would have had more sales with a one time $5, and made more money overall.
- pfranz 2y agoThe first option presented is a one month non-renewing subscription for $10. I think the intention is periodically (once a year, once every few years?) you run it to reclaim space. If it was reclaiming more than a few gigs I would do it. The author talked about being very conservative on launch; skipping directories like the Photo library or others apps that actively manage data or looking across user directories. He stumbled into writing this app because he noticed the duplicated data of shared Photo libraries between different users on the same machine. That use case isn't even supported in this version. He said he plans future development to safely dedup more data--making a one time purchase less sustainable for them.
- amelius 2y agoThere are several such tools for Linux, and they are free, so maybe just change operating systems.
- the_clarence 2y agoIts interesting how Linux tools are all free when even trivial mac tools are being sold. Nothing against someone trying to monetize but the linux culture sure is nice!
- dewey 2y agoIt's not that nice to call someone's work they spent months on "trivial" without knowing anything about the internals and what they ran into.
- MadnessASAP 2y agoI don't think they meant it in a disparaging way, except maybe against Apple. Moreso that typically filesystems that can support deduplication include a deduplication tool in it's standard suite of FS tools. I too find it odd that Apple does not do this.
- alwillis 2y agoBy now, you know it’s not de-duping anything-nothing is deleted. Of course the Copy-on-Write clone functionality has been available since APFS became the default file system in 2019.
- the_clarence 2y agoI didnt mean that the work was trivial, but the tool seems like something that could be a simple CLI
- deleted 2y ago[deleted]
- gblargg 2y agoThere might be a difference in robustness. There's a monetary consequence to this developer for getting it wrong.
- scraptor 2y ago
- eikenberry 2y agoI don't understand why a simple, closed source de-dup app is at the top of the front page with 160+ comments? What is so interesting about it? I read the blog and the comments here and I still don't get it.
- benced 2y agoThe developer is popular and APFS cloning is genuinely technically interesting. (no, it's not a symlink)
- augusto-moura 2y agoCOW filesystems are older than MacOS, no surprises for me. Maybe people aren't that aware of it?
- ForOldHack 2y agoCoW - Copy on Write. Most probably on older mainframes. ( Actually newer mainframes ). "CoW is used as the underlying mechanism in file systems like ZFS, Btrfs, ReFS, and Bcachefs" Obligatory: https://en.wikipedia.org/wiki/Copy-on-write https://en.wikipedia.org/wiki/Copy-on-write
- therockhead 2y agoI assume it’s because it’s from John Siracusa, a long-time Mac enthusiast, blogger, and podcaster. If you listen to him on ATP, it’s hard not to like him, and anything he does is bound to get more than the usual upvotes on HN.
- deleted 2y ago[deleted]
- dewey 2y agoFor those mentioning that there's no price listed, it's not that easy as in the App Store the price varies by country. You can open the App Store link and then look at "In App Purchases" though. For me on the German store it looks like this: Unlock for One Year 22,99 € Unlock for One Month 9,99 € Lifetime Unlock 59,99 € So it supports both one time purchases and subscriptions. Depending on what you prefer. More about that here: https://hypercritical.co/hyperspace/#purchase https://hypercritical.co/hyperspace/#purchase
- deleted 2y ago[deleted]
- gblargg 2y agoIt would be interesting if payments bought a certain amount of saved space, and the rate was based on current storage prices, to keep it competitive with the cost of just expanding storage.
- twp 2y agoCLI tool to find duplicate files unbelievably quickly: https://github.com/twpayne/find-duplicates https://github.com/twpayne/find-duplicates
- deleted 2y ago[deleted]
- bmn__ 2y agoIt lacks the deduplication part that make its competitors useful.
- archagon 2y agoI have to confess: it miffs me that a utility that would normally fly completely under the radar is likely to make the creator thousands of dollars just because he runs a popular podcast. (Am I jealous? Oh yes. But only because I tried to sell similar apps in the past and could barely get any downloads no matter how much I marketed them. Selling software without an existing network seems nigh-on impossible these days.) Anyway, congrats to Siracusa on the release, great idea, etc. etc.
- dewey 2y agoI can understand your criticism as it's easy to arrive at that conclusion (Also a common occurrence when levelsio launches a new product, as his Twitter following is large) but it's also not fair to discount it as "just because he runs a popular podcast". The author is a "household" name in the macOS / Apple scene for a long time even before the podcast. If someone is spending all their life blogging about all things Apple on outlets like ArsTechnica and is consistently putting out new content on podcasts for decades they will naturally have a better distribution. How many years did you spend on building up your marketing and distribution reach?
- archagon 2y agoI know! I actually like him and wish him the best. I just get a bit annoyed when one of the ATP folks releases some small utility with an unclear niche and then later talks about how they've "merely" earned thousands of dollars from it. When I was an app developer, I would have counted myself lucky to have made just a hundred bucks from a similar release. The gang's popularity gives them a distorted view of the market sometimes, IMHO.
- deleted 2y ago[deleted]
- karparov 2y agoTL;DR: He wrote an OS X dedup app which finds files with the same contents and tells the filesystem that their contents are identical, so it can save space (using copy-on-write features). He points out its dangerous but could be worth it cause space savings. I wonder if the implementation is using a hash only or does an additional step to actually compare the contents to avoid hash collision issues. It's not open source, so we'll never know. He chose a pay model instead. Also, some files might not be identical but have identical blocks. Something that could be explored too. Other filesystems have that either in their tooling or do it online or both.
- sgt 2y agoAny way it can be built for 14? It requires macOS 15.
- rusinov 2y agoJohn is a the legend.
- re 2y agoOn a related note: are there any utilities that can measure disk usage of a folder taking (APFS) cloned files into account?
- Take8435 2y agoDownloaded. Ran it. Tells me "900" files can be cleaned. No summary, no list. But I was at least asked to buy the app. Why would I buy the app if I have no idea if it'll help?
- crb 2y agoFrom the FAQ: > If some eligible files were found, the amount of disk space that can be reclaimed is shown next to the “Potential Savings” label. To proceed any further, you will have to make a purchase. Once the app’s full functionality is unlocked, a “Review Files” button will become available after a successful scan. This will open the Review Window. I half remember this being discussed on ATP; the logic being that if you have the list of files, you will just go and de-dupe them yourself.
- AyyEye 2y ago> the logic being that if you have the list of files, you will just go and de-dupe them yourself. If you can do that, you can check for duplicates yourself anyway. It's not like there aren't already dozens of great apps that dedupe.
- cowsandmilk 2y ago> there aren't already dozens of great apps that dedupe. Most of those delete rather than use the features of APFS.
- eps 2y agoThis reminds me - Back in the MS-DOS days, when the RAM was sparse, there was a class of so-called "memory optimization" programs. They all inevitably found at least few KB to be reclaimed through their magic even if the same optimizer was run back to back with itself and allowed to "optimize" things. That is, on each run they always find extra memory to be freed. They ultimately did nothing but claim they did the work. Must've sold pretty well nonetheless.
- 2y ago
- galaxyLogic 2y agoOn Windows there is "Dev Drive" which I believe does a similar "copy-on-write" -thing. If it works it's a no-brainer so why isn't it the default? https://learn.microsoft.com/en-us/windows/dev-drive/#dev-drive-faqs https://learn.microsoft.com/en-us/windows/dev-drive/#dev-dri...
- siranachronist 2y agorequires refs, which still isnt supported on the system drive on windows, iirc
- galaxyLogic 2y agoGood to know. It is interesting to think how much electricity is wasted when most people don't have Copy-on-Write, all over the world.
- p_ing 2y agoCoW is a function of ReFS, shipped with Server 2016. "DevDrive" is just a marketing term for a ReFS volume which has file system filters placed in async mode or optionally disabled altogether.
- o10449366 2y agoWhat would an equivalent tool be on linux? I guess it depends on the filesystem?
- kccqzy 2y agoI've used bees successfully. https://github.com/Zygo/bees https://github.com/Zygo/bees
- JackYoustra 2y agoWhat's the difference with jdupes?
- bmn__ 2y agoNothing much, jdupes already does the best thing. https://manpages.debian.org/jdupes/jdupes.1#B https://manpages.debian.org/jdupes/jdupes.1#B
- mattgreenrocks 2y agoWhat jumped out to me: > Finally, at WWDC 2017, Apple announced Apple File System (APFS) for macOS (after secretly test-converting everyone’s iPhones to APFS and then reverting them back to HFS+ as part of an earlier iOS 10.x update in one of the most audacious technological gambits in history). How can you revert a FS change like that if it goes south? You'd certainly exercise the code well but also it seems like you wouldn't be able to back out of it if something was wrong.
- quux 2y agoIIRC migrating from HFS+ to APFS can be done without touching any of the data blocks and a parallel set of APFS metadata blocks and superblocks are written to disk. In the test migrations Apple did the entire migration including generating APFS superblocks but held short of committing the change that would permanently replace the HFS+ superblocks with APFS ones. To roll back they “just” needed to clean up all the generated APFS superblocks and metadata blocks.
- MBCook 2y agoI think that’s what they did too. And it was a genius way of testing. They did it more than once too I think. Run the real thing, throw away the results, report all problems back to the mothership so you have a high chance of catching them all even on their multi-hundred million device fleet.
- k1t 2y agoYes, that's how it's described in this talk transcript: https://asciiwwdc.com/2017/sessions/715 https://asciiwwdc.com/2017/sessions/715 Let’s say for simplification we have three metadata regions that report all the entirety of what the file system might be tracking, things like file names, time stamps, where the blocks actually live on disk, and that we also have two regions labeled file data, and if you recall during the conversion process the goal is to only replace the metadata and not touch the file data. We want that to stay exactly where it is as if nothing had happened to it. So the first thing that we’re going to do is identify exactly where the metadata is, and as we’re walking through it we’ll start writing it into the free space of the HFS+ volume. And what this gives us is crash protection and the ability to recover in the event that conversion doesn’t actually succeed. Now the metadata is identified. We’ll then start to write it out to disk, and at this point, if we were doing a dry-run conversion, we’d end here. If we’re completing the process, we will write the new superblock on top of the old one, and now we have an APFS volume.
- formerphotoj 2y agoOmniDiskSweeper (I know, not exactly the same thing, but still...)
- jonhohle 2y agoI made a command line utility called `dedup` a while back to do the same thing. It has a dry-run mode, will “intelligently” choose the best clone source, understands hard links and other clones, preserves metadata, deals with HFS compressed files properly. It hasn’t destroyed any of my own data, but like any file system tool, use at your own risk. 0 - https://github.com/ttkb-oss/dedup https://github.com/ttkb-oss/dedup
- LVB 2y agoJust tried it, and it works well! I didn't realize the potential of this technique until I saw just how many dupes there were of certain types of files, especially in node_modules. It wasn't uncommon to see it replace 50 copies of some js file with one, and that was just in a specific subdirectory. I see it is "pre-release" and sort of low GH stars (== usage?), so I'm curious about the stability since this type of tool is relatively scary if buggy.
- jonhohle 2y agoI use it on my family photos, work documents, etc. and have not run into an issue that I haven’t added a test for. I didn’t commercialize it because I didn’t know what I didn’t know, but the utility does try to fail quickly if any of the file system operations fail (cloning, metadata duplication, atomic swaps, etc.). Whenever using it on something sensitive that I can’t back up first for whatever reason, I make checksum files and compare them afterwards. I’ve done this many times on hundreds of GB and haven’t seen corruption. Caveat emptor. There is one huge caveat I should add to the README - block corruption happens. Having a second copy of a file is a crude form of backup. Cloning causes all instances to use the same block, so if that one instance is corrupted, all clones are. That’s fine for software projects with generated files that can be rebuilt or checked out again, but introduces some risk for files that may not otherwise be replaceable. I keep multiple backups of all that stuff in hardware other than where I’m deduping, so I dedup with abandon. I’m a nobody with no audience. Maybe some attention here will get some users.
- ncann 2y agoSomewhat unrelated but I believe the dupe issue with node_modules is the main reason to use pnpn instead of npm - pnpm just uses a single global package repo on your machine and creates links inside node_modules as needed.
- animal_spirits 2y agoWish John went with the name superdeduper :/ :P
- kalleboo 2y agoI was hoping for Storacusa
- whalesalad 2y agoWas pleasantly surprised to see that this is John Siracusa - the original GOAT when it comes to macOS release articles on Ars.
- PStamatiou 2y agoWasn't able to use it on a few directories I tried as they were inside iCloud Drive.
- glitchc 2y agoThis is cool! Wait a minute, what happens to copies on different physical drives. Are they cloned too?
- pazer 2y ago;( requires macos 15
- sgt 2y agoAm also still on macOS 14. Keep postponing the upgrade. I've got stuff to do.
- alwillis 2y agoJust want to mention: Apple ships a modified version of the copy command (good old cp) that supports the ability to use the cloning feature of APFS by using the -c flag.
- kccqzy 2y agoAnd in case your cp doesn't support it, you could also do it by invoking Python. Something like `import Foundation; Foundation.NSFileManager.defaultManager().copyItemAtPath_toPath_error_(...)`.
- jlhawn 2y agothat function name `copyItemAtPath_toPath_error_` sounds like it was ported directly from objective-c!
- kccqzy 2y agoThey might not have ported it. They could have a Python __getattr__ implementation that returns a callable that simply uses objc_msgSend under the hood.
- xenadu02 2y agoCorrect. Foundation's NSFileManager / FileManager will automatically use clone for same-volume copies if the underlying filesystem supports it. This makes all file copies in all apps that use Foundation support cloning even if the app does nothing. libcopyfile also supports cloning via two flags: COPYFILE_CLONE and COPYFILE_CLONE_FORCE. The former clones if supported (same volume and filesystem supports it) and falls back to actual copy if not. The force variant fails if cloning isn't supported.
- alwillis 2y agoApparently the cp command in CoreUtilities also supports copy-on-write on macOS: https://unix.stackexchange.com/questions/311536/cp-reflink-auto-for-macos-x https://unix.stackexchange.com/questions/311536/cp-reflink-a...
- omnibrain 2y agoMy personal head canon is that Steve Jobs personally cancelled ZFS in OSX because Jonathan Schwartz prematurely announced it.
- staticfish 2y agoI assumed it was patent-encumbered that could not be reconciled by either party.
- wink 2y agoOh wow, what a funny coincidence. I hadn't visited the site in a couple of years but someone linked me "Front and Center" yesterday, so I saw the icon for this app and had no clue it had only appeared there maybe hours earlier. The idea is not new, of course, and I've written one of these (for Linux, with hardlinks) years ago but in the end just deleted all the duplicate files in my mp3 collection and didn't touch the rest of the files on the disk, because not a lot of size was reclaimed. I wonder for whom this really saves a lot of space. (I saw someone mentioning node_modules, had to chuckle there). But today I learned about this APFS feature, nice.
- cytocync 2y ago[dead]
- TrapLord_Rhodo 2y agoStart with a story, narrow it down to a problem and show how your solution magically solves that problem. Such a fine example of GREAT marketing.
- gcr 2y agoI’ve experimented with reflinks and other APFS operations. Here’s a question though: how does this work with transparently compressed files on APFS? In my past experience, using reflinks is fine and using transparent compression is fine, but combining them leads to hard-to-debug file corruption.