7 ms·

Replying to myself now that I've had a chance to try the scan, but not the deduplication. I work with disc images, program binaries, intermediate representation

Replying to myself now that I've had a chance to try the scan, but not the deduplication. I work with disc images, program binaries, intermediate representations in a workspace that's 7.6G. A few notes: * By default it doesn't scan everything. It ignores all files but those in an allow list. The way the allow list is structured, it seems like Hyperspace needs to understand the content of a file. As an end user, I have no idea what the difference between a Text file and a Source Code file would be or how Hyperspace would know. Hyperscan only found 360MB to dedup. Allowing all files increased that to 842MB. * It doesn't scan files smaller than 100 KB by default. Disabling the size limit along with allowing all files increased that to 1.1GB * With all files and no size limit it scanned 67,309 of 68,874 files. `dedup` scans 67,426. * It says 29,522 files are eligible. Eligible means they can be deduped. `dedup` only fines 29,447. There are 76 already deduped files, which is an off-by-one, so I'm not sure what the difference is. * Scanning files in Hyperspace took around 50s vs `dedup` at 14s * It seems to scan the file system, then do a duplication calculation, then do the deduplication. I'm not sure why the first shouldn't be done together. I chose to queue any filesystem metadata as it was scanned and in parallel start calculating duplicates. The vast majority of the time files can be mismatched by size, which is available from `fts_read` "for free" while traversing the directory. * Hyperspace found 1.1GB to save, `dedup` finds 1.04GB and 882MB already saved (from previous deduping) * I'm not going to buy Hyperspace at this time, so I don't know how long it takes to dedup or if it preserves metadata or deals with strange files. `dedup` took 31s to scan and deduplicate. * After deduping with `dedup`, Hyperscan thinks there are still 2 files that can be deduped. * Hyperspace seems to understand it can't dedup files with multiple hard links, empty files, and some of the other things `dedup` also checks for. * I can't test ACLs or any other attribute preservation like that without paying. `strings` suggests those are handled. HFS Compression is a tricky edge case, but I haven't tested how Hyperspace's scan deals with those.

karparov 2y ago
I'm a little surprised that folks here are investing so much time into this app. It's closed source, only available for a non-obious amount, time-limited or subscription-based and lots of details of how it works are missing. With a FOSS project this would have been expected, but with a ShareWare-style model? Idk..
- codemusings 2y ago
  John has reiterated multiple times on his podcast that he doesn't want to deal with thousands of support requests when making his apps open source and free. All his apps are personal itches he scratched and he sells them not to make a profit but to make the barrier of entry high enough to make user feedback manageable.
  - 0x0203 2y ago
    Absolutely no judgement for however people want to licence and distribute their software, but I've seen the support burden used as justification for closed source/selling software quite a bit recently, and wonder how often people might be conflating open source with open development. There's no reason an open source project has to accept bug reports or pull requests from anyone. See SQLite or many of the tools from Fabrice Bellard for example. Again, I've got no problem with people selling software or closed source models, but I've never understood using this justification. Maybe in this instance he's a well known public figure with published contact info that people will abuse?
    - cmckn 2y ago
      Didn’t SQLite developer(s) famously receive a flood of phone calls because McAfee antivirus used it in a way that was visible (and “suspicious”) to its users? One does not simply “not accept bug reports”. https://github.com/sqlite/sqlite/blob/e8346d0a889c89ec8a78e65abc33257a6c6fb81a/src/os.h#L65 https://github.com/sqlite/sqlite/blob/e8346d0a889c89ec8a78e6...
      - deleted 2y ago
        [deleted]
      - deleted 2y ago
        [deleted]
  - wubrr 2y ago
    > he doesn't want to deal with thousands of support requests when making his apps open source and free. Who says you have to deal with support requests if you open source something? > All his apps are personal itches he scratched and he sells them not to make a profit but to make the barrier of entry high enough to make user feedback manageable. That makes no sense
eduction 2y ago
On metadata, the excellent faq addresses that specifically (it does preserve). (I had the same question)

karparov 2y ago

I'm a little surprised that folks here are investing so much time into this app. It's closed source, only available for a non-obious amount, time-limited or subscription-based and lots of details of how it works are missing. With a FOSS project this would have been expected, but with a ShareWare-style model? Idk..

codemusings 2y ago
John has reiterated multiple times on his podcast that he doesn't want to deal with thousands of support requests when making his apps open source and free. All his apps are personal itches he scratched and he sells them not to make a profit but to make the barrier of entry high enough to make user feedback manageable.
- 0x0203 2y ago
  Absolutely no judgement for however people want to licence and distribute their software, but I've seen the support burden used as justification for closed source/selling software quite a bit recently, and wonder how often people might be conflating open source with open development. There's no reason an open source project has to accept bug reports or pull requests from anyone. See SQLite or many of the tools from Fabrice Bellard for example. Again, I've got no problem with people selling software or closed source models, but I've never understood using this justification. Maybe in this instance he's a well known public figure with published contact info that people will abuse?
  - cmckn 2y ago
    Didn’t SQLite developer(s) famously receive a flood of phone calls because McAfee antivirus used it in a way that was visible (and “suspicious”) to its users? One does not simply “not accept bug reports”. https://github.com/sqlite/sqlite/blob/e8346d0a889c89ec8a78e65abc33257a6c6fb81a/src/os.h#L65 https://github.com/sqlite/sqlite/blob/e8346d0a889c89ec8a78e6...
    - deleted 2y ago
      [deleted]
    - deleted 2y ago
      [deleted]
- wubrr 2y ago
  > he doesn't want to deal with thousands of support requests when making his apps open source and free. Who says you have to deal with support requests if you open source something? > All his apps are personal itches he scratched and he sells them not to make a profit but to make the barrier of entry high enough to make user feedback manageable. That makes no sense

eduction 2y ago

On metadata, the excellent faq addresses that specifically (it does preserve). (I had the same question)