3 ms·

this was not helpful at all, and i think you also did not read the goals of this project

brudgers 2y ago
I assumed the goal was archiving games for preservation. If the goal is algorithmic erasure of data rather than preservation of games, then “archive” might again create confusion like the type I probably have. If there is a strong business case for deduplication, I recommend hiring a consultant with expertise and experience in the problem. To be clear search is the only way to identify redundancy. If you have search, then redundancy is not a problem.
- rcxdude 2y ago
  Often these archive projects have the goal of propagating the archive across many different systems, for which reducing size is very valuable. This is basically a compression exercise in the case where there are many large duplicated blocks.
  - brudgers 2y ago
    Thanks, that makes sense. The question began, I am working on a game preservation project.
    - canucker2016 2y ago
      The line you referenced is a statement - not a question. The third line of the post is "The goals are: - bring the size down - retain good read speed (for further processing/reversing) - easy sharable format - lower end machines can use it"
  - DrFrugal 2y ago
    This is correct - the main goal is to have this rather compact, but still having good read times. This will allow me to store it on my new 4 TiB NVMe drive. A lot of iterative scanning will happen, because I search for interesting information, which helps reverse engineering. Also it allows me to share this with other people over the internet before I kick the bucket in a few decades... transferring 10.4 TiB would be rather boring :D
jrm4 2y ago
I think the confusion here (for me too) is what precisely is the purpose/meaning of "bring the size down?" aka, is the is a "make it easier to search" question or "we need to take up fewer bytes" question (which, perhaps, used to be the same question, but aren't necessarily now?)
- rcxdude 2y ago
  I think it's "I want people to make copies of this"
- DrFrugal 2y ago
  it's a question of both "i can store this on my 4 TiB NVMe disk" and "i can share this with other people over the internet in a timely manner"

I assumed the goal was archiving games for preservation. If the goal is algorithmic erasure of data rather than preservation of games, then “archive” might again create confusion like the type I probably have. If there is a strong business case for deduplication, I recommend hiring a consultant with expertise and experience in the problem. To be clear search is the only way to identify redundancy. If you have search, then redundancy is not a problem.

rcxdude 2y ago
Often these archive projects have the goal of propagating the archive across many different systems, for which reducing size is very valuable. This is basically a compression exercise in the case where there are many large duplicated blocks.
- brudgers 2y ago
  Thanks, that makes sense. The question began, I am working on a game preservation project.
  - canucker2016 2y ago
    The line you referenced is a statement - not a question. The third line of the post is "The goals are: - bring the size down - retain good read speed (for further processing/reversing) - easy sharable format - lower end machines can use it"
- DrFrugal 2y ago
  This is correct - the main goal is to have this rather compact, but still having good read times. This will allow me to store it on my new 4 TiB NVMe drive. A lot of iterative scanning will happen, because I search for interesting information, which helps reverse engineering. Also it allows me to share this with other people over the internet before I kick the bucket in a few decades... transferring 10.4 TiB would be rather boring :D

jrm4 2y ago

I think the confusion here (for me too) is what precisely is the purpose/meaning of "bring the size down?" aka, is the is a "make it easier to search" question or "we need to take up fewer bytes" question (which, perhaps, used to be the same question, but aren't necessarily now?)

rcxdude 2y ago
I think it's "I want people to make copies of this"
DrFrugal 2y ago
it's a question of both "i can store this on my 4 TiB NVMe disk" and "i can share this with other people over the internet in a timely manner"