4 ms·

Agreed. Deduplication should be the bigger goal, in my opinion.

DiggyJohnson 4y ago
Even then, I wouldn’t want a file with text + illustrations to be considered a dupe of a text-only copy of the same work.
- samatman 4y ago
  IMHO a process which is lossy should never be described as deduplication. What would work out fairly well for this use case is to group files by similarity, and compress them with an algorithm which can look at all 'editions' of a text. This should mean that storing a PDF with a (perhaps badly, perhaps brilliantly) type-edited version next to it would 'weigh' about as much as the original PDF plus a patch.
  - duskwuff 4y ago
    > IMHO a process which is lossy should never be described as deduplication. Depends. There are going to be some cases where files aren't literally duplicates, but the duplicates don't add any value -- for example, MOBI conversions of EPUB files, or multiple versions of an EPUB with different publisher-inserted content (like adding a preview of a sequel, or updating an author's bibliography).
    - samatman 4y ago
      Splitting those into two cases: I think getting rid of format conversions (which can, after all, be performed again) is worthwhile, but isn't deduplication, that's more like pruning. Multiple versions of an EPUB with slightly different content is exactly the case where a compression algorithm with an attention span, and some metadata to work with can, get the multiple copies down enough in size that there's no point in disposing of the unique parts.
- ajsnigrutin 4y ago
  Plus there are a lot of books, where one version is a high quality scan, but no OCR, and the other is OCRed scan (with a bunch of errors, but searching works 80% of the time) and horrible scan quality. Also, some books included appendices, that are scanned in some versions but not in others, plus large posters, that are shrunk to a4 size in one version, split onto multiple a4 pages in another, and one huge page in a third version. Then there are zips of books, containing 1 pdf + eg. example code, libraries, etc (eg progrmaming books).
CamperBob2 4y ago
Have to be careful there. A jihad against duplication means that poor-quality scans will drive out good ones, or prevent them from ever being created. Especially if you're misguided enough to optimize for minimum file size. I agree with samatman's position below: as long as the format is the slightest bit lossy -- and it always will be -- aggressive deduplication has more downsides than upsides.
- willnonya 4y ago
  While intended to agree the duplicates need to be easily identifiable and preferably filterable by quality for bulk downloads.
- exmadscientist 4y ago
  Deduplication doesn't have to mean removal. It might be just tagging. It would be very nice to be able to fetch the "best filesize" version of the entire collection, then pull down the "best quality" editions of only a few things I'm particularly interested in.
signaru 4y ago
Probably only safe in cases where the files in question are exactly the same binaries (if binary diffing can be automated somehow).

Even then, I wouldn’t want a file with text + illustrations to be considered a dupe of a text-only copy of the same work.

samatman 4y ago
IMHO a process which is lossy should never be described as deduplication. What would work out fairly well for this use case is to group files by similarity, and compress them with an algorithm which can look at all 'editions' of a text. This should mean that storing a PDF with a (perhaps badly, perhaps brilliantly) type-edited version next to it would 'weigh' about as much as the original PDF plus a patch.
- duskwuff 4y ago
  > IMHO a process which is lossy should never be described as deduplication. Depends. There are going to be some cases where files aren't literally duplicates, but the duplicates don't add any value -- for example, MOBI conversions of EPUB files, or multiple versions of an EPUB with different publisher-inserted content (like adding a preview of a sequel, or updating an author's bibliography).
  - samatman 4y ago
    Splitting those into two cases: I think getting rid of format conversions (which can, after all, be performed again) is worthwhile, but isn't deduplication, that's more like pruning. Multiple versions of an EPUB with slightly different content is exactly the case where a compression algorithm with an attention span, and some metadata to work with can, get the multiple copies down enough in size that there's no point in disposing of the unique parts.
ajsnigrutin 4y ago
Plus there are a lot of books, where one version is a high quality scan, but no OCR, and the other is OCRed scan (with a bunch of errors, but searching works 80% of the time) and horrible scan quality. Also, some books included appendices, that are scanned in some versions but not in others, plus large posters, that are shrunk to a4 size in one version, split onto multiple a4 pages in another, and one huge page in a third version. Then there are zips of books, containing 1 pdf + eg. example code, libraries, etc (eg progrmaming books).

CamperBob2 4y ago

Have to be careful there. A jihad against duplication means that poor-quality scans will drive out good ones, or prevent them from ever being created. Especially if you're misguided enough to optimize for minimum file size. I agree with samatman's position below: as long as the format is the slightest bit lossy -- and it always will be -- aggressive deduplication has more downsides than upsides.

willnonya 4y ago
While intended to agree the duplicates need to be easily identifiable and preferably filterable by quality for bulk downloads.
exmadscientist 4y ago
Deduplication doesn't have to mean removal. It might be just tagging. It would be very nice to be able to fetch the "best filesize" version of the entire collection, then pull down the "best quality" editions of only a few things I'm particularly interested in.

signaru 4y ago

Probably only safe in cases where the files in question are exactly the same binaries (if binary diffing can be automated somehow).