DrFrugal

5 ms·

1.
Why Can't Hashes Just Agree on Endianness?
8 points by DrFrugal 2y ago|2 comments
2.
by DrFrugal 2y ago
thanks for the link, but i think i found my favored solution right now, which is extracting the archive into raw files (and header, hashtable, blocktable) and then reassembling the files on demand into a byte-for-byte equal archive on deman
3.
by DrFrugal 2y ago
jdupes is quite powerful and allows you to create different types of links and even more. i used it for creating hardlinks, since this was the most efficient for my use case. from what i read, there is currently also a rewrite of jdupes in
4.
by DrFrugal 2y ago
shouldn't be the case, and i would implement a proper verify when exploding the MPQ file, by running a reassamble and hash comparison right afterwards.
5.
by DrFrugal 2y ago
a BTRFS subvolume can be exported as a snapshot, and this can be passed around rather easily. i know the concern mounting BTRFS, but there are windows drivers for that as well, and you can also mount it via WSL nowadays, to have proper linu
6.
by DrFrugal 2y ago
thanks for mentioning those 2 projects, will check them out over the holidays and do some experimenting ^^
7.
by DrFrugal 2y ago
it's a question of both "i can store this on my 4 TiB NVMe disk" and "i can share this with other people over the internet in a timely manner"
8.
by DrFrugal 2y ago
This is correct - the main goal is to have this rather compact, but still having good read times. This will allow me to store it on my new 4 TiB NVMe drive. A lot of iterative scanning will happen, because I search for interesting informati
9.
by DrFrugal 2y ago
World of Warcraft - from the earliest (publicly leaked) Alpha up and including the very last Wrath of the Lich King version. Other people are already working on getting backups of the next 2 expansions working again to add those client vers
10.
by DrFrugal 2y ago
will probably write the MPQ blobs down to disk and deduplicate via hardlinks and additionally on block level. i don't know about restic (or borg, which was also recommended), but i will read up on it and doe some tests with it, regardl
11.
by DrFrugal 2y ago
thanks for the link - i will write my extraction code though, since the format is very simplistic, and it gives me fine grained control over how things are done. appreciate your help though!
12.
by DrFrugal 2y ago
prying apart the MPQ file into it's parts and writing them down to disk is at the moment probably the strongest contender for my solution. it will cause the parts to be block aligned, be able to be hardlinked and cut down on metadata a
13.
by DrFrugal 2y ago
even though I do not like it too much, i think i will have to pry apart the MPQ files into it's distinct parts, and write them down to the filesystem individually (and then deleting the original file) - basically what i wanted to do wi
14.
by DrFrugal 2y ago
i tried FICLONERANGE via a python wrapper btw - it turns out, that i can only clone ranges aligned to block boundaries :( BTRFS is very neat per se, but documentation and help (most of all in very niche cases like this one here lol) is not
15.
by DrFrugal 2y ago
you are absolutely on point - i would prefer having a real filesystem with deduplication (not compression), which offers data in a compact form, with good read speed for further processing. i was already brainstorming of writing a custom pu
16.
by DrFrugal 2y ago
duperemove has "--dedupe-options=partial" which also enables this, not just full extents. the issue still is, that the data within the archive is not block aligned, thus preventing me from deduplicating them properly
17.
by DrFrugal 2y ago
this is either a very big coincidence, or you are in the datamining discord as well. the original archive i base my project on uses RMAN to store everything :D --- thanks for the hint about the FICLONERANGE ioctl... it seems to be fine grai
18.
by DrFrugal 2y ago
this was not helpful at all, and i think you also did not read the goals of this project
19.
Tell HN: Deduplicating a 10.4 TiB game preservation archive (WIP)
30 points by DrFrugal 2y ago|53 comments