The main problem is that you don't have the time and energy to go through 23TB of files. You want a simple solution to pack them all in a safe place and sort th

The main problem is that you don't have the time and energy to go through 23TB of files. You want a simple solution to pack them all in a safe place and sort them later (which will never happen, but it's ok, we all have this problem). Here's what to do: 1) Get a few very large HDDs, the largest you deem afordable. In total, you should have about 4x - 8x the capacity of your current files. 2) Split them in two groups: main storage and backup storage. 3) On each group, create a btrfs RAID1 and mount it with compression. Start with a btrfs raid1c3 if you can afford 3 drives, and downgrade to raid1 if you run out of space. 4) Copy all your files there. You may want to make a directory for each media you copy data from. You should extract all archives that you may already have created and let btrfs deal with compression - this allows for deduplication. 5) Run a deduplication tool that supports btrfs (https://github.com/Zygo/bees https://github.com/Zygo/bees). Creating btrfs with sha256 checksums is probably better for block deduplication while using crypto acceleration available on most systems. Safety rules: - Only connect the backup drives when you do the backup. Keep the backup in the closet or, preferably, in another building (in case of fire, etc.). - Label the drives. You don't want to accidentally mix the two groups. Later, when you run out of space: - buy a pair of even bigger drives and add them to the btrfs volume - remove the old drives from the volume (this step is very time consuming) or - have a huge case or hdd rack and keep adding drives without removing any One last thing: you may want to use a NAS or DAS, at least for the backup group.