3 ms·

The costs of moving that amount of data is crazy. I am surprised the government hasn't seized the hardware, and chucked it in a warehouse. I did some back of a

by jlawer 15y ago

The costs of moving that amount of data is crazy. I am surprised the government hasn't seized the hardware, and chucked it in a warehouse. I did some back of an envelope calculations... and its absolutely crazy. Tape would require over 17,000 Ultrium tapes. Now you could De-dupe... but the hardware to process and dedupe that much data.... not really an option. Not to mention the time to write that many tapes... Something like thumpers (48 disk sun x86 boxes) would be expensive, last time I looked they were around say $30k for a large order... 160tb usable assuming 4tb disks are the thumper is split into 4 Raid 6 arrays... thats 160 thumpers... 4.8 Million Even backblaze pods would likely be well over a Million... This doesn't even cover hosting costs, transfer and such. Not to mention to be usable in court there are going to have to be processes in place to document compliance and validity of the copy.... All in all not a great place for Carpathia to be in.

Tuna-Fish 15y ago
> Now you could De-dupe... but the hardware to process and dedupe that much data.... not really an option. FYI, the data is already comprehensively de-duped.
- jlawer 15y ago
  I know they dedupe on the file level, but I wonder if they are doing block level deduping... as without a big shared storage infrastructure block level deduping becomes pretty hard to serve at high speed from as the reads potentially become distributed across hundreds of nodes... To build out web scale systems you generally use commodity gear and accept the overhead of duplication, heavy deduping requires massive IO, and there is no way i can see you can be dealing with that much data have that level of IOPS and be profitable charging what they charge.