4 ms·

Yeah. It gets even more complicated, because tape drives and tape media has different failure rates. At the places where I used tape, we used a more efficient

Yeah. It gets even more complicated, because tape drives and tape media has different failure rates. At the places where I used tape, we used a more efficient encoding for archival tape backups. Rather than storing 2 or 3 copies, we used forward error correction with something like 30% overhead. This, then, gets even more complicated to evaluate, because it multiplies how “hungry” your backup / archive system is for ingesting data in order to remain efficient & still write data out to tape by whatever deadlines you have set. If you store 3 copies of data on LTO-8, then you write data in blocks of 12TB, with one copy on three tapes. If you use forward error correction, you might do something like write out 96TB at once on 11 tapes. You use less than half as many tapes in the long run, but you need to feed the machine faster in order to meet deadlines.

adrian_b 3y ago
While it is true that more complex error correction methods can reduce a lot the amount of tapes needed for storage, the simple approach of making 2 or 3 copies remains necessary in many cases, because storing the copies in different geographical locations is typically the only method that can protect an archive against a calamity that may destroy an entire storage center. It is possible to use RAID-5/RAID-6-like encoding schemes that can survive the complete destruction of 1 or even 2 storage centers while using less tapes than with simple copies, but such encoding schemes can be used only by very large organizations, which use more than 3 separate storage centers.
- klodolph 3y ago
  Yes, the needs for archives (sole copy) and backups (additional copies of live data) are different. > It is possible to use RAID-5/RAID-6-like encoding schemes that can survive the complete destruction of 1 or even 2 storage centers while using less tapes than with simple copies, but such encoding schemes can be used only by very large organizations, which use more than 3 separate storage centers. Scenarios where two storage centers are destroyed—that’s extreme. The most paranoid scenarios I’m normally willing to entertain are along the lines of one data center burns to the ground in a generator fire, and somebody drives a truck full of backup tapes into a ditch and they’re all covered with mud and sand. Tapes have a high enough failure rate that you benefit from forward error correction and you benefit from planning to handle individual tape failures. This includes stuff like the tape leader breaking, somebody losing a tape, damage during transport, water damage in storage, etc. There’s a layered approach here, where you plan for different disasters at different levels of the stack. Each layer exposes some certain failure rate to the layer above it, and deals with some certain failure rate at the layer below it. When I think of backups, I often imagine a top-level data storage system that has a geographically distributed hot backup, and then an offline cold backup. This lets you survive complete destruction of one data center, or lets you survive a catastrophic software bug that destroys data (and a bunch of tapes are damaged on top of that). Pretty good baseline, IMO.

adrian_b 3y ago

While it is true that more complex error correction methods can reduce a lot the amount of tapes needed for storage, the simple approach of making 2 or 3 copies remains necessary in many cases, because storing the copies in different geographical locations is typically the only method that can protect an archive against a calamity that may destroy an entire storage center. It is possible to use RAID-5/RAID-6-like encoding schemes that can survive the complete destruction of 1 or even 2 storage centers while using less tapes than with simple copies, but such encoding schemes can be used only by very large organizations, which use more than 3 separate storage centers.

klodolph 3y ago
Yes, the needs for archives (sole copy) and backups (additional copies of live data) are different. > It is possible to use RAID-5/RAID-6-like encoding schemes that can survive the complete destruction of 1 or even 2 storage centers while using less tapes than with simple copies, but such encoding schemes can be used only by very large organizations, which use more than 3 separate storage centers. Scenarios where two storage centers are destroyed—that’s extreme. The most paranoid scenarios I’m normally willing to entertain are along the lines of one data center burns to the ground in a generator fire, and somebody drives a truck full of backup tapes into a ditch and they’re all covered with mud and sand. Tapes have a high enough failure rate that you benefit from forward error correction and you benefit from planning to handle individual tape failures. This includes stuff like the tape leader breaking, somebody losing a tape, damage during transport, water damage in storage, etc. There’s a layered approach here, where you plan for different disasters at different levels of the stack. Each layer exposes some certain failure rate to the layer above it, and deals with some certain failure rate at the layer below it. When I think of backups, I often imagine a top-level data storage system that has a geographically distributed hot backup, and then an offline cold backup. This lets you survive complete destruction of one data center, or lets you survive a catastrophic software bug that destroys data (and a bunch of tapes are damaged on top of that). Pretty good baseline, IMO.