4 ms·

When I did the calculations a few years ago, break-even was somewhere around 150 TB, and I don’t think it’s changed too much since then. This is just considerin

When I did the calculations a few years ago, break-even was somewhere around 150 TB, and I don’t think it’s changed too much since then. This is just considering the cost of drives and media. Obviously, the real inflection point is going to be different depending on all sorts of factors that may be specific to your situation and your priorities. Usability isn’t something you can ignore, but it’s not like hard drives are perfect either—do you buy a big NAS / SAN setup and plug drives into it? Will it get full? And tape has the advantage that it’s completely immune, out of the box, to ransomware. I think there are four cases that really scream for tape. 1. Data hoarders, who just want to store as much data as possible for the cost. There’s an r/DataHoarder subreddit if you’re curious about these people. 2. Archivists, who want to store lots of data long-term. Tape is a lot easier to store. I recommend that archivists standardize on a specific generation of tape for as long as possible and don’t mix generations (don’t mix LTO4 and LTO5, for example, despite the fact that the drives are advertised as working with both). 3. Companies with recordkeeping requirements, like SOX (Sarbanes-Oxley). Tape is just really good for that. It has a way of surviving problems in your IT department. 4. Companies with enough data that they can put a line-item on the budget for backups, and justify the operational cost of tape—support contracts, keeping staff on hand who know how to use tape, that kind of thing. Of these, the data hoarders are going to use the 150 TB break-even point just because they want as much data as possible. Everybody else is going to make decisions based on other factors, like staffing or compliance. There are a number of gotchas, like problems with mixing tape generations, the prospect of using robotic tape libraries, and support contracts, that make the tradeoff much more nuanced.

adrian_b 3y ago
Yes, considering the additional cost of a tape drive versus the price difference per stored TB between tapes and HDDs gives an intersection point somewhere between 100 TB and 300 TB. Taking into account that for long term archiving it is necessary to store 2 or 3 copies of the data reduces proportionally the threshold above which tape is preferable. The storage cost per TB includes not only the purchase price but also the lifetime of the media, e.g. a HDD model with a warranty of 2 years cannot be trusted to store data for a longer time. Tape is guaranteed for 30 years, but the storage time is normally reduced to about 10 years by the risk that the corresponding tape drives may become obsolete.
- klodolph 3y ago
  Yeah. It gets even more complicated, because tape drives and tape media has different failure rates. At the places where I used tape, we used a more efficient encoding for archival tape backups. Rather than storing 2 or 3 copies, we used forward error correction with something like 30% overhead. This, then, gets even more complicated to evaluate, because it multiplies how “hungry” your backup / archive system is for ingesting data in order to remain efficient & still write data out to tape by whatever deadlines you have set. If you store 3 copies of data on LTO-8, then you write data in blocks of 12TB, with one copy on three tapes. If you use forward error correction, you might do something like write out 96TB at once on 11 tapes. You use less than half as many tapes in the long run, but you need to feed the machine faster in order to meet deadlines.
  - adrian_b 3y ago
    While it is true that more complex error correction methods can reduce a lot the amount of tapes needed for storage, the simple approach of making 2 or 3 copies remains necessary in many cases, because storing the copies in different geographical locations is typically the only method that can protect an archive against a calamity that may destroy an entire storage center. It is possible to use RAID-5/RAID-6-like encoding schemes that can survive the complete destruction of 1 or even 2 storage centers while using less tapes than with simple copies, but such encoding schemes can be used only by very large organizations, which use more than 3 separate storage centers.
    - klodolph 3y ago
      Yes, the needs for archives (sole copy) and backups (additional copies of live data) are different. > It is possible to use RAID-5/RAID-6-like encoding schemes that can survive the complete destruction of 1 or even 2 storage centers while using less tapes than with simple copies, but such encoding schemes can be used only by very large organizations, which use more than 3 separate storage centers. Scenarios where two storage centers are destroyed—that’s extreme. The most paranoid scenarios I’m normally willing to entertain are along the lines of one data center burns to the ground in a generator fire, and somebody drives a truck full of backup tapes into a ditch and they’re all covered with mud and sand. Tapes have a high enough failure rate that you benefit from forward error correction and you benefit from planning to handle individual tape failures. This includes stuff like the tape leader breaking, somebody losing a tape, damage during transport, water damage in storage, etc. There’s a layered approach here, where you plan for different disasters at different levels of the stack. Each layer exposes some certain failure rate to the layer above it, and deals with some certain failure rate at the layer below it. When I think of backups, I often imagine a top-level data storage system that has a geographically distributed hot backup, and then an offline cold backup. This lets you survive complete destruction of one data center, or lets you survive a catastrophic software bug that destroys data (and a bunch of tapes are damaged on top of that). Pretty good baseline, IMO.
mauvehaus 3y ago
Another big case besides SOX compliance is medical records generally or imagery specifically. Basically anywhere where you have a lot of data that has to be retained indefinitely for regulatory compliance or practical reasons is a great case for tape. But yeah, the robotic library and the service costs are pretty high until you hit a huge amount of data. Source: supported a couple of large medical installations for a couple years many years back. Can confirm that you're dealing with a lot of mechanical complexity per GB until you get to an absolutely enormous amount of data. I genuinely can't imagine breaking even with a robotic library you can't climb into.

adrian_b 3y ago

Yes, considering the additional cost of a tape drive versus the price difference per stored TB between tapes and HDDs gives an intersection point somewhere between 100 TB and 300 TB. Taking into account that for long term archiving it is necessary to store 2 or 3 copies of the data reduces proportionally the threshold above which tape is preferable. The storage cost per TB includes not only the purchase price but also the lifetime of the media, e.g. a HDD model with a warranty of 2 years cannot be trusted to store data for a longer time. Tape is guaranteed for 30 years, but the storage time is normally reduced to about 10 years by the risk that the corresponding tape drives may become obsolete.

klodolph 3y ago
Yeah. It gets even more complicated, because tape drives and tape media has different failure rates. At the places where I used tape, we used a more efficient encoding for archival tape backups. Rather than storing 2 or 3 copies, we used forward error correction with something like 30% overhead. This, then, gets even more complicated to evaluate, because it multiplies how “hungry” your backup / archive system is for ingesting data in order to remain efficient & still write data out to tape by whatever deadlines you have set. If you store 3 copies of data on LTO-8, then you write data in blocks of 12TB, with one copy on three tapes. If you use forward error correction, you might do something like write out 96TB at once on 11 tapes. You use less than half as many tapes in the long run, but you need to feed the machine faster in order to meet deadlines.
- adrian_b 3y ago
  While it is true that more complex error correction methods can reduce a lot the amount of tapes needed for storage, the simple approach of making 2 or 3 copies remains necessary in many cases, because storing the copies in different geographical locations is typically the only method that can protect an archive against a calamity that may destroy an entire storage center. It is possible to use RAID-5/RAID-6-like encoding schemes that can survive the complete destruction of 1 or even 2 storage centers while using less tapes than with simple copies, but such encoding schemes can be used only by very large organizations, which use more than 3 separate storage centers.
  - klodolph 3y ago
    Yes, the needs for archives (sole copy) and backups (additional copies of live data) are different. > It is possible to use RAID-5/RAID-6-like encoding schemes that can survive the complete destruction of 1 or even 2 storage centers while using less tapes than with simple copies, but such encoding schemes can be used only by very large organizations, which use more than 3 separate storage centers. Scenarios where two storage centers are destroyed—that’s extreme. The most paranoid scenarios I’m normally willing to entertain are along the lines of one data center burns to the ground in a generator fire, and somebody drives a truck full of backup tapes into a ditch and they’re all covered with mud and sand. Tapes have a high enough failure rate that you benefit from forward error correction and you benefit from planning to handle individual tape failures. This includes stuff like the tape leader breaking, somebody losing a tape, damage during transport, water damage in storage, etc. There’s a layered approach here, where you plan for different disasters at different levels of the stack. Each layer exposes some certain failure rate to the layer above it, and deals with some certain failure rate at the layer below it. When I think of backups, I often imagine a top-level data storage system that has a geographically distributed hot backup, and then an offline cold backup. This lets you survive complete destruction of one data center, or lets you survive a catastrophic software bug that destroys data (and a bunch of tapes are damaged on top of that). Pretty good baseline, IMO.

mauvehaus 3y ago

Another big case besides SOX compliance is medical records generally or imagery specifically. Basically anywhere where you have a lot of data that has to be retained indefinitely for regulatory compliance or practical reasons is a great case for tape. But yeah, the robotic library and the service costs are pretty high until you hit a huge amount of data. Source: supported a couple of large medical installations for a couple years many years back. Can confirm that you're dealing with a lot of mechanical complexity per GB until you get to an absolutely enormous amount of data. I genuinely can't imagine breaking even with a robotic library you can't climb into.