10 ms·
The future of data storage is still magnetic tape (2018)
- Bluestein 5y ago"It should come as no surprise that recent advances in big-data analytics and artificial intelligence have created strong incentives for enterprises to amass information about every measurable aspect of their businesses. And financial regulations now require organizations to keep records for much longer periods than they had to in the past. So companies and institutions of all stripes are holding onto more and more. Studies show [PDF] that the amount of data being recorded is increasing at 30 to 40 percent per year. At the same time, the capacity of modern hard drives, which are used to store most of this, is increasing at less than half that rate. Fortunately, much of this information doesn’t need to be accessed instantly. And for such things, magnetic tape is the perfect solution. Seriously? Tape? The very idea may evoke images of reels rotating fitfully next to a bulky mainframe in an old movie like Desk Set or Dr. Strangelove. So, a quick reality check: Tape has never gone away!"
- Bluestein 5y agoI mean, I really grok tape having detractors. Granted ... ... but, to the point of downvotes? :)
- blakesterz 5y agoThis is from 2018, I'd be curious to hear if anything has changed in the past few years.
- solresol 5y agoAnecdatum: In 2018 I was running a consulting business helping customers with backup to tape, but even back then most customers were doing backup to a dedupe (magnetic disk) library like the HP StoreOnce, but then often copying that spooled-to-disk data to tape one per week or once per month. Now (2021) the flavour-of-the-month is backing up to cloud storage (s3 or Azure blobstore), sometimes directly, sometimes as a copy from deduplication storage. This morning's email summary tells me that one of my customers has just recently stopped using their on-site tape library; I have a quote out for some work with another customer to decommission theirs and replace it with writing to a cloud-hosted provider. So tape is still getting used (assuming that's what Deep Glacier actually is), but it isn't owned by the customer. But if you are backing up to Google's blob storage, then no, it isn't tape, it's just magnetic storage again, just in low-speed access sections of disk. The product I was working (DataProtector) has gone through a bit of churn as it changed hands from HP to Microfocus, so this could cloud the numbers a bit.
- Kuinox 5y agoMagnetic tape have simply a far bigger working area than HDDs. An HDD have only it's disk area, multiplied by the number of disks. Magnetic tape is wrapped which make the total surface way way higher.
- Dylan16807 5y agoNot "simply" when the densities are so different. They both happen to be similar in price, but if the density gap grew by 3x then suddenly tape would be more expensive and only used for certain types of archival.
- Kuinox 5y agoThe areal density of tape is far lower, but the volume density is higher, simply because HDDs plays only on 2D planes while tapes are wrapped. If the density gap grew, magnetics tapes will try to put more tape in a single enclosure, like HDDs try to put more disk in a single enclosure.
- Dylan16807 5y ago> the volume density is higher Not really. If you compare the current best tapes and hard drives, the hard drives store exactly the same amount per cubic centimeter. If you look at just the 10cm by 10cm part with the platters, and compare it to the 10cm by 10cm tape cartridge, the hard drive is fitting more data into less space. Once LTO-9 is available then it will win the first comparison but it will be just about tied for the second comparison. > If the density gap grew, magnetics tapes will try to put more tape in a single enclosure, like HDDs try to put more disk in a single enclosure. You can only fit so much more tape in the same box. They've already been increasing the tape length as much as is reasonable, but from LTO-3 to LTO-9 it's less than a 50% increase. Much like hard drives have managed to fit in a couple extra platters.
- adrian_b 5y agoWhat really counts is the volume density for the entire device, i.e. for a LTO tape cartridge vs. a 3.5" HDD. Comparing a 12 TB tape with an 18 TB 3.5" HDD, the volume of the cartridge is about a half or less, so the tape still has better density. Moreover the tape cartridge is many times lighter, if you carry in your pocket a tape cartridge vs a 3.5" HDD, the difference is very significant. You would not notice the tape cartridge, while the HDD would be like having a stone in your pocket. Experimental tapes have already been demonstrated at volumic densities much better than what HDDs hope to achieve in the future.
- dang 5y agoDiscussed at the time (of the article, not the future): Why the Future of Data Storage Is Still Magnetic Tape - https://news.ycombinator.com/item?id=17864392 https://news.ycombinator.com/item?id=17864392 - Aug 2018 (155 comments)
- lizknope 5y agoThe article briefly mentions HAMR (Heat Assisted Magnetic Recording) hard drives. Seagate has been shipping them since 2020 to some customers. Seagate's roadmap is 120TB drives in 2030. https://www.anandtech.com/show/16544/seagates-roadmap-120-tb-hdds https://www.anandtech.com/show/16544/seagates-roadmap-120-tb... If you are a business then tape is great. But as a consumer with around 50TB of data the problem is that current generation LTO tape drives are in the over $3,000 range. The tapes are reasonably priced but the drive cost is so high that you can buy about 200TB of hard drives for the cost of just the tape drive.
- tzs 5y agoI wish tape drives weren't so expensive. I'd love to have on each personal computer a backup system that just mirrors every disk write to tape. According to the SMART data for my current system I write about 9 TB per year. That would fit on one LTO 8 cartridge.
- klodolph 5y agoWhen I last calculated the break-even point for a thrifty home user, it was around 50TB of data. Above 50TB and there was a decent chance that tape is cheaper than disk, below 50TB it's unlikely. I'll just say that the exact tradeoff will depend on a lot of factors, it's just as a rule of thumb, you can forget about tape below about 50TB. The media itself is cheap and durable, but the drives are $1,000 or more and sometimes break--in other words, if you really must have your data, you want two drives.
- userbinator 5y agoin other words, if you really must have your data, you want two drives. On the bright side, since it's removable media, anyone with a working drive can read your tapes for you.
- ineedasername 5y agoDepends on the needs. Tape will keep longer unpowered. A traditional HDD is a bit riskier if left offline and unpowered for a while. Even if they're all powered in a tower, I'd still recommend something like Backblaze for redundancy.
- klodolph 5y agoIf you're storing 50TB of data in B2, that's $3k/year. For that price, you can make 2x copies of your data on tape, and then buy two tape drives to read them.
- ineedasername 5y agoB2 is an upgrade option. You can still store an unlimited amount on their standard plan, or upgrade for a little bit more for access to a year's worth of changes. It would be worth it to keep a separate desktop with a dozen drives connected and pay for a second annual subscription if you have that much data. The one-year retention plan adds $24/year. Even the "forever" retention plan for 50TB would only add $250/year. If we're talking about home use, B2 really isn't necessary. And if you want a daily rotation every day for a year the tape costs can add up, depending on how much data Cha he's day to day. For home use, an HDD backup combined with something like Backblaze seems like a pretty reasonable cost for two layers of redundancy and a year's worth of file versions changes.
- axx 5y agoKind of semi related: What do you buy if you want to do tape backups at home?
- scheme271 5y agoProbably a LTO drive a few generations old and a SAS card. Current generations cost too much and you aren't going to be able to keep up with the data rate that they prefer.
- Dylan16807 5y ago> Current generations cost too much and you aren't going to be able to keep up with the data rate that they prefer. Maybe. Keeping up only really matters if your slowest data rate is going to be between 50MB/s and 120MB/s. If it's below that then you have a strong need for a buffer drive no matter what tape generation. And if it's above that then you shouldn't have any real trouble no matter what tape generation.
- aduitsis 5y agoVery insightful comment on the data rate, if you can't feed the LTO properly, it may be stopping and starting all the time, which could have a detrimental effect to its mechanism. But this is probably easily fixed by spooling (using bacula terminology). You buffer like 100Gbytes, write them in one batch. Plus, some drives support variable speeds. Also, in addition to SAS, a fibre channel card would also fit the bill, albeit probably a trifle more expensive. But if you go for low speeds (e.g. 8Gbps), those cards can be less than 100$.
- adrian_b 5y agoYes, that is how it should be done. For example, I collect all the files that I send to backup in archive files whose size is approximately 50 GB, then I copy to the tape the 50 GB files. Doing like this, the write speed to a LTO-7 drive is always constant at 300 MB per second, which is the maximum possible for that standard (which is much higher than what a HDD can sustain, so the 50 GB files must come from a fast SSD or from a RAM disk).
- dabiged 5y agoI work with a lot of tape in the oil and gas sector and one of the major issues we see is that industry wide file formats for data exchange are still based on old tapes. These formats have fixed block sizes that haven't changed since the 1950's. Modern tapes have exponentially larger capacities and bandwidth but due to the use of these file formats we rarely see 20% of the theoretical throughput on a brand new tape drive and cartridge. The industry has no desire to change the file format so we are stuck in this situation where tape read/write times grow exponentially with every new tape generation.
- kstrauser 5y agoCan you give an example? What could change if we stopped optimizing for legacy stuff?
- dabiged 5y agoThe main issue is that the standard was written back when file formats and tape formats were essentially the same thing. Data was read from tape, processed and written to a second tape. The memory of machine at the time were of the order of 1 block. Decoupling tape formats from file format effectively resolves the issue, that is: read the tape in 6k blocks, write the file to a modern file system. Set the tape block size to 10mb. Write the file back to tape as a tar. You get the bandwidth limit of the tape drive. The issue is almost all legacy industry software is still designed to read and write directly to tape objects so your tape isn't readable in all the software this data is designed for.
- skissane 5y agoCouldn’t this be addressed with virtual tape devices? The application gets presented with something that is indistinguishable in behaviour from a physical tape drive, but actually backed by a file on SSD or hard disk. Then you can copy that file to a real tape device separately
- newswasboring 5y ago
- WalterBright 5y agoThe other advantage of tape is that ransomware shouldn't be able to encrypt your backups (if written in append mode).
- selfhoster11 5y agoThis is what gets me about read-write online backup media. How do you ensure it's not compromised by ransomware? How do you ensure that your snapshots aren't compromised? It boggles the mind to to think that defense in depth and software write protection is considered fine these days.
- WalterBright 5y agoI regularly bring up the subject of physical write-enable switches (that hard drives used to have). Inevitably, someone responds that it's a great idea, and their software has a write-enable setting. The other mind-boggling thing is I argue that remote updating shouldn't be allowed unless there's a physical switch to enable it. The response is that remote updating is necessary to keep systems secure from malicious remote updates. I have a paper sleeve that goes over my webcam when not in use.
- deepstack 5y agoFloppy days! Way back then I remember there are bootable linux or freebsd server running off a DVD disk. The idea is that the server data then is immutable.
- deleted 5y ago[deleted]
- mhh__ 5y agoI would probably pay extra for a switch although a big heavy mechanical switch in the middle of the signal path for a modern SSD would probably not be as cheap as you might expect.
- 8bitsrule 5y agoPeople are still working to use lasers to write data much faster. TU/e paper from a year ago: [https://phys.org/news/2020-07-ultra-fast-laser-based-storage-devices.html https://phys.org/news/2020-07-ultra-fast-laser-based-storage...] "Ultra-fast laser-based writing of data to storage devices" "researchers ...have demonstrated a new approach that can achieve deterministic single pulse writing in magnetic storage materials, making the writing process much more accurate." (on a 3-layer, 15nm surface) Paper: [https://www.nature.com/articles/s41467-020-17676-6 https://www.nature.com/articles/s41467-020-17676-6] "Deterministic all-optical magnetization writing facilitated by non-local transfer of spin angular momentum" Still writing to 2D. I guess laser-write/read to 3D cubes is still a pipe dream.
- luma 5y ago3D may be closer than you’d think! https://www.microsoft.com/en-us/research/blog/in-search-for-future-of-cloud-storage-researchers-look-to-holographic-storage-solutions/ https://www.microsoft.com/en-us/research/blog/in-search-for-...
- 8bitsrule 5y agoOK, cool. My browser's blind to MS, but I found an alternative: https://techxplore.com/news/2020-09-microsoft-holographic-solutions-cloud-storage.html https://techxplore.com/news/2020-09-microsoft-holographic-so...
- chunkyks 5y agoWhere I work we moved to a fancy backup solution a few years ago. Which one doesn't matter but suffice to say the name implies the focus isn't on metal Not a single one of our metal machines ever successfully backed up with it. Tens of TBs per system seems to be where it just fails every single time. We put those systems back on the official-retiring tape system and now all it stuff is backed up without a hitch every time.
- williesleg 5y agoTape is great because once you write it and forget it, you think the data is safe. Until you try to read it.
- guerby 5y agoProxmox Backup Server has LTO tape support now: https://pbs.proxmox.com/docs/tape-backup.html https://pbs.proxmox.com/docs/tape-backup.html IIRC they rewrote the tape driver in userspace in rust.
- avhception 5y agoHuh. I glossed over the article but found no explanation for why that was necessary. Do you happen to know by any chance?
- kwdc 5y agoI feel like there's a place for home users to rent a tape drive, write everything to tape and return the drive. Uber for tape-based storage. You don't need to own the drive. Just the tapes. Maybe not even the tapes. Dropbox using tape storage? Maybe. Personally I'd like to own the actual medium. Ycombinator! Make it happen.
- dabiged 5y agoJust use a cloud provider. If you are worried about privacy encrypt it. For example, AWS Glacier deep archive would be years of storage for the cost of shipping a tape drive. If you have bandwidth issues use snowcones.
- bb101 5y agoA competing but emerging technology to keep an eye on is holographic storage. "IBM has already demonstrated the possibility of holding 1 TB of data in a crystal the size of a sugar cube and of data access rates of one trillion bits per second"[1] [1] https://en.wikipedia.org/wiki/Holographic_Data_Storage_System https://en.wikipedia.org/wiki/Holographic_Data_Storage_Syste...
- rdines 5y agoThe author's bio "Mark Lantz is manager of the Advanced Tape Technologies at IBM Research Zurich." LOL, ok sure man, the future of storage is definitely tape. :rolls eyes:
- helloworld11 5y agoThree questions as one who is mostly ignorant of magnetic tape: 1. How expensive is this medium per unit of memory (for example, per terabyte)? 2. How well (practical effectiveness) can it connect to home devices and be used to store information such as photos, texts, documents, videos etc? Related sub-question: how well does its lasting power compare to HDD? 3. Where/how would one buy reliable magnetic tape storage devices? In case anyone who knows the subject well wouldn't mind answering a bit.