14 ms·
Why the Future of Data Storage Is Still Magnetic Tape
- nishantvyas 8y agoWhile, Intel Introduces "Ruler" Server SSD Form-Factor. "Intel on Tuesday introduced its new form-factor for server-class SSDs. The new "ruler" design is based on the in-development Enterprise & Datacenter Storage Form Factor (EDSFF), and is intended to enable server makers to install up to 1 PB of storage into 1U machines while supporting all enterprise-grade features." ref: https://www.anandtech.com/show/11702/intel-introduces-new-ruler-ssd-for-servers https://www.anandtech.com/show/11702/intel-introduces-new-ru...
- sliken 8y agoSeems kinda odd to mention uber expensive flash when talking about, cheaper than spinning disk, tape.
- paavoova 8y agoBut for how many years is unpowered data integrity guaranteed for modern enterprise flash storage? Magnetic media excels at archival. In fact, plain old paper does, too, provided it's stored properly. I bring this up only because storage doesn't necessarily require on-demand retrieval, and so flash storage may not be much use in those cases.
- sliken 8y agoDunno, had a room with 120 tapes and 100 spinning disks or so. The room hit 110F or so during an AC failure. 120 tapes died, zero disks. Actually I tried only 10 or so, before giving up. Tapes do seem much more sensitive, the storage temperature (non-operating) was the same as the operating temp of the disks. Don't tapes have issues with print through and adhesion if not regularly used? I seem to recall best practices including something like annual seek to the end and seek to the beginning. Seems pretty similar to spinning disks. I'm less sure of flash drives on a shelf.
- jjeaff 8y agoThis seems to indicate that these tapes are safe up to 120c for short periods of time. But that they should be archived at no more than 77. https://www.ibm.com/support/knowledgecenter/en/STCMML8/com.ibm.storage.ts3500.doc/ipg_3584_meesi.html https://www.ibm.com/support/knowledgecenter/en/STCMML8/com.i...
- zaarn 8y agoFlash drives guarantee data integrity for up to a year (JEDEC), so you'd have to spin them up and read data, though I'm unsure if this is sufficient on flash.
- zaarn 8y agoFor the price of a 1TB SSD I can get you a 12TB/30TB Tape, easily between 10 or 30 times as much storage. And as a bonus, while a SSD is only guaranteed to retain data without power for a year, tapes are designed for decades of cold storage. Additionally, you could put up to 2x4x9 (HxWxD) tapes in there, at max capacity that is about 864TB/2.16PB The only downside is you need another 1U to read the tapes, though the rest is automatic if you have a robotic tape library.
- nishantvyas 8y agoWell my point was not to show flash as superior but to point out the development in that area... every use-case has reason be it flash or Magnetic... author choose to paint Magnetic as better alternative to Flash (in capacity dimension), which is obviously can not be true for all the use-cases... (same for flash) my comment it to high-light that fact...
- mmt 8y ago> author choose to paint Magnetic as better alternative to Flash (in capacity dimension) I'm having trouble finding where in the article that occurred. To me, it seemed to ignore flash/SSD, as the principle premise was that tape survives due to its low cost (at scale), so only the highest-density HDDs were mentioned.
- CharlesW 8y ago> Mark Lantz is manager of the Advanced Tape Technologies at IBM Research Zurich Seems a bit odd to see such a blatant P.R. piece in IEEE Spectrum. Wasn't optical going to eat rust-based cold data storage technologies? What happened?
- sliken 8y agoSoftware got better. Compression, deduplication, pure incremental backups (single level zero) , reverse diffs, etc all make random access read/write backups cheaper and more attractive. I've heard it speculated that glacier's slow restore times is consistent with a large number of optical disks and robots for loading. Never heard it substantiated though.
- wmf 8y agoOptical discs are now up to 300 GB and they have caddies the size of an LTO tape that hold 12 discs. Seek time is obviously better than tape but I don't know about the price. The only remaining vendors are/were Sony and Panasonic. https://panasonic.net/cns/archiver/ https://panasonic.net/cns/archiver/ https://perspectives.mvdirona.com/2016/03/everspan-optical-cold-stroage/ https://perspectives.mvdirona.com/2016/03/everspan-optical-c... (Sony Everspan dropped off the Internet; did they go out of business?)
- jl6 8y agoArchival Disc doesn’t appear to be available to buy yet.
- Jaruzel 8y agoFrom a cheaper end-user/consumer point of view, the only real option for optical disc storage is BD-XL, which is a 3 or 4 layer Writable BluRay Disc supporting 100GB/128GB per disc. In comparison to tape, it's not that much storage, but fairly cheap to implement at home. £60 for the drives, and about £15 per 100GB disc.
- Rafuino 8y agoWhile there's an explosion of data that we keep hearing about, the aspect I'm curious about is how much of that data will at any one point be "hot" or even "warm," in that it needs to be processed and analyzed in short order (talking microseconds or nanoseconds, not in tens of seconds like tape (or more as the data then needs to be sent over a network to a server somewhere else)). Cold storage can be archived on tape or HDDs, sure, but as compute continues to grow, surely more and more data will need to be hot or warm at any given moment. Just whether that hot/warm data will grow its overall share of data in existence or simply grow at the same rate is what I wonder about.
- CaliforniaKarl 8y agoSo, first I would quibble a bit with the time scales you're talking about, as tape access could easily be more than tens of seconds, depending on status: Even a loaded tape might take a minute to reposition, if the data you want is at the opposite end. In the world I support, everything has to be funded by something. We have our "cheap and deep" storage, for example (http://oak-storage.stanford.edu http://oak-storage.stanford.edu), but labs who use it have to pay for it. Unchecked growth is highly unlikely, as grants run out eventually, and then the data will have to go somewhere else. Often these days, that ends up being Google Drive, but there is demand for tape archival storage. Even the Stanford Digital Repository (https://library.stanford.edu/research/stanford-digital-repository https://library.stanford.edu/research/stanford-digital-repos...), who are extremely serious when they talk about "long-term", say they're developing pricing models.
- Rafuino 8y agoWell, I was keying off the article itself, which said the following: The long length of the tape held in a cartridge—normally hundreds of meters—results in average data-access times of 50 to 60 seconds It's tens of seconds, up to a minute, but that's just average, so I think we're on the same page here! Even when not in academia, everything has to be funded by someone at some point. Sounds like you have a pretty awesome job though, and I imagine Stanford is one of the least likely to get cut off from precious NSF grants!
- 8y ago
- gHosts 8y agoAhh, the old old old saying, "Never underestimate the bandwidth of a stationwagon full of tapes" still holds true.
- samschooler 8y agoThis actually seems like a fun math calculation... Has anyone done this before?
- jmpman 8y agoIf I remember right, it was a problem in Tanenbaum’s book.
- simonebrunozzi 8y agoCorrect.
- scrooched_moose 8y agoXKCD What If #31 is a good one. https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
- zaarn 8y agoSince we have 512GB µSDs the milk bottle would go from 1.6PB to 12.8PB, all other results need to be multiplied by the 8 times increase in storage density...
- penagwin 8y agoCould easily stack dozens of milk bottles in most SUVs/Trucks. I have a new business venture guys!
- zaarn 8y agoNot really a new business venture, sneakernets have existed for a long time. On Cube having someone check by your house with a few terabytes of internet to copy to your disk is a thing. And companies regularly have interns carry harddrives between servers or even have someone drive it between cities because it's simply cheaper to pay for gas and driver than to pay the transfer over wire. And usually faster too.
- Aaron1011 8y ago> And tape is very secure, with built-in, on-the-fly encryption What exactly is this referring to?
- raziel2701 8y agoI can't seem to find this information with my google-fu, but does anyone know if magnetic tapes such as in vhs and cassette tapes have perpendicular magnetic anisotropy?
- reaperducer 8y agoI’m not entirely sure what it is that you’re asking. But if you’re asking if the heads for an audio cassette are angled, the answer is no. They’re perfectly perpendicular. For VHS, yes, they are angled. 30 degrees, IIRC. If that’s, indeed, what you’re asking.
- raziel2701 8y agoI was referring to whether the magnetization lies along the normal of the film/tape or if it's in plane. HDDs were able to increase density by moving to materials that had a magnetization with perpendicular anisotropy but I can't seem to find confirmation if cassette tapes and vhs had in plane anisotropy back then. More so I can't seem to find if today's magnetic tape storage has moved to perpendicular magnetic anisotropy. It seems
- gammatrigono 8y agoDoesn't seem like it would be possible given the substrate material.
- coding123 8y agoSo... who actually uses them in the age of AWS / S3. Is this becoming a much much much smaller group?
- SteveNuts 8y agoDon't forget about glacier.
- Phlarp 8y agoI thought the consensus on glacier was that it was optical backed storage? Has that changed?
- wmf 8y agoI would say there has never been a consensus. The Glacier API is defined such that it could be backed by any technology. It's possible that early "MVP" versions of Glacier were just S3 with lower pricing and artificial delays. Once demand was established, they could have swapped out the backend with MAID, tape, optical, or some combination. There are a lot of comments out there saying "everyone knows Glacier is X" but nobody says "I know Glacier is X".
- gwbas1c 8y agohttps://aws.amazon.com/glacier/ https://aws.amazon.com/glacier/ Looks like tape as a service.
- sneak 8y agoGlacier’s actual storage backend technology has never been disclosed, although this is plausible. Other proposed technologies are racks of older (less efficient) disks that are not always spun up, huge slow flash memory banks, et c.
- sliken 8y agoSome past hacker news thread made a plausible case for 12" optical disks, much like bluray.
- sytelus 8y agoTLDR; Today, a modern tape cartridge can hold 15 terabytes. And a single robotic tape library can contain up to 278 petabytes of data.
- sliken 8y agoSeems pretty terrible considering that a tape has crazy more surface area (100 to 1000s of feet x 1" or so) and a 14TB disk is just a few square inches of usable area (2 inch^2 or so per platter).
- scottLobster 8y agoNot if you consider that the tape can be spooled, elimating any concerns about physical volume, and data spread over a larger area means potential corruption is more distributed. You also don't need to fill the tape cases with helium.
- sliken 8y agoLTO-8 is 12TB native, 960 meters x 12mm is 11.52 meters square! That's 17,856 square inches. Somewhere around 6000 mbit per square inch. Current 14TB disks are around 1000 gbit per square inch or 166 times denser. Sure you need a motor, seek head, etc. All about the size of a tape. But you don't need a robot or tape vault either. Seems unclear where the cross over for tape vs disk is. Especially when you use things like a backblaze storage pod and spin down disks when not needed to avoid the heat, power, and wear and tear. Does make one wonder why not disk robots? The disks are the same size, support random access, and can be hot swapped. I dug around and couldn't find a price on a 15TB tape, or even what it was called.
- smallnamespace 8y ago> Does make one wonder why not disk robots? The disks are the same size, support random access, and can be hot swapped. For long term storage, it's unlikely that disks will last as long as tapes. All the electronic and mechanical components of a disk drive must still work in order for you to be able to retrieve data, and it's somewhat likely that if you just stick a drive on a shelf for ten years some percent won't be working afterwards. For tapes, the separation of electronics from the actual storage medium itself means better maintainability; you can rpelace the tape drive without touching the cartridges.
- ggm 8y agoIBM bubble memory
- sqldba 8y agoThe problem is it's near impossible to get a tape backup system at home. It's all fun to say "oh a single tape can do 15TB" but it must cost in the thousands and thousands of dollars.
- CaliforniaKarl 8y agoI think that's because the market has shrunk so much. I remember when I was younger, I got a single SCSI external tape drive, and some tapes, from the Micro Center in Cincinnati Ohio. I then used that to back up my computer while I was in college. I've forgotten so many of the details, like the model of drive and tape, and the backup software (it was Mac OS). I do know it wasn't LTO, though, because it had to be rewound. These days, were I to do something like that, I'd do BD-R. Yeah, you're only talking 50 GB or so (uncompressed) per disc, but that's not bad!
- mrpippy 8y agoProbably Retrospect for the backup software, it was the standard on Mac OS. I feel like 8mm was probably the most common consumer (i.e. cheap) tape format, but also could’ve been 4mm.
- CaliforniaKarl 8y agoYup, it was Retrospect!
- jabl 8y agoAs a kid, we had one of those home/small business targeted drives that attached to the PC via the floppy drive controller. QIC, or whatever it was called. Native capacity was ~120 MB, and the drive had some built-in compression so it was marketed as 250 MB. It made a horrible screeching noise when running. Linux even had a driver for it at some point (ftape), but IIRC it has since been removed due to lack of use. Never used it though, by the time I got more into Linux that drive was already obsolete. But yeah, today the startup cost of a tape drive is so high that it doesn't make sense for home usage. At home I use borg backup (https://www.borgbackup.org/ https://www.borgbackup.org/ ) nowadays, backing up to external USB hard drives.
- TheAceOfHearts 8y agoIt would be great if magnetic tape were more accessible for consumers. A while back I was reviewing local backup and storage options, and I read up a bit on it. I think it scales better when you have massive amounts of data, but the initial price is fairly steep. At the consumer level it makes way more sense to buy HDDs. My suggestion would be to pick up something like the EasyStore 8TB external HDD [0] (on sale right now at Best Buy for $160, will probably drop again soon). You can open it up, extract the drive, and drop it into your computer if you want internal storage. If you want to reduce the risk of data loss you can also fill one up with an encrypted snapshot of your important things and ship it to a family member. Very important if you live somewhere that's at risk of being destroyed by natural disasters. [0] https://www.bestbuy.com/site/wd-easystore-8tb-external-usb-3-0-hard-drive-black/5792401.p https://www.bestbuy.com/site/wd-easystore-8tb-external-usb-3...
- _iyig 8y agoI’ve been happy with M-Disk DVDs for durable, disaster-proof, write-once backups. Gets a little pricey when you start talking about TB, but for family photos and important documents it more than meets my needs.
- cptskippy 8y agoI'm not familiar with M-Disks but I swore off optical media for long term storage around 2004 when the discs in the binder I use to carry around started turning up empty. Several of them were those Kodak Gold Archival CDRs that were supposed to last a decade but were dead inside 3 years.
- jacobush 8y agoThank you, had no idea. I will consider this for photos and such.
- Johnny555 8y agoFor the majority of consumers, cloud backup makes more sense than tape backup -- few people have the discipline to stick to a safe backup regimen.
- ed_blackburn 8y agoGDPR - please delete me from all your systems. JFC. Harry get the key to the basement, need to dig out some tapes. San, can you pop down to the store and get some matches and lighter fluid...
- supertrope 8y agoI don't know how GDPR handles backups and what time tables are allowed for reasonable data retrieval and expunging. Encryption decreases the volume of data that needs to be most carefully protected from the entire data set down to the key (and backups of said key). I imagine that data on mediums that make data retrieval and writing expensive can be encrypted with compartmentalized keys. So when Jane Doe wants her data deleted, delete the keys.
- ed_blackburn 8y agoIt's another reason why I think the cost of managing keys appropriately is worth it for crypto-shredding.
- spiralx 8y ago> I don't know how GDPR handles backups and what time tables are allowed for reasonable data retrieval and expunging. Keep a log of user ids who have requested deletion and purge their data whenever doing a restore. And don't keep backups for longer than required, which I think is generally something like five to seven years barring specific legal requirements.
- Osmanthus 8y agoWhat about scratches
- oliwarner 8y agoI was thinking —like other top-level comments here— that it's a great shame that consumer level drives have died out. They're thousands of dollars and all wired for enterprise interfaces. But I'm really surprised that nobody[1] is offering "tape backup as a service". Pay $30 per 3TB cassette (slightly over market rate), a $10 loading fee and then $1/h for 100MB/s write access. When you're done, pay another $10 and it's shipped to you. Or $5/month for fireproof storage. This might have to be a pre-booked service, or they could buffer upload onto spinning disks and write to tape after the fact (much faster), but for storing more than 1TB, this method could be vastly cheaper. [1]: That one quick Google could find.
- kqr 8y agoHow would this, in effect, be different from any serious backup service in existence today? I'm sure they do their backups of your backups on tape, and at least rsync.net is willing to ship physical media with your data to you.
- oliwarner 8y agoPrice! In the simplest version of my post, you're paying for media, the time for somebody to pop it in the machine, the hire of that machine while you copy data to it over the network, and the postage of a tiny little LTO back to you for archival. On higher density tape, you could be looking at a one-off $100 total for writing 10TB of data to tape and mailing it back. There's a lot of room between that price and the nearest incumbent to make a strong profit there. rsync.net seems very bespoke, so they may be able to give you a tape, but you're looking at $250 per "incident" plus hardware. They waive that fee if you store more than 100TB ($2k/month). Yeah, they're much more available but most of us aren't after cloud storage, we want a lasting backup in case the NAS dies, or the house burns down. Restore speed doesn't matter.
- kqr 8y agoAh, true. I didn't consider the "one time dump of a boatload of data" as a use case. My backups tend to run relatively often, and at that point, $10 for each tape load would easily make the tape service more expensive!
- peterburkimsher 8y agoMy dad's PhD thesis is stored on a magnetic tape formatted for a PDP-11. He'd like to copy it off, but it's difficult to find a computer that can read it! Back in 2006 we went from Geneva to Lausanne to visit Musée Bolo, but their PDP-11 had a problem with its Winchester drive (hard disk) and wouldn't boot. After a few hours, we gave up. Please get in touch if you have the tools - I know he'd like his digital copy back! Even more modern formats are difficult to copy forwards. I'm currently tidying out my old computer collection, copying off data from SCSI disks, 5.25" floppies, DD floppies, HD floppies, and CDs. My USB floppy drive can't read DD floppies. The SCSI path requires copying to a PowerBook G3, removing the internal hard drive, putting it into a USB-PATA controller, and copying it to my laptop. I'll need to do the same for all the 2GB JAZ cartridges. I don't even have a ZIP disc reader, so I gave those two discs away to another collector so he can read them for me. The 5.25" floppies are the hardest - they don't mount even on old Macs, so I'll need to figure out how to set up ADTPro to copy them over serial from an Apple IIgs. If you're in the area and want some old hardware/software/magazines/user manuals, please let me know! https://www.reddit.com/r/VintageApple/comments/99223h/peters_den_geneva_switzerland/ https://www.reddit.com/r/VintageApple/comments/99223h/peters... The lesson? If you want to keep the data, also keep the computer that can read it. And make backups.
- Aaargh20318 8y ago> The lesson? If you want to keep the data, also keep the computer that can read it. And make backups. Or just copy the data over the moment you get a new computer, instead of waiting for several decades. As long as you keep moving the data from generation N to generation N+1, there's no problem.
- dghughes 8y agoNot quite. But possible. I have two 1TB external hard drives. The data was added over a few years. But now to copy 1TB over USB 2.0 is beyond painful. Fortunately I can just disassemble the drives and connect them directly. But that's if I have an old system with IDE controllers. If I can find an IDE to USB converter I'd be back to square one (unless it is USB3). And I don't know if there is such a thing as an IDE to SATA converter.
- ksec 8y agoIt is suggesting there are research that shows it is possible to have 20x times the density of current types, which tops at 15TB. But these research are like battery breakthrough, good in theory but not thought through in practice. While we have a real roadmap, and a working solution of MAMR, much more so than the proposed HAMR from Seagate, that will scale us to 40TB per HDD by 2025, and may be more so given we don't really know the limits of MAMR yet, and we manage to put more platter inside a helium drives. Which I don't think is possible with HAMR, laser heating plate with helium filled drive has got to fun inside labs. NAND Flash is also dropping in price, the current NAND spot price are already backed to 2016 price or may be even slightly lower. We have Fabs from Toshiba, Samsung and SK Hynix all coming online in 2019, all while 3D NAND are achieving better yields. The "Ruler" that Intel proposed has now standardises and becomes EDSFF, with up to 1PB storage in 1U. All current roadmap suggest we should hit 8PB or 16PB in 1U by 2025. In terms of 48 Drive in 4U server, HDD offers only 480TB per 1U by 2025. That is 16 to 32 times the density difference comparing to NAND. In a system where information is stored in different drives like black blaze[1] , your reliability reaches a point ( I think, forgive me if I am wrong ) no different to those offered by Magnetic Type. What are the advantage left for using magnetic type? When cost, performance, space, reliability does no longer flavours them? [1] https://www.backblaze.com/blog/cloud-storage-durability/ https://www.backblaze.com/blog/cloud-storage-durability/
- robdachshund 8y agoTapes can last for 50 years. They are the MOST stable and reliable form of storage we have. We have also not even reached any kind of limit on density. Sony has tapes that hold hundreds of terabytes. So, they last forever, are extremely reliable, and extremely data dense. HDDs exploit the same concept (magnetic storage) but trade speed for reliability. NAND is more reliable but still has a shelf life, and it also relies on support chips to constitute the overall media. It's also expensive as hell. Tape is tape. You could smash a cassette and respool the tape and read it. Its also cheap as hell per gb. Tape will be around for decades if not centuries to come. NAND is not a challenger whatsoever. A key piece of the puzzle here is that not only are tapes used for backup, but to free up space from data center HDDs in order to increase their shelf life. Stop thinking of tape as archaic. Magnetic storage is the foundation of modern technology and computing, and in past decades we barely scratched the surface of its potential. We aren't talking about audio cassettes with analog audio recorded to them. We are talking about a storage medium that is more data dense than HDD or NAND could ever be and we have yet to reach any sort of limit. With tape you can achieve more and more space by manipulating the width and length of the tape, the tape heads, the angle at which tape is read, and the way that the data is read and stored. It's also just a plastic cassette with polyester tape coated with rust particles. It's inherently cheaper, requires no circuitry, power, or motors within the media itself, and will last 50 years. Tape is the purest implementation of magnetic storage concepts. As such it is going to improve faster than HDDs because there are no other design factors like platters, motors, and heads. If an HDD dies, you might be able to recover the data through expensive and time consuming procedures. If your tape drive stops working, you buy another drive and still have an intact tape with all of your data. Additionally, while SSDs are wonderful, they are a pretty backwards design when it comes to efficiency. You are essentially trying to model magnetic storage by using integrated circuits. You are storing values with digital logic in a physical state on a chip. Its fast and more reliable than a spinning metal disk, but it's expensive and not data dense at all. The only threat to tape is public perception of it being "old" when it's really the most robust and best engineered data format we have.