20 ms·

LTO Tape data storage for Linux nerds

293 points by detaro 5y ago

amelius 5y ago
With these prices for drives the market seems ripe for disruption.
- dsr_ 5y ago
  It already is, by spinning disks. Cheaper at the low end, faster the whole way through, random access beats linear access for end user expectations.
  - amelius 5y ago
    You are too much focused on read speed. I just want to write huge amounts of data at a low cost, and don't mind waiting a day for retrieval. I.e., how one normally uses backups.
    - TheCondor 5y ago
      And durability. I’ve had a portable usb hard drive fall over on my desk and it had major problems after that. Solid state fixes that but it’s expensive and I’ve heard they can lose data if not plugged in with regularity
      - lazide 5y ago
        Yeah, SSD is not good for long term storage (like a copy of your tax documents from last year you might need in 5 years). The expense for size also makes it infeasible to copy ongoing roll up copies of everything which is one way of solving that.
    - lazide 5y ago
      Most of the time when people think backups they need faster than 24 hr turn around to restore - because it usually takes about that long to figure out they even need a backup, and most people don’t think ahead enough for 2 day recovery time to be useful for most use cases now a days. If their local snapshots are dead too, or they look for it and realize they can’t find a copy of something they thought they had, it’s often because they needed that data right away and it wasn’t there when they went to get it. Hence ‘user expectations’. That’s not in a catastrophic case (which rarely happens) that’s the ‘bob just realized he deleted the folder containing the key customer presentation last Friday’ or ‘mary just tried to open the contract copy she needed and it’s corrupted’. If it’s a once in 10 or 100 year or whatever event, a 1-2 day turnaround is not unexpected and everything else is probably broken too. The file deleted or something got screwed up happens more often and slow response there grinds things to a halt - and causes a lot of stress knowing it’s not ‘solved’.
      - amelius 5y ago
        I bet most companies who are confronted with ransom demands would die for tape backup even if restoration took a week (which is the amount of time they need anyway to get the whole mess sorted out).
        lazide 5y ago
        For sure - that would fall under catastrophic IMO (odds of any company getting crypto ransomed are low on any given year, though increasing, and can kill the company if there isn’t any prep). A lot of companies are pretty terrible at figuring out catastrophic tail risks like this too. Tape has it’s niche, this is one of them for sure.
  - zozbot234 5y ago
    SMR spinning disks are also being widely repurposed as "archival", somewhat tape-like media since they turned out to be quite low-performance for the most common use scenarios (which means they were getting dropped from soft-RAID arrays, etc.).
    - floatboth 5y ago
      > repurposed Weren't they even originally marketed as such? At least the external enclosures that people like to shuck are usually called something like "my backup".
  - KaiserPro 5y ago
    kinda but not. The problem with spinny disks is that you have allocate space for them. You can't quickly swap out drives to take offsite. Whats grand about tape is that its still faster to dump to your library, eject the magazines and store off site. Whilst you can do that with HDDs (think snowballs but bigger) its a lot more expensive and error prone. Tape serves a purpose, but thats pretty niche by todays standards.
tlamponi 5y ago
Interesting read, as with most of ben's blog.. And yeah, buffering is definitively required to get acceptable speed out of tape tech. If you want a LTO Tape solution with more bells and whistles you could check out Proxmox Backup Server's tape support: https://pbs.proxmox.com/docs/tape-backup.html https://pbs.proxmox.com/docs/tape-backup.html We also rewrote mt and mtx (for robots/changers) in rust, well the relevant parts: https://pbs.proxmox.com/docs/command-syntax.html#pmt https://pbs.proxmox.com/docs/command-syntax.html#pmt https://pbs.proxmox.com/docs/command-syntax.html#pmtx https://pbs.proxmox.com/docs/command-syntax.html#pmtx The introduction/main feature section of the docs contain more info, if you're interested: https://pbs.proxmox.com/docs/introduction.html https://pbs.proxmox.com/docs/introduction.html If you have your non-Linux workload contained in VMs and maybe even already use Proxmox VE for that it's really covering safe and painless self-hosted backup needs. Disclaimer: I work there, but our projects are 100% open source, available under the AGPLv3: https://git.proxmox.com/ https://git.proxmox.com/
- Brian-Puccio 5y ago
  The docs seem to indicate that if you have a directory structure (say over NFS share) you want to backup to tape, you must first back it up to a datastore (read: local storage on the PBS), that you cannot backup the files straight from a directory structure you must first store them locally in the datastore?
  - tlamponi 5y ago
    Yes, normally tape is a second level backup for the worst case and long term archival. Tapes normally are not kept on site, to ensure that even a destruction of the office/datacenter/... like, for example, a fire does not make one lose all data. With PBS this also allows to leverage the whole thing in a more flexible, efficient and powerful manner, as one can do tape-backups async, the deduplication layer can be used so that subsequent backups only need to add the delta of new backup chunks from the content addressable storage (note, the backups themself are still full backups, and one can configure a schedule for when a new media set needs to be created), besides that backing up file-level to tape is normally pretty slow compared to writing out big chunks that are ordered by physical disk location.
Synaesthesia 5y ago
That's a lot of storage! Can't really think of a use for this (200tb plus) personally but it is appealing.
- throw0101a 5y ago
  Tape makes more sense the larger you go, as it help amortize the fixed/upfront costs. The incremental costs of buying more tapes (that are re-usable) isn't that much at scale. It's often relatively cheap insurance against data loss for many organizations. A lot of 'enterprise' backup software is also now coming with hooks into cloud storage (e.g., S3 APIs), but then you have to worry about bandwidth and the time it takes to get the bits offsite at "x" bits/second. Of course you also have to worry about retrieving the data in case of disaster per the Recovery Time Objective: * https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Tim... Also: a backup has not happened until you try and succeed your recovery process.
  - metabagel 5y ago
    > but then you have to worry about bandwidth and the time it takes to get the bits offsite at "x" bits/second. Reminds me of the saying that the fastest throughput is achieved by a 747 full of hard drives. > Also: a backup has not happened until you try and succeed your recovery process. A thousand times this.
- simcop2387 5y ago
  For me it's a lot about just being a data hoarder and never having to delete something because i'm low on storage. About half of my system though is taken up by system backups and virtual machines. I should do a cleanup of those, but the freedom of just being able to spin up something new or put a new backup on there without ever going, "do i have enough space for this?" is rather nice.
  - organsnyder 5y ago
    I also rarely/never delete anything, but my ~2tb NAS still has plenty of room. I guess it makes a difference that the only media I store is my own photos and videos.
einpoklum 5y ago
> LTO Tape is ... much cheaper than hard drives ... a 12TB SATA drive costs around £18.00 per TB ... a LTO-8 tape that has the same capacity costs around £7.40 per TB ... That’s a significant price difference. Actually, it isn't very significant. Price factor of 2.5. I had thought tape storage was cheaper than that. And then there are the drives: A drive to write (3,000 GBP for LTO-8), and at least a couple more drives for reading tapes. At this price ratio, I would say that ease-of-use and safety/robustness of the backed-up material are more important considerations.
- AshamedCaptain 5y ago
  I am also worried for the long-term. If there are new generations so frequently and backwards compatibility is limited or not guaranteed, I ponder if you'd be able to find a working-condition tape reader for your 20-year old tape... At least it's likely I can find a USB port 20 years from now, or a DVD reader (they are still being manufactured today, when even more than 20 years have passed since their introduction, and they are even compatible with much older CDs...).
- ktpsns 5y ago
  What was actually ignored at that comparison is energy costs. Which can get quite somewhere, if you have all your disks running 24/7 and do not use power saving functions (which is frequently turned off in server contexts). Costs are in the ballpark of 5W per drive, given a contemprary 16TB drive this means 0.3W/TB, with 0.25EUR/kWh (a typical consumer price in Germany), this is roughly 0.6 EUR per TB per year. However, probably the replacement costs for these always-on disk drives will be even higher.
  - q3k 5y ago
    Another consideration related to this is that tapes, being usually offline, as much more secured against accidental (or malicious!) erasure when compared to always-on hard drives. Also related is that tapes can easily be transported around/offsite, literally thrown in the back of a truck as they are. Try doing that to hard drives and see how many start throwing bad sectors after a round-trip.
epilys 5y ago
What I never see explained, is what exactly PCI cards should I get to get the full sized SAS drive to work with my desktop PC? Because looking at server component stores, I see there digit prices for SAS controllers, and the author mentions they are cheap.
- c0l0 5y ago
  My advice: On eBay (or any other platform that makes it easy to buy used hardware components), go look for "sas2008 4e", and check out the offers. You should be able to get a decent HBA driven by mpt2sas/mpt3sas for around 40 to 60US$.
c0l0 5y ago
I recently started looking into using an LTO-7 tape drive that I got handed down, along with a few dozens of pristine LOT-6 tapes, for archiving purposes. I got to play around a bit with SAS HBAs, and was kinda shocked how much of a difference that can make in the user (or shall I say sysadmin?) experience: LTO-6 tapes are spec'd to transfer rates of around 150MB/s, so well within the reach of even the the first SAS gear generation. However, the very first SAS HBA with external SFF-8088 connector I managed to get my hands on (an LSI SAS1068e) topped out at a disappointing 80MiB/s, no matter what I tried in terms of blocking and buffering. Switching to a more modern (but still old) LSI SAS2008-based HBA got me close to the theoretical maximum. Then there's the (to me, still open) question of how to best use the actual tape storage capactiy... Since my hardware is newer than LTO-5, LTFS (https://github.com/LinearTapeFileSystem/ltfs https://github.com/LinearTapeFileSystem/ltfs) is an option for convenient access, especially listing tape contents, but that could make it hard for other people down the line to restore data from the tapes I create. It's probably safest to assume that tar will always be there, at least wherever there's tape, too. GNU tar also handles multi-volume/-tape archives, which seems like a necessity if you need to back up amounts of data that exceed a single tape's capacity. Then again, if you want to use encryption with actual tar (important for the kind of data I need to archive), your only option seems to be piping the whole archive through something to compress the stream, which will make accessing individual records in the archive opaque to the drive itself... and you can't just dispose of individual keys to make select parts of the archived data go away for good, either. Also, I would like to conserve as much tape as (conveniently) possible in my archiving adventure. There's "projects" (i.e., top-level directories of directory trees) that consume more than one tape of their own, and then there's smaller projects that you can bin-pack together onto tapes that can fit more than one such project. I've started implementing a small python wrapper around GNU tar to solve a number of these problems by bin-packing projects into "tape slots" and also keeping track of tape-to-file mappings in a small sqlite database, but a workable solution for the encryption problem(s) is not something I managed to come up with yet... If someone has an idea (or better yet, a complete and free implementation of what I am trying to hack together :)), please be so kind and let me know!
- __d 5y ago
  Using a multi-tape archive increases your chance of failing to restore. Rather than having one possible tape that can be corrupt, you know have more than one. Any single failure will corrupt the whole archive, and you've lost (easy access to) the data. Better to break up the source volume somehow, and have each tape self-contained.
Spooky23 5y ago
Backups are a really interesting business. I helped out a colleague a few years ago with a project in a big data center and it was like a whole world that nobody knew existed. Because of the RTOs and backup windows, the supporting infrastructure was fast. The caching layer stuff was the fastest disk in the data center by far, and the team was a small, tight group of people who basically honed their craft by meeting auditor and other requirements. The management left them alone and they did their thing. That was about a decade ago now; those guys have all moved on to really big things.
- StillBored 5y ago
  Its still that way, the netflix guys get a lot of press for their bandwidth numbers but plenty of backup systems were getting similar (or greater) bandwidth numbers years ago, since many of the caching stacks are basically pcie or mem bandwidth limited. The 300MB/sec number the author lists is really slow, and likely appropriate for LTO3/4, (IIRC, the wikipedia numbers are understated) LTO7+ can peak at > 1GB/sec with the modern drives going even faster if the compression is left enabled. So, given a library with a few dozen drives, the bandwidth gets insane. (ex: SL8500)
  - trasz 5y ago
    Someone should ask Spectra Logic folks for their numbers :-) (Spectra Logic's tape libraries run FreeBSD too.)
    - monocasa 5y ago
      SpectraLogic's code isn't in the data plane, you hook up to the drives directly, and the drives can forward changer requests to the internals of the library. So it's however fast the drives are (which are all third party). Also last I checked freebsd was used for their disk product, not tape.
smackeyacky 5y ago
There is one recommendation there I find a bit questionable and thats encryption. If you are out of options and restoring from tape, might be better to have it uncompressed and not encrypted. Its possible, after some physical disaster that you are on somebody elses infrastructure and having some encryption on your data doubles the problems you might have. I use an ancient LTO2 drive for last resort backups that are off cloud and off premises. Its more peace of mind than practical on a daily basis but I did find myself restoring a few files a couple of weeks ago as I had fat fingered an rm command. It was quicker than getting them from S3 glacier.
- benjojo12 5y ago
  I agree somewhat. Encryption is more critical on tape because there is no easy path to wiping a tape, and in a company situation if you need to erase something in your backups too (think GDPR erasure), then encryption is reasonably critical unless you want to go though all of your cold backups. For my archival use (the reason why I got into this in the first place) I do not encrypt nor compress the data going to tape. For server/desktop backups. they are compressed and encrypted.
  - throw0101a 5y ago
    > […] nor compress the data going to tape. Just to note that tape drives have built-in compression that generally is done transparently in the background. So while using something like zstd (per the article) may get more bits on a given tape, there is some compression that one gets "for free" without doing anything at all. * https://en.wikipedia.org/wiki/Linear_Tape-Open#Optional_technology https://en.wikipedia.org/wiki/Linear_Tape-Open#Optional_tech... * https://en.wikipedia.org/wiki/Magnetic_tape_data_storage#Data_compression https://en.wikipedia.org/wiki/Magnetic_tape_data_storage#Dat...
    - benjojo12 5y ago
      I mention this in the post itself
kragen 5y ago
Hey uh is that a DECTape?
- benjojo12 5y ago
  The header image is the insides of a LTO 5 tape
  - kragen 5y ago
    Compare https://en.wikipedia.org/wiki/Digital_Linear_Tape https://en.wikipedia.org/wiki/Digital_Linear_Tape
PaulHoule 5y ago
I notice that he talks alot about dealing with malfunctioning drives and malfunctioning tapes. That is my experience too. There is that time I got kicked out of the computer lab as an undergraduate because I'd created a number of newsgroups and they 'wrote' all my files... to what turned out to be an empty SunTape. That time I tried to recover a configuration file from an IBM tape robot and it took 14 hours. When I was successful with tape I always did a lot of practicing and testing. A sysadmin who taught me a lot (esp. how to get things done in a place where you need 'social engineering' to get things done) told me "you don't have a backup plan until you've tested it" and many people learned that the hard way.
- ansible 5y ago
  > "you don't have a backup plan until you've tested it" Yep. Though that's what makes small-shop disk-to-disk backups easy, depending on the backup software used. We use rsnapshot, which uses rsync and "cp -l" to make backups. So restoring is as easy as using cd to go into the appropriate directory and copying out the files. No special utilities needed. Yes, we encrypt the backup drives using cryptfs / LUKS.
archi42 5y ago
Something not mentioned by the author, but what I was told here on Hacker News some years ago: If your drive has too much wear (or misalignment of the drive head?) you might end up with tapes that you can only read with exactly your drive.
- detaro 5y ago
  That's something I've seen mentioned too but never could verify if that is something that's actually true with modern tape standards or not. (i.e. last I asked on HN I was told it wasn't a concern anymore) If the drive needs to adjust to get precise enough positioning anyhow, misalignment seems way less likely.
  - op00to 5y ago
    It's absolutely true. There is a LOT more to tape storage than meets the eye. Let's say you're using LTO tapes as an archive. Did you know LTO tape itself is abrasive, but that abrasive is meant to wear over time with the intended use of the cartridges, which was backups? If you use new tapes a single time, the abrasive doesn't wear and destroys the tape heads. You will go through a drive head at month, running the drives 24/7. I had a library used as a genomic storage archive with 8 drives (always write, almost never read), and two were constantly out of service, as we averaged two head replacements from IBM a week. This is much less a factor on use tapes that have been run through a drive a few times.
    - detaro 5y ago
      But that's different than "drive will produce tapes that it can read, but other drives don't"? Because sure, drives can fail and need service/replacement, but that's less insidious than a drive producing tapes that are silently unusable in other drives.
      - op00to 5y ago
        Yes, mystery tapes that only work in certain drives is very bad!
johnklos 5y ago
LTO has been around for more than twenty years, true, but not quite thirty, so we can't test the claim of thirty years of shelf life, but DLT, which are surprisingly similar, came out in 1984, and lots of thirty year old and older DLT media has been shown to be readable. The tape drives themselves are much more of an issue than the tapes. It's a shame, because it necessitates moving data on older tapes to newer generation tapes after a few generations (which reminds me I have to do that with some LTO-3 tapes).
- jgrahamc 5y ago
  Speaking of tape lifetimes, my old cassette CrO2 tapes seem to have survived my parents' house: https://blog.jgc.org/2009/08/in-which-i-switch-on-30-year-old.html https://blog.jgc.org/2009/08/in-which-i-switch-on-30-year-ol...
- wheybags 5y ago
  My one experience of digital magnetic tape is mini-dv cassettes. I recently ripped a bunch of old home videos from some cassettes from the 2000s, and quite a few were fairly damaged. Compared to the vhses from the same time and even older, they were way worse.
- rodgerd 5y ago
  > The tape drives themselves are much more of an issue than the tapes This is absolutely the pain point. Having to either keep multiple generations of drive around, and hope that they don't die in storage; or continually migrate your old tape to new tape is a huge pain in the arse.
albertzeyer 5y ago
I'm interested specifically for long term archiving. So these tapes claim 30 years. I have read that some types of CD, DVD or Blue-rays can last much longer. https://superuser.com/a/71239/37009 https://superuser.com/a/71239/37009 For example the M-DISC (https://en.wikipedia.org/wiki/M-DISC https://en.wikipedia.org/wiki/M-DISC). > Millenniata claims that properly stored M-DISC DVD recordings will last 1000 years.
- londons_explore 5y ago
  If I had a large amount of data I needed to archive long term and cost effectively, I would archive it to 12 different medias with an 4,8 erasure code, such that if any 4 of the 12 media types are readable, then I can recover the data. I'd choose media like a few types of hard disk (different vendors), DVD's, SD cards, USB memory sticks, tapes. I would then store those bits of media geographically and politically distributed. And I'd store it with paper documents describing the encoding, the file formats, the compression, any encryption, etc. I'd also include a few physical computers (eg. a raspberry pi or laptop) that has all necessary software to read, decode, and display the data. Set it up to be usable by a non-expert - in 1000 years time, there may be nobody who knows how to use a shell or open a file! And I'd have a 2nd copy of the whole lot on hard drives connected to the internet for day to day serving of the data to people who need to see it. All the stuff above is only needed in case of organisational failure, war, civilisation collapse, etc.
  - albertzeyer 5y ago
    That sounds all nice... but do you actually do that? I'm sure you have some amount of data (maybe not so large) that you want to backup long-term? As most of us do? If you do that, I would really love to read some more details on how you actually organize that.
    - londons_explore 5y ago
      My personal data I have no need to keep beyond my own lifespan, and I don't have much of it, so it's easy. The above is what I have set up for some organisations who want to keep data for thousands of years. There are other bits to the process, like every 10-30 years, repeat the process with the new data and the old data. This time, the 'old' data will be much smaller compared to the storage mediums, so keep that data uncompressed, preferably unencrypted, and un-erasure coded in every geographic location. That removes many barriers to access the data, and increases the chances someone that finds it in 200 years bothers recovering the data. Sadly in the future world there is a high chance some of the data is copyright, illegal knowledge or gdpr-impacted and all records need to be erased. There isn't really a good solution to that. It's almost impossible to protect against future humans wanting your data gone.
magicalhippo 5y ago
Nice calculator. Crossover point here for LTO-8 seems to be around 250TB. I think I'll stick with my HDD's for now.
grapescheesee 5y ago
Many clients I have seen using tapes for archive or onsite backup keep them in a humidity and temperature controlled device (looks like a mini fridge). Seems the emphasis is on humidity for the onsite backup rotations.
amelius 5y ago
Would love to see an article of someone taking a drive apart, and hooking an oscilloscope to the read head of a tape drive.
dmitrybrant 5y ago
Funny, I just recently did a similar thing: found an LTO-4 tape drive on eBay for $40, and a few used cartridges (2TB each) for $20. But before writing my backup to the cartridges, I tried reading their contents, and found that they actually came from a major film studio, with backups of raw animated film content on them!
- ChuckNorris89 5y ago
  Damn, that's cool. I wish the second hand market in my country was abundant with cheap exotic hardware. Then again, maybe not, because I'd probably fill my small apartment from hoarding stuff like this. Still did you try to recover any material and wach it?
- NavinF 5y ago
  I was looking into the same thing recently. The price is right ($10/TB tape vs $13/TB HDD) and it’d be nice to have fewer HBAs and SAS cables, but having to swap the tapes manually every 2TB (every 6 hours?) kinda ruins it for me. An automatic tape library would be ideal, but I couldn’t find any in the 100TB range that are cheaper than spinning rust.
  - numpad0 5y ago
    I have a 2U sized LTO2 robot that might have collapsed from stuffs on top by now, but it seemed to have a standard 5" bay drive inside with a passthrough adapter marshaling the drive and the loader mechanism. I wonder if a more recent drive can just be dropped into those libraries or if they need firmware supports.
- paulmd 5y ago
  one thing to emphasize is that the quoted LTO capacity numbers are usually including transparent device compression - if your data is not compressible, such as ZIP/RAR files or compressed audio/video, that's not the number you will get! Home users will really want to think in terms of the "raw capacity" imo. This is normally half of the advertised capacity for the older standards (I believe the newer ones have stronger compression that squeezes a bit more). LTO-5 tapes are 1.5tb raw, for example. Maybe you'll get a little bit out of it, but a lot of the things you'd want to back up (and especially the bulkier stuff that really eats space) are already compressed. Family photo library, audio/video storage? JPGs are compressed, H264/H265 or MP3/FLAC/etc are already compressed. System images? A lot of application files are already compressed. Home user scenarios are not outlook mailboxes and database backups like the "official" scenarios.
CharleFKane 5y ago
I would like to thank the author for bringing back memories. Not all of which are good... (I used to work for a four letter computer corporation doing enterprise technical support, mostly on tape-based products.)
kortex 5y ago
Is there a unix-style streaming tool, like tar/zstd/age, that does forward error correction? I'd love to stick some ECC in that pipeline, data>zstd>age>ecc>tape, cause I'm paranoid about bitrot. I search for such a thing every few months and haven't scratched the itch. The closest is things like inFECtious, which is more of just a library. I would prefer something in go/rust, since these languages have shown really high backwards compatibility over time. Last thing you want is finding 10 years later building your recovery tool that you can't build it. Will also accept some dusty c util with a toolpath that hasn't changed in decades. https://github.com/vivint/infectious https://github.com/vivint/infectious Ok I just dug up blkar, this looks promising, but the more the merrier. https://github.com/darrenldl/blockyarchive https://github.com/darrenldl/blockyarchive
- dmitrybrant 5y ago
  If I'm not mistaken, the tape drive automatically adds ECC to each written block, and then uses it to verify the block next time you read it. So if there's bit rot on the tape (i.e. too much for ECC to fix), it will just be reported as a bad block with no data, and there wouldn't be any point of adding "second-order" ECC from the user end.
  - BenjiWiebe 5y ago
    There might be a point if you interleaved data and/or had a much higher amount of EC, such that you could recover from isolated bad blocks.
  - metabagel 5y ago
    You’re exactly right. There is substantial ECC in the LTO format. If the drive can recover the data, then it’s valid.
  - kortex 5y ago
    I'm not using tapes through, I'm using ssds, flash drives, and hdds. I'm sure they have some internal ecc, but is it enough?
    - 5y ago
cbm-vic-20 5y ago
> Unlike most block devices these are devices that do not enjoy seeking of any kind. Old-school DECtapes were actually random-access, seekable block devices! They help 578 blocks of data, each block being 512 bytes (or to be more period correct, 256 16 bit words), so 144kiB. They could be read/written in both directions. When mounted on a tape drive, the OS (like DEC RT-11) would treat it just like how a PC DOS computer treats a floppy: you could get a directly listing, work with files, etc. The random access nature caused the tape to move quickly back and forth across the tape head, a process known as "shoe shining". https://youtu.be/ZGBS8mBAfYo?t=579 https://youtu.be/ZGBS8mBAfYo?t=579
- StillBored 5y ago
  Tape can do random seeks, but its generally append only. LTO, though supports partitioning which is utilized by LTFS (https://www.lto.org/linear-tape-file-system/ https://www.lto.org/linear-tape-file-system/) to provide a mountable filesystem abstraction. It works just like any other filesystem, but one has to remember that seeks are much slower than HDs and that overwriting/updating a file basically is like a versioned FS where the old data is still being stored. Edit: Also, tape formats tend to come in two scan methods since they are generally wider than the tape heads (which frequently are actually multiple heads). Helical scan (think VHS/DAT) and serpentine. LTO is serpentine which means it writes a track from beginning to end, then writes the next track in "reverse" from end to beginning, then the next track again from beginning to end. Back and forth until it hits its track limit. So basically just about every modern drive reads and writes in both forward and reverse. Although shoe shining (backing up to start the next read/write) is still a thing despite variable speed drives which try to speed match to the data rate the host is reading/writing at.
- tssva 5y ago
  The Coleco Adam home computer also had tape drives which were random-access seekable block devices. 2 tracks with 128 1k blocks per track for a total capacity of 256k. Coleco called their tapes digital data packs. They were standard compact cassette tapes with some additional holes. If you drilled the appropriate holes you could use standard tapes instead of paying the Coleco premium. CP/M required booting from a block device and as far as I know the Coleco Adam was the only computer which could boot CP/M from a tape. Once booted to CP/M the tape drives were treated just as floppies.
wazoox 5y ago
Hum, that reminds me that I've written a somewhat more complete user guide for LTO tapes, but in French: https://blogs.intellique.com/tech/2021/08/20#BandesCLI https://blogs.intellique.com/tech/2021/08/20#BandesCLI Let me know if you'd like an English version :)
- wazoox 5y ago
  I did it anyway: http://blogs.intellique.com/tech/2022/01/27#TapeCLI http://blogs.intellique.com/tech/2022/01/27#TapeCLI
MayeulC 5y ago
The tech doesn't seem to be too complex, is there an open hardware project? Seems like one could go quite far in terms of performance with just some basic HW and an FPGA. Is there significant difference between multiple generation of the tapes themselves, or is it just data encoding patterns that change? More specifically, I was a bit appalled by the "magnetic erasing" bit. Seems like DRM to me, on a medium that is conceptually extremely simple. One could probably take a VHS drive and convert it to a data drive, unless I'm being naively optimistic about it?
- tssva 5y ago
  Back in the day there were a few backup products available which connected to standard VHS VCRs.
- justsomehnguy 5y ago
  > One could probably take a VHS drive and convert it to a data drive https://en.wikipedia.org/wiki/ArVid https://en.wikipedia.org/wiki/ArVid > More specifically, I was a bit appalled by the "magnetic erasing" Nobody laments what there is no 'low level format' for HDDs anymore.
- zaarn 5y ago
  A VHS drive uses a different encoding pattern, the head of the VHS player is physically incapable of moving like the head of an LTO tape. Additionally it lacks precision as an LTO tape is much more densely packed. Lastly, LTO drives use different magnetic materials and signalling, so by all chance the VHS head is probably only going to pick up noise.
MayeulC 5y ago
I'm wondering what would be the best way to store archival data? A disk image plus compressed, encrypted then forward-corrected `btrfs-send` snapshots sounds quite efficient to me. Take your hourly, etc snapshots to a regular disk, write monthly ones to the tape until fills up, then take another tape and repeat. The downside is that you need to replay multiple diffs. Or would it be a good idea to make more frequent writes? I'm not sure what best practices are when it comes to tape and backup.
wglb 5y ago
It would appear the Google backs up the internet on tape: https://www.youtube.com/watch?v=eNliOm9NtCM https://www.youtube.com/watch?v=eNliOm9NtCM Or at least did at one time.
- fishnchips 5y ago
  It probably still does. I was on the gTape SRE team until 2014 and we had lots and lots of tapes and tape libraries back then, most of them giant beasts with 8 robots each. With the capacity of new LTO generations constantly growing and the existing investment in hardware and software it would be unusual to discard that.
  - wglb 5y ago
    One thing that I found very useful (not to forget the use of RAID on tape!) was the way to delete an entire user's data was to encrypt everything with a per-user key, and upon request to delete, the key would be securely destroyed.
    - fishnchips 5y ago
      Ah yes, it's a standard practice https://en.wikipedia.org/wiki/Crypto-shredding https://en.wikipedia.org/wiki/Crypto-shredding
cassepipe 5y ago
"Unlike most block devices these are devices that do not enjoy seeking of any kind. So you generally end up writing streaming file formats to tape, unsurprisingly this is exactly what the Tape ARchive (.tar) is actually for. " Haha! moment
cassepipe 5y ago
Apart from archiving huuuuuge amounts of data, does it make sense for any business to invest in those when you add up in the qualified work time it necessitates for the halved priced it provides. Plus the constant reinvestment in hardware. Plus the fact that to get the data you actually need a human to fetch for you and operate a machine. Who uses this ?
- madduci 5y ago
  A lot of companies, trust me
- motoboi 5y ago
  Everyone. It’s much easier to store tapes in a fire proof and water resistant safe than to find a fire and water resistant storage. So you can keep you backups in disk, but last resort disaster recoveries should be on tape somewhere. Gmail has tapes[1]. And they saved me their asses at least once. This can give you a hint of how important and how much use tapes get. 1 - https://www.datacenterknowledge.com/archives/2011/03/01/google-turns-to-tape-to-rescue-lost-gmail https://www.datacenterknowledge.com/archives/2011/03/01/goog...
robohoe 5y ago
I cut my sysadmin teeth doing tape work in early 2000s. It was quite fun but I don't miss changing tapes and ensuring that the FC tape loader library properly labeled them.
paulmd 5y ago
I picked up an LTO-5 drive a couple years ago and one thing I found (this is probably a good place to bring it up!) is that the software documentation for tape utilities and high-level overviews of the strategies employed to build and manage libraries of data on this model is pretty thin on the ground at this point. Completely understandable given how few people have tapes these days, but it also makes it a little tougher to pick up from scratch. (And in particular the high-level overviews are important because tapes are wear items, you only have on the order-of a hundred or two (don't remember the exact figures) full tape reads before the tape wears out, so this is something you want to go into it knowing a strategy and not making it up as you go!) Since it's complimentary to this discussion I'll link a few: https://www.cyberciti.biz/hardware/unix-linux-basic-tape-management-commands/ https://www.cyberciti.biz/hardware/unix-linux-basic-tape-man... https://databasetutorialpoint.wordpress.com/to-know-more/how-to-work-with-tape-driveslto-in-linux/ https://databasetutorialpoint.wordpress.com/to-know-more/how... https://sites.google.com/site/linuxscooter/linux/backups/tape-commands https://sites.google.com/site/linuxscooter/linux/backups/tap... https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_storage_devices/managing-tape-devices_managing-storage-devices https://access.redhat.com/documentation/en-us/red_hat_enterp... https://access.redhat.com/solutions/68115 https://access.redhat.com/solutions/68115 That is, unfortunately, essentially the apex of LTO tape documentation in 2022, as far as I can tell. Do note that in terms of tape standards, LTO-5 is an important threshold because that's where LTFS support got added, and that's the closest thing to a "normal" filesystem abstraction that's available for tape (sort of like packet-formatted CDRWs I guess, in the sense of presenting an abstraction over the raw seekable block device). There is also very little documentation on init, care, and feeding of LTFS iirc - and again, it would be nice to know any pitfalls that might cause shoeshining and tape death. Although I suppose in practice it's mostly going to get used more in a "multi session" scenario where you mostly aren't deleting files, you write till it's full and then maybe wipe the whole tape at once, and it's just a nice abstraction to allow the abstraction of "files" rather than sequential records (tape archives/TARs, in fact!) along an opaque track with no contextualization.
connorgutman 5y ago
I recently purchased a LTO-5 drive for my Gentoo-based NAS and have a few key takeaways for those who are interested. Don't buy a HP tape drive if you want to use LTFS on Linux! HPE Library & Tape Tools is prety much dead on modern Linux. Official support is only for RHE 7.x and a few versions of Suse. Building from source is a dependency nightmare that will leave you pulling hair. IBM drives have much better Linux support thanks to https://github.com/LinearTapeFileSystem/ltfs https://github.com/LinearTapeFileSystem/ltfs. That being said, IMO, you should consider ditching LTFS for good ol' TAR! It's been battle tested since 1979 and can be installed on basically anything. TAR is easy to use, well documented, and makes way more sense for linear filesystems. While drag&drop is nice and all, it really does not make sense for linear storage.
- smackeyacky 5y ago
  Upvote for tar! LTFS seems like an overly complex solution to a relatively simple problem that tar already solved. Treating tapes like disks and trying to run a file system on them ignores the way they work.
- bsdice 5y ago
  Can tar control block size and maximum file size? If you can't set those to 1 MB and 20 GB respectively, then for LTO-5 and later you will be leaving alot of performance on the table. Like half or more, check file size graph in https://www.bareos.com/wp-content/uploads/2021/05/optimizing-the-tape-speed.pdf https://www.bareos.com/wp-content/uploads/2021/05/optimizing....
  - jwiz 5y ago
    idk man, tar is pretty new, they probably haven't had time to do that yet.
abbbi 5y ago
if one wants to play with virtual tape libraries, quadstorvtl is a nice solution to that: https://quadstor.com/ https://quadstor.com/ unfortunately they dont seem to have an open vcs for the source... (other than really old versions on github) other than that there is mhvtl: http://www.mhvtl.com http://www.mhvtl.com
dark-star 5y ago
PSA: Don't do a cleaning run unless your tape drive tells you to (there are SCSI sense codes for that). The tapes can pretty well assess the need (or not) for cleaning, and excessive cleaning can negatively affect the lifetime of the r/w head (the cleaning tapes are abrasive)
wolfgang42 5y ago
The mt(1) manpage describes seeking on files, records, and file marks, but doesn't explain what any of them are. What's the difference between these these options? (It sounds like file marks are stored on the tape on a special track or something, but I can't seem to find any discussion of the others.)
- StillBored 5y ago
  So, all from a few years old memory and its a complex interwoven mess. Lets start with tape has two types of head positioning commands, locate and space. Locate is absolute (and mt calls it seek), and space is relative. Mt is generally using space (although one can read the current position with tell then do relative space) for all the commands that aren't "seek". Hence the mt commands are things like "fsf" which is forward space file (mark), or "bsf" for back space file (mark). At some point in the past someone thought that each "file" would fit in a tape block, but then reality hit because there are limits on how large the blocks can actually be (in linux its generally the number of scatter gather entries that can fit in a page reliably). So there are filemarks, which are like "special" tape blocks without any data in them. Instead if you attempt to read over a filemark the drive returns a soft error telling you that you just tried to read a filemark. There are also "fsr" for forward space records with are just the individual blocks forming a "file". So back to seeking. If you man st, you will notice that each tape drive gets a bunch of /dev/st* aliases, which control the close behavior/etc, as well as some ioctls that match the mt commands. The two important close behaviors to remember are that if the tape is at EOD due to the last command being a write it will write a filemark, then rewind the tape unless a /dev/stXn device is being used, in which case it will leave the head position just past the FM (this is actually a bit more complex too because IIRC there may be two filemarks at EOD, and the tape position gets left between them). This allows one to do something like "for (x in *.txt); do cat $x >> /dev/st0n; done" and write a bunch of files separated by filemarks (at the default blocking size which will be slow (probably 10k), replace the cat with tar to control blocking/etc). Or if you want to read the previous file `mt -f /dev/st0n bsf 2` to back space 2 filemarks. Now, the actual data format on tape is going to be dictated by the backup utility used to write it. Some never use filemarks, some do but as a volume separator (eg tar), old ones actually put FM's between files, but that tends to be slow because it kills read perf because it takes the drive out of streaming mode whenever you either read over the filemark (not the part on man st about reading a filemark). Now you can pick which file to read via "mt -f /dev/st0 rewind; mt -f /dev/st0n fsf X; cat /dev/st0n > restore.file" There are also tape partition control commands, and tape set marks and various other options which may/may not apply to a given type of tape. Noticeably there are also density flags on the special file (some unix'es) and via mt. LTO for example doesn't have settable densities because its fixed by the physical tape in the drive. Some drives STK T10K/IBM TS11X0/3592 can upgrade the tape density/capacity when used in a newer drive. That got long...
watersb 5y ago
Everyone who cares about backups chooses a backup system design. Anyone who cares about their stuff needs to practice a full emergency RESTORE. I have met very few people who actually do that. For most systems I've seen, the first full test of the restore process is a very scary first production usage of the restore process. Which is very exciting, sure. I don't want excitement in my data management life. (I actually see weekly test of onsite backup power at the local banks, and at some large commercial kitchens. Those diesel generators are very loud. I've never seen systematic test of UPS or generators in a front-office environment.)
dragontamer 5y ago
I've come to the understanding that tape-drives are for people who need to "build a custom-sized storage solution", especially if you need capacity but not necessarily read/write speeds. A tape-drive is your read-heads. The tape is like a platter. The tape-library / jukebox is just a robotic mechanism for switching tapes into and/or out of the read-head. ---------- If you need a Petabyte of uncompressed storage, you can reach it with a tape-library consisting of 84 LTO8 tapes (12TB each). If read/write of 400MB/s is sufficient, one tape drive is sufficient. If you need faster access speeds, you buy a 2nd, 3rd, or 4th tape drive. So lets say you need 2GB/s read/write speed and a petabyte of storage. You simply get 4x LTO 8 drives, 84 LTO8 tapes, and stick them into a tape library of some kind. You then buy a certain amount of SSDs + HDDs sufficient for caching, so that you can read/write to this tape library at sufficient speeds (especially since it could be many minutes before a specific byte is accessed).
- metadat 5y ago
  What software is going to be gluing and integrating all these drives together into a usable interface? Does it already exist as open-source?
  - dragontamer 5y ago
    LTFS (Linear Tape File System) shows each each "file" as often a pretty dumb "write-only" mechanism / append-only file system. That is: if a file A is modified, LTFS "represents" this by marking fileA as dead, then copying most of the contents of A to the end of the tape... with the few modifications in the correct location. Because tapes have so much capacity, this ends up working much better in practice than you'd expect. Especially if your backup-like software (or Apache-log files or whatever you're archiving) are largely append-only (ex: Wikipedia dumps or whatever huge data-storage requirements you're doing). ----------- Allegedly anyway, I don't really use tapes myself, but I amuse myself by looking at their specs / usability every now and then. It seems like the details of LTFS depend on each manufacturer's system, but it seems like LTFS itself is somewhat cross-compatible. Some LTFS-library docs suggest that different tapes are different folders. Others suggest that different tapes are mount/unmounted (which sends a sequence of commands to the little tape-robot to fetch the correct tape, which it does so by scanning barcodes or maybe going to correct slot) ----------- Tapes are meant to be used in a "jukebox-style" application IMO. If you need just one tape, you're doing it wrong. If you need 80-tapes, then doing it manually is obviously wrong, you should pay a bit extra for that "jukebox / tape-library" robot to fetch the right tapes automatically. > Does it already exist as open-source? Red Hat claims LTFS support. But I imagine this is only "disk-level LTFS", and not "library / robot LTFS".
watersb 5y ago
I use a cloud storage provider to back up via Arq https://arqbackup.com https://arqbackup.com But I don't expect to restore more than a few gigabytes at a time from that. It would take me a week or more to download a terabyte of data. I have very little power over internet connection speed, and there are very few alternatives here. I believe there are two different vendors providing connectivity to our town, and you can pick between four retail resellers. With those limitations, I have tested a full restore process exactly once. That's not good enough. Data at rest on LTO or offline hard disk is something I can control. Distributed offsite storage, too. Restore within 12 hours, I can do that. The downside to tape or cold disk is more in the management of hourly/daily/weekly backups: you have to provision a media rotation schedule, whereas that's sort of built into an online cloud storage service.
omgwtfbyobbq 5y ago
How durable are LTO drives? I've been thinking about moving some backups to tape because of the stated durability and good cost (~$7-$12+/tb for used LTO4 tapes+drive, depending on how much storage I get). But if the drives don't also last 15+ years, then I think the cost could be significantly higher, especially if I'm shelling out who knows what for drives a decade after their low point on the resale market.
- justsomehnguy 5y ago
  > LTO4 Is useless for a long-term, both from $/GB and from sanity standpoint. LTO-5 is more than 10 years old already, there is no point in trying to use LTO-4. > How durable are LTO drives? Just look at the comments here. 99% of time it just works. In 1% you throw the shit out of the window (well, you want to). Also: > Both drives and media should be kept free from airborne dust or other contaminants from packing and storage materials, paper dust, cardboard particles, printer toner dust etc.[49] And "5 Environmental" in [0] https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol4_E4/LTO5_Vol4_E4.pdf https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol4_E4/LTO5_Vo...
vajrabum 5y ago
Make sure you test restores periodically! I had a co-worker who got fired when they tried to reformat a failing disk instead of sending it out for recovery and felt safe because the user "had tape backups." Those were DLTs and perhaps today such failure would be a lot less likely but still. Just do it. Part of basic tape/backup hygiene.

293 points by detaro 5y ago

amelius 5y ago

With these prices for drives the market seems ripe for disruption.

dsr_ 5y ago
It already is, by spinning disks. Cheaper at the low end, faster the whole way through, random access beats linear access for end user expectations.
- amelius 5y ago
  You are too much focused on read speed. I just want to write huge amounts of data at a low cost, and don't mind waiting a day for retrieval. I.e., how one normally uses backups.
  - TheCondor 5y ago
    And durability. I’ve had a portable usb hard drive fall over on my desk and it had major problems after that. Solid state fixes that but it’s expensive and I’ve heard they can lose data if not plugged in with regularity
    - lazide 5y ago
      Yeah, SSD is not good for long term storage (like a copy of your tax documents from last year you might need in 5 years). The expense for size also makes it infeasible to copy ongoing roll up copies of everything which is one way of solving that.
  - lazide 5y ago
    Most of the time when people think backups they need faster than 24 hr turn around to restore - because it usually takes about that long to figure out they even need a backup, and most people don’t think ahead enough for 2 day recovery time to be useful for most use cases now a days. If their local snapshots are dead too, or they look for it and realize they can’t find a copy of something they thought they had, it’s often because they needed that data right away and it wasn’t there when they went to get it. Hence ‘user expectations’. That’s not in a catastrophic case (which rarely happens) that’s the ‘bob just realized he deleted the folder containing the key customer presentation last Friday’ or ‘mary just tried to open the contract copy she needed and it’s corrupted’. If it’s a once in 10 or 100 year or whatever event, a 1-2 day turnaround is not unexpected and everything else is probably broken too. The file deleted or something got screwed up happens more often and slow response there grinds things to a halt - and causes a lot of stress knowing it’s not ‘solved’.
    - amelius 5y ago
      I bet most companies who are confronted with ransom demands would die for tape backup even if restoration took a week (which is the amount of time they need anyway to get the whole mess sorted out).
      - lazide 5y ago
        For sure - that would fall under catastrophic IMO (odds of any company getting crypto ransomed are low on any given year, though increasing, and can kill the company if there isn’t any prep). A lot of companies are pretty terrible at figuring out catastrophic tail risks like this too. Tape has it’s niche, this is one of them for sure.
- zozbot234 5y ago
  SMR spinning disks are also being widely repurposed as "archival", somewhat tape-like media since they turned out to be quite low-performance for the most common use scenarios (which means they were getting dropped from soft-RAID arrays, etc.).
  - floatboth 5y ago
    > repurposed Weren't they even originally marketed as such? At least the external enclosures that people like to shuck are usually called something like "my backup".
- KaiserPro 5y ago
  kinda but not. The problem with spinny disks is that you have allocate space for them. You can't quickly swap out drives to take offsite. Whats grand about tape is that its still faster to dump to your library, eject the magazines and store off site. Whilst you can do that with HDDs (think snowballs but bigger) its a lot more expensive and error prone. Tape serves a purpose, but thats pretty niche by todays standards.

tlamponi 5y ago

Interesting read, as with most of ben's blog.. And yeah, buffering is definitively required to get acceptable speed out of tape tech. If you want a LTO Tape solution with more bells and whistles you could check out Proxmox Backup Server's tape support: https://pbs.proxmox.com/docs/tape-backup.html https://pbs.proxmox.com/docs/tape-backup.html We also rewrote mt and mtx (for robots/changers) in rust, well the relevant parts: https://pbs.proxmox.com/docs/command-syntax.html#pmt https://pbs.proxmox.com/docs/command-syntax.html#pmt https://pbs.proxmox.com/docs/command-syntax.html#pmtx https://pbs.proxmox.com/docs/command-syntax.html#pmtx The introduction/main feature section of the docs contain more info, if you're interested: https://pbs.proxmox.com/docs/introduction.html https://pbs.proxmox.com/docs/introduction.html If you have your non-Linux workload contained in VMs and maybe even already use Proxmox VE for that it's really covering safe and painless self-hosted backup needs. Disclaimer: I work there, but our projects are 100% open source, available under the AGPLv3: https://git.proxmox.com/ https://git.proxmox.com/

Brian-Puccio 5y ago
The docs seem to indicate that if you have a directory structure (say over NFS share) you want to backup to tape, you must first back it up to a datastore (read: local storage on the PBS), that you cannot backup the files straight from a directory structure you must first store them locally in the datastore?
- tlamponi 5y ago
  Yes, normally tape is a second level backup for the worst case and long term archival. Tapes normally are not kept on site, to ensure that even a destruction of the office/datacenter/... like, for example, a fire does not make one lose all data. With PBS this also allows to leverage the whole thing in a more flexible, efficient and powerful manner, as one can do tape-backups async, the deduplication layer can be used so that subsequent backups only need to add the delta of new backup chunks from the content addressable storage (note, the backups themself are still full backups, and one can configure a schedule for when a new media set needs to be created), besides that backing up file-level to tape is normally pretty slow compared to writing out big chunks that are ordered by physical disk location.

Synaesthesia 5y ago

That's a lot of storage! Can't really think of a use for this (200tb plus) personally but it is appealing.

throw0101a 5y ago
Tape makes more sense the larger you go, as it help amortize the fixed/upfront costs. The incremental costs of buying more tapes (that are re-usable) isn't that much at scale. It's often relatively cheap insurance against data loss for many organizations. A lot of 'enterprise' backup software is also now coming with hooks into cloud storage (e.g., S3 APIs), but then you have to worry about bandwidth and the time it takes to get the bits offsite at "x" bits/second. Of course you also have to worry about retrieving the data in case of disaster per the Recovery Time Objective: * https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Tim... Also: a backup has not happened until you try and succeed your recovery process.
- metabagel 5y ago
  > but then you have to worry about bandwidth and the time it takes to get the bits offsite at "x" bits/second. Reminds me of the saying that the fastest throughput is achieved by a 747 full of hard drives. > Also: a backup has not happened until you try and succeed your recovery process. A thousand times this.
simcop2387 5y ago
For me it's a lot about just being a data hoarder and never having to delete something because i'm low on storage. About half of my system though is taken up by system backups and virtual machines. I should do a cleanup of those, but the freedom of just being able to spin up something new or put a new backup on there without ever going, "do i have enough space for this?" is rather nice.
- organsnyder 5y ago
  I also rarely/never delete anything, but my ~2tb NAS still has plenty of room. I guess it makes a difference that the only media I store is my own photos and videos.

einpoklum 5y ago

> LTO Tape is ... much cheaper than hard drives ... a 12TB SATA drive costs around £18.00 per TB ... a LTO-8 tape that has the same capacity costs around £7.40 per TB ... That’s a significant price difference. Actually, it isn't very significant. Price factor of 2.5. I had thought tape storage was cheaper than that. And then there are the drives: A drive to write (3,000 GBP for LTO-8), and at least a couple more drives for reading tapes. At this price ratio, I would say that ease-of-use and safety/robustness of the backed-up material are more important considerations.

AshamedCaptain 5y ago
I am also worried for the long-term. If there are new generations so frequently and backwards compatibility is limited or not guaranteed, I ponder if you'd be able to find a working-condition tape reader for your 20-year old tape... At least it's likely I can find a USB port 20 years from now, or a DVD reader (they are still being manufactured today, when even more than 20 years have passed since their introduction, and they are even compatible with much older CDs...).
ktpsns 5y ago
What was actually ignored at that comparison is energy costs. Which can get quite somewhere, if you have all your disks running 24/7 and do not use power saving functions (which is frequently turned off in server contexts). Costs are in the ballpark of 5W per drive, given a contemprary 16TB drive this means 0.3W/TB, with 0.25EUR/kWh (a typical consumer price in Germany), this is roughly 0.6 EUR per TB per year. However, probably the replacement costs for these always-on disk drives will be even higher.
- q3k 5y ago
  Another consideration related to this is that tapes, being usually offline, as much more secured against accidental (or malicious!) erasure when compared to always-on hard drives. Also related is that tapes can easily be transported around/offsite, literally thrown in the back of a truck as they are. Try doing that to hard drives and see how many start throwing bad sectors after a round-trip.

epilys 5y ago

What I never see explained, is what exactly PCI cards should I get to get the full sized SAS drive to work with my desktop PC? Because looking at server component stores, I see there digit prices for SAS controllers, and the author mentions they are cheap.

c0l0 5y ago
My advice: On eBay (or any other platform that makes it easy to buy used hardware components), go look for "sas2008 4e", and check out the offers. You should be able to get a decent HBA driven by mpt2sas/mpt3sas for around 40 to 60US$.

c0l0 5y ago

I recently started looking into using an LTO-7 tape drive that I got handed down, along with a few dozens of pristine LOT-6 tapes, for archiving purposes. I got to play around a bit with SAS HBAs, and was kinda shocked how much of a difference that can make in the user (or shall I say sysadmin?) experience: LTO-6 tapes are spec'd to transfer rates of around 150MB/s, so well within the reach of even the the first SAS gear generation. However, the very first SAS HBA with external SFF-8088 connector I managed to get my hands on (an LSI SAS1068e) topped out at a disappointing 80MiB/s, no matter what I tried in terms of blocking and buffering. Switching to a more modern (but still old) LSI SAS2008-based HBA got me close to the theoretical maximum. Then there's the (to me, still open) question of how to best use the actual tape storage capactiy... Since my hardware is newer than LTO-5, LTFS (https://github.com/LinearTapeFileSystem/ltfs https://github.com/LinearTapeFileSystem/ltfs) is an option for convenient access, especially listing tape contents, but that could make it hard for other people down the line to restore data from the tapes I create. It's probably safest to assume that tar will always be there, at least wherever there's tape, too. GNU tar also handles multi-volume/-tape archives, which seems like a necessity if you need to back up amounts of data that exceed a single tape's capacity. Then again, if you want to use encryption with actual tar (important for the kind of data I need to archive), your only option seems to be piping the whole archive through something to compress the stream, which will make accessing individual records in the archive opaque to the drive itself... and you can't just dispose of individual keys to make select parts of the archived data go away for good, either. Also, I would like to conserve as much tape as (conveniently) possible in my archiving adventure. There's "projects" (i.e., top-level directories of directory trees) that consume more than one tape of their own, and then there's smaller projects that you can bin-pack together onto tapes that can fit more than one such project. I've started implementing a small python wrapper around GNU tar to solve a number of these problems by bin-packing projects into "tape slots" and also keeping track of tape-to-file mappings in a small sqlite database, but a workable solution for the encryption problem(s) is not something I managed to come up with yet... If someone has an idea (or better yet, a complete and free implementation of what I am trying to hack together :)), please be so kind and let me know!

__d 5y ago
Using a multi-tape archive increases your chance of failing to restore. Rather than having one possible tape that can be corrupt, you know have more than one. Any single failure will corrupt the whole archive, and you've lost (easy access to) the data. Better to break up the source volume somehow, and have each tape self-contained.

Spooky23 5y ago

Backups are a really interesting business. I helped out a colleague a few years ago with a project in a big data center and it was like a whole world that nobody knew existed. Because of the RTOs and backup windows, the supporting infrastructure was fast. The caching layer stuff was the fastest disk in the data center by far, and the team was a small, tight group of people who basically honed their craft by meeting auditor and other requirements. The management left them alone and they did their thing. That was about a decade ago now; those guys have all moved on to really big things.

StillBored 5y ago
Its still that way, the netflix guys get a lot of press for their bandwidth numbers but plenty of backup systems were getting similar (or greater) bandwidth numbers years ago, since many of the caching stacks are basically pcie or mem bandwidth limited. The 300MB/sec number the author lists is really slow, and likely appropriate for LTO3/4, (IIRC, the wikipedia numbers are understated) LTO7+ can peak at > 1GB/sec with the modern drives going even faster if the compression is left enabled. So, given a library with a few dozen drives, the bandwidth gets insane. (ex: SL8500)
- trasz 5y ago
  Someone should ask Spectra Logic folks for their numbers :-) (Spectra Logic's tape libraries run FreeBSD too.)
  - monocasa 5y ago
    SpectraLogic's code isn't in the data plane, you hook up to the drives directly, and the drives can forward changer requests to the internals of the library. So it's however fast the drives are (which are all third party). Also last I checked freebsd was used for their disk product, not tape.

smackeyacky 5y ago

There is one recommendation there I find a bit questionable and thats encryption. If you are out of options and restoring from tape, might be better to have it uncompressed and not encrypted. Its possible, after some physical disaster that you are on somebody elses infrastructure and having some encryption on your data doubles the problems you might have. I use an ancient LTO2 drive for last resort backups that are off cloud and off premises. Its more peace of mind than practical on a daily basis but I did find myself restoring a few files a couple of weeks ago as I had fat fingered an rm command. It was quicker than getting them from S3 glacier.

benjojo12 5y ago
I agree somewhat. Encryption is more critical on tape because there is no easy path to wiping a tape, and in a company situation if you need to erase something in your backups too (think GDPR erasure), then encryption is reasonably critical unless you want to go though all of your cold backups. For my archival use (the reason why I got into this in the first place) I do not encrypt nor compress the data going to tape. For server/desktop backups. they are compressed and encrypted.
- throw0101a 5y ago
  > […] nor compress the data going to tape. Just to note that tape drives have built-in compression that generally is done transparently in the background. So while using something like zstd (per the article) may get more bits on a given tape, there is some compression that one gets "for free" without doing anything at all. * https://en.wikipedia.org/wiki/Linear_Tape-Open#Optional_technology https://en.wikipedia.org/wiki/Linear_Tape-Open#Optional_tech... * https://en.wikipedia.org/wiki/Magnetic_tape_data_storage#Data_compression https://en.wikipedia.org/wiki/Magnetic_tape_data_storage#Dat...
  - benjojo12 5y ago
    I mention this in the post itself

kragen 5y ago

Hey uh is that a DECTape?

benjojo12 5y ago
The header image is the insides of a LTO 5 tape
- kragen 5y ago
  Compare https://en.wikipedia.org/wiki/Digital_Linear_Tape https://en.wikipedia.org/wiki/Digital_Linear_Tape

PaulHoule 5y ago

I notice that he talks alot about dealing with malfunctioning drives and malfunctioning tapes. That is my experience too. There is that time I got kicked out of the computer lab as an undergraduate because I'd created a number of newsgroups and they 'wrote' all my files... to what turned out to be an empty SunTape. That time I tried to recover a configuration file from an IBM tape robot and it took 14 hours. When I was successful with tape I always did a lot of practicing and testing. A sysadmin who taught me a lot (esp. how to get things done in a place where you need 'social engineering' to get things done) told me "you don't have a backup plan until you've tested it" and many people learned that the hard way.

ansible 5y ago
> "you don't have a backup plan until you've tested it" Yep. Though that's what makes small-shop disk-to-disk backups easy, depending on the backup software used. We use rsnapshot, which uses rsync and "cp -l" to make backups. So restoring is as easy as using cd to go into the appropriate directory and copying out the files. No special utilities needed. Yes, we encrypt the backup drives using cryptfs / LUKS.

archi42 5y ago

Something not mentioned by the author, but what I was told here on Hacker News some years ago: If your drive has too much wear (or misalignment of the drive head?) you might end up with tapes that you can only read with exactly your drive.

detaro 5y ago
That's something I've seen mentioned too but never could verify if that is something that's actually true with modern tape standards or not. (i.e. last I asked on HN I was told it wasn't a concern anymore) If the drive needs to adjust to get precise enough positioning anyhow, misalignment seems way less likely.
- op00to 5y ago
  It's absolutely true. There is a LOT more to tape storage than meets the eye. Let's say you're using LTO tapes as an archive. Did you know LTO tape itself is abrasive, but that abrasive is meant to wear over time with the intended use of the cartridges, which was backups? If you use new tapes a single time, the abrasive doesn't wear and destroys the tape heads. You will go through a drive head at month, running the drives 24/7. I had a library used as a genomic storage archive with 8 drives (always write, almost never read), and two were constantly out of service, as we averaged two head replacements from IBM a week. This is much less a factor on use tapes that have been run through a drive a few times.
  - detaro 5y ago
    But that's different than "drive will produce tapes that it can read, but other drives don't"? Because sure, drives can fail and need service/replacement, but that's less insidious than a drive producing tapes that are silently unusable in other drives.
    - op00to 5y ago
      Yes, mystery tapes that only work in certain drives is very bad!

johnklos 5y ago

LTO has been around for more than twenty years, true, but not quite thirty, so we can't test the claim of thirty years of shelf life, but DLT, which are surprisingly similar, came out in 1984, and lots of thirty year old and older DLT media has been shown to be readable. The tape drives themselves are much more of an issue than the tapes. It's a shame, because it necessitates moving data on older tapes to newer generation tapes after a few generations (which reminds me I have to do that with some LTO-3 tapes).

jgrahamc 5y ago
Speaking of tape lifetimes, my old cassette CrO2 tapes seem to have survived my parents' house: https://blog.jgc.org/2009/08/in-which-i-switch-on-30-year-old.html https://blog.jgc.org/2009/08/in-which-i-switch-on-30-year-ol...
wheybags 5y ago
My one experience of digital magnetic tape is mini-dv cassettes. I recently ripped a bunch of old home videos from some cassettes from the 2000s, and quite a few were fairly damaged. Compared to the vhses from the same time and even older, they were way worse.
rodgerd 5y ago
> The tape drives themselves are much more of an issue than the tapes This is absolutely the pain point. Having to either keep multiple generations of drive around, and hope that they don't die in storage; or continually migrate your old tape to new tape is a huge pain in the arse.

albertzeyer 5y ago

I'm interested specifically for long term archiving. So these tapes claim 30 years. I have read that some types of CD, DVD or Blue-rays can last much longer. https://superuser.com/a/71239/37009 https://superuser.com/a/71239/37009 For example the M-DISC (https://en.wikipedia.org/wiki/M-DISC https://en.wikipedia.org/wiki/M-DISC). > Millenniata claims that properly stored M-DISC DVD recordings will last 1000 years.

londons_explore 5y ago
If I had a large amount of data I needed to archive long term and cost effectively, I would archive it to 12 different medias with an 4,8 erasure code, such that if any 4 of the 12 media types are readable, then I can recover the data. I'd choose media like a few types of hard disk (different vendors), DVD's, SD cards, USB memory sticks, tapes. I would then store those bits of media geographically and politically distributed. And I'd store it with paper documents describing the encoding, the file formats, the compression, any encryption, etc. I'd also include a few physical computers (eg. a raspberry pi or laptop) that has all necessary software to read, decode, and display the data. Set it up to be usable by a non-expert - in 1000 years time, there may be nobody who knows how to use a shell or open a file! And I'd have a 2nd copy of the whole lot on hard drives connected to the internet for day to day serving of the data to people who need to see it. All the stuff above is only needed in case of organisational failure, war, civilisation collapse, etc.
- albertzeyer 5y ago
  That sounds all nice... but do you actually do that? I'm sure you have some amount of data (maybe not so large) that you want to backup long-term? As most of us do? If you do that, I would really love to read some more details on how you actually organize that.
  - londons_explore 5y ago
    My personal data I have no need to keep beyond my own lifespan, and I don't have much of it, so it's easy. The above is what I have set up for some organisations who want to keep data for thousands of years. There are other bits to the process, like every 10-30 years, repeat the process with the new data and the old data. This time, the 'old' data will be much smaller compared to the storage mediums, so keep that data uncompressed, preferably unencrypted, and un-erasure coded in every geographic location. That removes many barriers to access the data, and increases the chances someone that finds it in 200 years bothers recovering the data. Sadly in the future world there is a high chance some of the data is copyright, illegal knowledge or gdpr-impacted and all records need to be erased. There isn't really a good solution to that. It's almost impossible to protect against future humans wanting your data gone.

magicalhippo 5y ago

Nice calculator. Crossover point here for LTO-8 seems to be around 250TB. I think I'll stick with my HDD's for now.

grapescheesee 5y ago

Many clients I have seen using tapes for archive or onsite backup keep them in a humidity and temperature controlled device (looks like a mini fridge). Seems the emphasis is on humidity for the onsite backup rotations.

amelius 5y ago

Would love to see an article of someone taking a drive apart, and hooking an oscilloscope to the read head of a tape drive.

dmitrybrant 5y ago

Funny, I just recently did a similar thing: found an LTO-4 tape drive on eBay for $40, and a few used cartridges (2TB each) for $20. But before writing my backup to the cartridges, I tried reading their contents, and found that they actually came from a major film studio, with backups of raw animated film content on them!

ChuckNorris89 5y ago
Damn, that's cool. I wish the second hand market in my country was abundant with cheap exotic hardware. Then again, maybe not, because I'd probably fill my small apartment from hoarding stuff like this. Still did you try to recover any material and wach it?
NavinF 5y ago
I was looking into the same thing recently. The price is right ($10/TB tape vs $13/TB HDD) and it’d be nice to have fewer HBAs and SAS cables, but having to swap the tapes manually every 2TB (every 6 hours?) kinda ruins it for me. An automatic tape library would be ideal, but I couldn’t find any in the 100TB range that are cheaper than spinning rust.
- numpad0 5y ago
  I have a 2U sized LTO2 robot that might have collapsed from stuffs on top by now, but it seemed to have a standard 5" bay drive inside with a passthrough adapter marshaling the drive and the loader mechanism. I wonder if a more recent drive can just be dropped into those libraries or if they need firmware supports.
paulmd 5y ago
one thing to emphasize is that the quoted LTO capacity numbers are usually including transparent device compression - if your data is not compressible, such as ZIP/RAR files or compressed audio/video, that's not the number you will get! Home users will really want to think in terms of the "raw capacity" imo. This is normally half of the advertised capacity for the older standards (I believe the newer ones have stronger compression that squeezes a bit more). LTO-5 tapes are 1.5tb raw, for example. Maybe you'll get a little bit out of it, but a lot of the things you'd want to back up (and especially the bulkier stuff that really eats space) are already compressed. Family photo library, audio/video storage? JPGs are compressed, H264/H265 or MP3/FLAC/etc are already compressed. System images? A lot of application files are already compressed. Home user scenarios are not outlook mailboxes and database backups like the "official" scenarios.

CharleFKane 5y ago

I would like to thank the author for bringing back memories. Not all of which are good... (I used to work for a four letter computer corporation doing enterprise technical support, mostly on tape-based products.)

kortex 5y ago

Is there a unix-style streaming tool, like tar/zstd/age, that does forward error correction? I'd love to stick some ECC in that pipeline, data>zstd>age>ecc>tape, cause I'm paranoid about bitrot. I search for such a thing every few months and haven't scratched the itch. The closest is things like inFECtious, which is more of just a library. I would prefer something in go/rust, since these languages have shown really high backwards compatibility over time. Last thing you want is finding 10 years later building your recovery tool that you can't build it. Will also accept some dusty c util with a toolpath that hasn't changed in decades. https://github.com/vivint/infectious https://github.com/vivint/infectious Ok I just dug up blkar, this looks promising, but the more the merrier. https://github.com/darrenldl/blockyarchive https://github.com/darrenldl/blockyarchive

dmitrybrant 5y ago
If I'm not mistaken, the tape drive automatically adds ECC to each written block, and then uses it to verify the block next time you read it. So if there's bit rot on the tape (i.e. too much for ECC to fix), it will just be reported as a bad block with no data, and there wouldn't be any point of adding "second-order" ECC from the user end.
- BenjiWiebe 5y ago
  There might be a point if you interleaved data and/or had a much higher amount of EC, such that you could recover from isolated bad blocks.
- metabagel 5y ago
  You’re exactly right. There is substantial ECC in the LTO format. If the drive can recover the data, then it’s valid.
- kortex 5y ago
  I'm not using tapes through, I'm using ssds, flash drives, and hdds. I'm sure they have some internal ecc, but is it enough?
  - 5y ago

cbm-vic-20 5y ago

> Unlike most block devices these are devices that do not enjoy seeking of any kind. Old-school DECtapes were actually random-access, seekable block devices! They help 578 blocks of data, each block being 512 bytes (or to be more period correct, 256 16 bit words), so 144kiB. They could be read/written in both directions. When mounted on a tape drive, the OS (like DEC RT-11) would treat it just like how a PC DOS computer treats a floppy: you could get a directly listing, work with files, etc. The random access nature caused the tape to move quickly back and forth across the tape head, a process known as "shoe shining". https://youtu.be/ZGBS8mBAfYo?t=579 https://youtu.be/ZGBS8mBAfYo?t=579

StillBored 5y ago
Tape can do random seeks, but its generally append only. LTO, though supports partitioning which is utilized by LTFS (https://www.lto.org/linear-tape-file-system/ https://www.lto.org/linear-tape-file-system/) to provide a mountable filesystem abstraction. It works just like any other filesystem, but one has to remember that seeks are much slower than HDs and that overwriting/updating a file basically is like a versioned FS where the old data is still being stored. Edit: Also, tape formats tend to come in two scan methods since they are generally wider than the tape heads (which frequently are actually multiple heads). Helical scan (think VHS/DAT) and serpentine. LTO is serpentine which means it writes a track from beginning to end, then writes the next track in "reverse" from end to beginning, then the next track again from beginning to end. Back and forth until it hits its track limit. So basically just about every modern drive reads and writes in both forward and reverse. Although shoe shining (backing up to start the next read/write) is still a thing despite variable speed drives which try to speed match to the data rate the host is reading/writing at.
tssva 5y ago
The Coleco Adam home computer also had tape drives which were random-access seekable block devices. 2 tracks with 128 1k blocks per track for a total capacity of 256k. Coleco called their tapes digital data packs. They were standard compact cassette tapes with some additional holes. If you drilled the appropriate holes you could use standard tapes instead of paying the Coleco premium. CP/M required booting from a block device and as far as I know the Coleco Adam was the only computer which could boot CP/M from a tape. Once booted to CP/M the tape drives were treated just as floppies.

wazoox 5y ago

Hum, that reminds me that I've written a somewhat more complete user guide for LTO tapes, but in French: https://blogs.intellique.com/tech/2021/08/20#BandesCLI https://blogs.intellique.com/tech/2021/08/20#BandesCLI Let me know if you'd like an English version :)

wazoox 5y ago
I did it anyway: http://blogs.intellique.com/tech/2022/01/27#TapeCLI http://blogs.intellique.com/tech/2022/01/27#TapeCLI

MayeulC 5y ago

The tech doesn't seem to be too complex, is there an open hardware project? Seems like one could go quite far in terms of performance with just some basic HW and an FPGA. Is there significant difference between multiple generation of the tapes themselves, or is it just data encoding patterns that change? More specifically, I was a bit appalled by the "magnetic erasing" bit. Seems like DRM to me, on a medium that is conceptually extremely simple. One could probably take a VHS drive and convert it to a data drive, unless I'm being naively optimistic about it?

tssva 5y ago
Back in the day there were a few backup products available which connected to standard VHS VCRs.
justsomehnguy 5y ago
> One could probably take a VHS drive and convert it to a data drive https://en.wikipedia.org/wiki/ArVid https://en.wikipedia.org/wiki/ArVid > More specifically, I was a bit appalled by the "magnetic erasing" Nobody laments what there is no 'low level format' for HDDs anymore.
zaarn 5y ago
A VHS drive uses a different encoding pattern, the head of the VHS player is physically incapable of moving like the head of an LTO tape. Additionally it lacks precision as an LTO tape is much more densely packed. Lastly, LTO drives use different magnetic materials and signalling, so by all chance the VHS head is probably only going to pick up noise.

MayeulC 5y ago

I'm wondering what would be the best way to store archival data? A disk image plus compressed, encrypted then forward-corrected `btrfs-send` snapshots sounds quite efficient to me. Take your hourly, etc snapshots to a regular disk, write monthly ones to the tape until fills up, then take another tape and repeat. The downside is that you need to replay multiple diffs. Or would it be a good idea to make more frequent writes? I'm not sure what best practices are when it comes to tape and backup.

wglb 5y ago

It would appear the Google backs up the internet on tape: https://www.youtube.com/watch?v=eNliOm9NtCM https://www.youtube.com/watch?v=eNliOm9NtCM Or at least did at one time.

fishnchips 5y ago
It probably still does. I was on the gTape SRE team until 2014 and we had lots and lots of tapes and tape libraries back then, most of them giant beasts with 8 robots each. With the capacity of new LTO generations constantly growing and the existing investment in hardware and software it would be unusual to discard that.
- wglb 5y ago
  One thing that I found very useful (not to forget the use of RAID on tape!) was the way to delete an entire user's data was to encrypt everything with a per-user key, and upon request to delete, the key would be securely destroyed.
  - fishnchips 5y ago
    Ah yes, it's a standard practice https://en.wikipedia.org/wiki/Crypto-shredding https://en.wikipedia.org/wiki/Crypto-shredding

cassepipe 5y ago

"Unlike most block devices these are devices that do not enjoy seeking of any kind. So you generally end up writing streaming file formats to tape, unsurprisingly this is exactly what the Tape ARchive (.tar) is actually for. " Haha! moment

cassepipe 5y ago

Apart from archiving huuuuuge amounts of data, does it make sense for any business to invest in those when you add up in the qualified work time it necessitates for the halved priced it provides. Plus the constant reinvestment in hardware. Plus the fact that to get the data you actually need a human to fetch for you and operate a machine. Who uses this ?

madduci 5y ago
A lot of companies, trust me
motoboi 5y ago
Everyone. It’s much easier to store tapes in a fire proof and water resistant safe than to find a fire and water resistant storage. So you can keep you backups in disk, but last resort disaster recoveries should be on tape somewhere. Gmail has tapes[1]. And they saved me their asses at least once. This can give you a hint of how important and how much use tapes get. 1 - https://www.datacenterknowledge.com/archives/2011/03/01/google-turns-to-tape-to-rescue-lost-gmail https://www.datacenterknowledge.com/archives/2011/03/01/goog...

robohoe 5y ago

I cut my sysadmin teeth doing tape work in early 2000s. It was quite fun but I don't miss changing tapes and ensuring that the FC tape loader library properly labeled them.

paulmd 5y ago

I picked up an LTO-5 drive a couple years ago and one thing I found (this is probably a good place to bring it up!) is that the software documentation for tape utilities and high-level overviews of the strategies employed to build and manage libraries of data on this model is pretty thin on the ground at this point. Completely understandable given how few people have tapes these days, but it also makes it a little tougher to pick up from scratch. (And in particular the high-level overviews are important because tapes are wear items, you only have on the order-of a hundred or two (don't remember the exact figures) full tape reads before the tape wears out, so this is something you want to go into it knowing a strategy and not making it up as you go!) Since it's complimentary to this discussion I'll link a few: https://www.cyberciti.biz/hardware/unix-linux-basic-tape-management-commands/ https://www.cyberciti.biz/hardware/unix-linux-basic-tape-man... https://databasetutorialpoint.wordpress.com/to-know-more/how-to-work-with-tape-driveslto-in-linux/ https://databasetutorialpoint.wordpress.com/to-know-more/how... https://sites.google.com/site/linuxscooter/linux/backups/tape-commands https://sites.google.com/site/linuxscooter/linux/backups/tap... https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_storage_devices/managing-tape-devices_managing-storage-devices https://access.redhat.com/documentation/en-us/red_hat_enterp... https://access.redhat.com/solutions/68115 https://access.redhat.com/solutions/68115 That is, unfortunately, essentially the apex of LTO tape documentation in 2022, as far as I can tell. Do note that in terms of tape standards, LTO-5 is an important threshold because that's where LTFS support got added, and that's the closest thing to a "normal" filesystem abstraction that's available for tape (sort of like packet-formatted CDRWs I guess, in the sense of presenting an abstraction over the raw seekable block device). There is also very little documentation on init, care, and feeding of LTFS iirc - and again, it would be nice to know any pitfalls that might cause shoeshining and tape death. Although I suppose in practice it's mostly going to get used more in a "multi session" scenario where you mostly aren't deleting files, you write till it's full and then maybe wipe the whole tape at once, and it's just a nice abstraction to allow the abstraction of "files" rather than sequential records (tape archives/TARs, in fact!) along an opaque track with no contextualization.

connorgutman 5y ago

I recently purchased a LTO-5 drive for my Gentoo-based NAS and have a few key takeaways for those who are interested. Don't buy a HP tape drive if you want to use LTFS on Linux! HPE Library & Tape Tools is prety much dead on modern Linux. Official support is only for RHE 7.x and a few versions of Suse. Building from source is a dependency nightmare that will leave you pulling hair. IBM drives have much better Linux support thanks to https://github.com/LinearTapeFileSystem/ltfs https://github.com/LinearTapeFileSystem/ltfs. That being said, IMO, you should consider ditching LTFS for good ol' TAR! It's been battle tested since 1979 and can be installed on basically anything. TAR is easy to use, well documented, and makes way more sense for linear filesystems. While drag&drop is nice and all, it really does not make sense for linear storage.

smackeyacky 5y ago
Upvote for tar! LTFS seems like an overly complex solution to a relatively simple problem that tar already solved. Treating tapes like disks and trying to run a file system on them ignores the way they work.
bsdice 5y ago
Can tar control block size and maximum file size? If you can't set those to 1 MB and 20 GB respectively, then for LTO-5 and later you will be leaving alot of performance on the table. Like half or more, check file size graph in https://www.bareos.com/wp-content/uploads/2021/05/optimizing-the-tape-speed.pdf https://www.bareos.com/wp-content/uploads/2021/05/optimizing....
- jwiz 5y ago
  idk man, tar is pretty new, they probably haven't had time to do that yet.

abbbi 5y ago

if one wants to play with virtual tape libraries, quadstorvtl is a nice solution to that: https://quadstor.com/ https://quadstor.com/ unfortunately they dont seem to have an open vcs for the source... (other than really old versions on github) other than that there is mhvtl: http://www.mhvtl.com http://www.mhvtl.com

dark-star 5y ago

PSA: Don't do a cleaning run unless your tape drive tells you to (there are SCSI sense codes for that). The tapes can pretty well assess the need (or not) for cleaning, and excessive cleaning can negatively affect the lifetime of the r/w head (the cleaning tapes are abrasive)

wolfgang42 5y ago

The mt(1) manpage describes seeking on files, records, and file marks, but doesn't explain what any of them are. What's the difference between these these options? (It sounds like file marks are stored on the tape on a special track or something, but I can't seem to find any discussion of the others.)

StillBored 5y ago
So, all from a few years old memory and its a complex interwoven mess. Lets start with tape has two types of head positioning commands, locate and space. Locate is absolute (and mt calls it seek), and space is relative. Mt is generally using space (although one can read the current position with tell then do relative space) for all the commands that aren't "seek". Hence the mt commands are things like "fsf" which is forward space file (mark), or "bsf" for back space file (mark). At some point in the past someone thought that each "file" would fit in a tape block, but then reality hit because there are limits on how large the blocks can actually be (in linux its generally the number of scatter gather entries that can fit in a page reliably). So there are filemarks, which are like "special" tape blocks without any data in them. Instead if you attempt to read over a filemark the drive returns a soft error telling you that you just tried to read a filemark. There are also "fsr" for forward space records with are just the individual blocks forming a "file". So back to seeking. If you man st, you will notice that each tape drive gets a bunch of /dev/st* aliases, which control the close behavior/etc, as well as some ioctls that match the mt commands. The two important close behaviors to remember are that if the tape is at EOD due to the last command being a write it will write a filemark, then rewind the tape unless a /dev/stXn device is being used, in which case it will leave the head position just past the FM (this is actually a bit more complex too because IIRC there may be two filemarks at EOD, and the tape position gets left between them). This allows one to do something like "for (x in *.txt); do cat $x >> /dev/st0n; done" and write a bunch of files separated by filemarks (at the default blocking size which will be slow (probably 10k), replace the cat with tar to control blocking/etc). Or if you want to read the previous file `mt -f /dev/st0n bsf 2` to back space 2 filemarks. Now, the actual data format on tape is going to be dictated by the backup utility used to write it. Some never use filemarks, some do but as a volume separator (eg tar), old ones actually put FM's between files, but that tends to be slow because it kills read perf because it takes the drive out of streaming mode whenever you either read over the filemark (not the part on man st about reading a filemark). Now you can pick which file to read via "mt -f /dev/st0 rewind; mt -f /dev/st0n fsf X; cat /dev/st0n > restore.file" There are also tape partition control commands, and tape set marks and various other options which may/may not apply to a given type of tape. Noticeably there are also density flags on the special file (some unix'es) and via mt. LTO for example doesn't have settable densities because its fixed by the physical tape in the drive. Some drives STK T10K/IBM TS11X0/3592 can upgrade the tape density/capacity when used in a newer drive. That got long...

watersb 5y ago

Everyone who cares about backups chooses a backup system design. Anyone who cares about their stuff needs to practice a full emergency RESTORE. I have met very few people who actually do that. For most systems I've seen, the first full test of the restore process is a very scary first production usage of the restore process. Which is very exciting, sure. I don't want excitement in my data management life. (I actually see weekly test of onsite backup power at the local banks, and at some large commercial kitchens. Those diesel generators are very loud. I've never seen systematic test of UPS or generators in a front-office environment.)

dragontamer 5y ago

I've come to the understanding that tape-drives are for people who need to "build a custom-sized storage solution", especially if you need capacity but not necessarily read/write speeds. A tape-drive is your read-heads. The tape is like a platter. The tape-library / jukebox is just a robotic mechanism for switching tapes into and/or out of the read-head. ---------- If you need a Petabyte of uncompressed storage, you can reach it with a tape-library consisting of 84 LTO8 tapes (12TB each). If read/write of 400MB/s is sufficient, one tape drive is sufficient. If you need faster access speeds, you buy a 2nd, 3rd, or 4th tape drive. So lets say you need 2GB/s read/write speed and a petabyte of storage. You simply get 4x LTO 8 drives, 84 LTO8 tapes, and stick them into a tape library of some kind. You then buy a certain amount of SSDs + HDDs sufficient for caching, so that you can read/write to this tape library at sufficient speeds (especially since it could be many minutes before a specific byte is accessed).

metadat 5y ago
What software is going to be gluing and integrating all these drives together into a usable interface? Does it already exist as open-source?
- dragontamer 5y ago
  LTFS (Linear Tape File System) shows each each "file" as often a pretty dumb "write-only" mechanism / append-only file system. That is: if a file A is modified, LTFS "represents" this by marking fileA as dead, then copying most of the contents of A to the end of the tape... with the few modifications in the correct location. Because tapes have so much capacity, this ends up working much better in practice than you'd expect. Especially if your backup-like software (or Apache-log files or whatever you're archiving) are largely append-only (ex: Wikipedia dumps or whatever huge data-storage requirements you're doing). ----------- Allegedly anyway, I don't really use tapes myself, but I amuse myself by looking at their specs / usability every now and then. It seems like the details of LTFS depend on each manufacturer's system, but it seems like LTFS itself is somewhat cross-compatible. Some LTFS-library docs suggest that different tapes are different folders. Others suggest that different tapes are mount/unmounted (which sends a sequence of commands to the little tape-robot to fetch the correct tape, which it does so by scanning barcodes or maybe going to correct slot) ----------- Tapes are meant to be used in a "jukebox-style" application IMO. If you need just one tape, you're doing it wrong. If you need 80-tapes, then doing it manually is obviously wrong, you should pay a bit extra for that "jukebox / tape-library" robot to fetch the right tapes automatically. > Does it already exist as open-source? Red Hat claims LTFS support. But I imagine this is only "disk-level LTFS", and not "library / robot LTFS".

watersb 5y ago

I use a cloud storage provider to back up via Arq https://arqbackup.com https://arqbackup.com But I don't expect to restore more than a few gigabytes at a time from that. It would take me a week or more to download a terabyte of data. I have very little power over internet connection speed, and there are very few alternatives here. I believe there are two different vendors providing connectivity to our town, and you can pick between four retail resellers. With those limitations, I have tested a full restore process exactly once. That's not good enough. Data at rest on LTO or offline hard disk is something I can control. Distributed offsite storage, too. Restore within 12 hours, I can do that. The downside to tape or cold disk is more in the management of hourly/daily/weekly backups: you have to provision a media rotation schedule, whereas that's sort of built into an online cloud storage service.

omgwtfbyobbq 5y ago

How durable are LTO drives? I've been thinking about moving some backups to tape because of the stated durability and good cost (~$7-$12+/tb for used LTO4 tapes+drive, depending on how much storage I get). But if the drives don't also last 15+ years, then I think the cost could be significantly higher, especially if I'm shelling out who knows what for drives a decade after their low point on the resale market.

justsomehnguy 5y ago
> LTO4 Is useless for a long-term, both from $/GB and from sanity standpoint. LTO-5 is more than 10 years old already, there is no point in trying to use LTO-4. > How durable are LTO drives? Just look at the comments here. 99% of time it just works. In 1% you throw the shit out of the window (well, you want to). Also: > Both drives and media should be kept free from airborne dust or other contaminants from packing and storage materials, paper dust, cardboard particles, printer toner dust etc.[49] And "5 Environmental" in [0] https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol4_E4/LTO5_Vol4_E4.pdf https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol4_E4/LTO5_Vo...

vajrabum 5y ago

Make sure you test restores periodically! I had a co-worker who got fired when they tried to reformat a failing disk instead of sending it out for recovery and felt safe because the user "had tape backups." Those were DLTs and perhaps today such failure would be a lot less likely but still. Just do it. Part of basic tape/backup hygiene.