12 ms·

I'd be curious to hear from any folks who actually run tape drive setups for personal use. Where did you get the gear? How cost-effective is the overall setup

I'd be curious to hear from any folks who actually run tape drive setups for personal use. Where did you get the gear? How cost-effective is the overall setup compared to spinning rust? How is driver support in popular operating systems? Is there more you'd recommend taking into consideration or learning before jumping in? Especially beyond the obvious realities of extremely high latency, of course ;)

netrap 7y ago
eBay. 1-1 HDD backups is probably close in price even in the used tape market. I use LTO-4 with FC. Works fine in Windows with Retrospect. I'd expect it to work in Linux as well. The hard part is backup software. I guess with LTO-5+ you can just directly access the tapes with LTFS, but I haven't tried that yet.
- mc32 7y ago
  Anything which supports ndmp should be good. Both bacula and Amanda among other SW support NDMP. I mean of course enterprise solutions support it too.
klodolph 7y ago
> Where did you get the gear? I never ended up using it personally, but when I was considering it, used gear on eBay was top of the list. > How cost-effective is the overall setup compared to spinning rust? For personal use, it is not even remotely cost-effective unless you are a data hoarder. The cost of the drive is simply too high. Also keep in mind that the drive can become damaged or need servicing. It only makes sense if you are dealing with e.g. large quantities of video. The article cites a minimum 140TB where tape becomes cheaper but this involves some dubious decision-making. > How is driver support in popular operating systems? I have only ever used tape with Linux + custom software so I can’t speak of software support in general. > Is there more you'd recommend taking into consideration or learning before jumping in? Especially beyond the obvious realities of extremely high latency, of course ;) The obvious reality to me here is that it just doesn’t make sense unless you’re in spitting distance of a petabyte. The only thing I can imagine someone doing personally is video, and it would have to be a lot of video like some kind of long-running documentary project that you are producing in your spare time. The other factor is to be more paranoid about mixing tape generations than the vendors suggest. Ignore the specs and stick to same-generation media and drives. Efficiently writing to a tape means filling e.g. a 20 MB/s write channel for LTO-6, or 750 MB/s for LTO-8. Can you sustain 750 MB/s? It’s not so easy in practice. You throw away tape capacity if you get buffer underruns.
- jolmg 7y ago
  > Can you sustain 750 MB/s? It’s not so easy in practice. You throw away tape capacity if you get buffer underruns. The drive doesn't pause while the buffer fills again? Considering the price of the drives, I find it very surprising that they don't do that.
  - namibj 7y ago
    No, they stop, rewind, and accelerate again, causing massive mechanical wear.
    - Yeroc 7y ago
      The higher-end tape devices are capable of running at slower speeds to better match the rate you're feeding data to it to avoid this.
      - namibj 7y ago
        Yes. But it was argued that they could just stop and wait for the buffer. As said, they do, but you don't want them doing so.
  - klodolph 7y ago
    > The drive doesn't pause while the buffer fills again? The drive does stop, and it runs at different speeds depending on how fast you are feeding it data. However, this process wastes tape capacity as you switch speeds.
    - AstralStorm 7y ago
      Thus you should pick one speed that's fast enough and stick to it.
    - jolmg 7y ago
      I don't understand how stopping or switching speeds wastes tape capacity. Are you not talking about storage capacity but some other type of capacity?
ijiiijji1 7y ago
I have, back in the day.. both floppy-controller and IDE based... with a whopping range of storage from 220 MiB to 4 GiB. Since I don't have enough karma to write another comment anonymously: Restoring from backups is a last resort for very important files deleted more than a few days ago (beyond what hourly/daily snapshots contain) or for business continuity (BCP) / disaster recovery (DR) of an entire system that cannot be recovered via automated configuration management rebuild. Snapshots to nearline storage reduce RTO for most common use-cases, and most users and systems don't generate much churn of data in a day. For recovery of important files, sending a courier out to retrieve a backup tape set from the vaulting provider implies the RTO would accept in a matter of several hours. An RTO of under an hour would imply backups would need to be stored on nearline media (commodity spinning rust) or at a backup cloud vendor like AWS Glacier or Tarsnap, which is more expensive than tape. The deciding factor of how much to rationally spend on DR/BCP solutions follows from a Business Impact Analysis (BIA) to quantify the loss of a service over a certain time, which drives RTO and RPO goals. (A common factoid of DR/BCP is "about half of businesses who lose their data go out of business within six months.") RTO - Recovery Time Objective - desire/need to recover within a certain time window RPO - Recovery Point Objective - how close to the original is the recovery
TwoNineFive 7y ago
> Where did you get the gear? I have an old Quantum DLT S4 drive and something like 18 tapes. Each tape is 800GB uncompressed and I get about 1.8x compression on average. It's an external unit connected via SCSI ultra 320. The ultra 320 SCSI card is something I picked up from ebay for like $20. The drive and tapes I got from a former consulting client who was going to trash it. It was the last of the DLT line and is similar to LTO4. Get an LTO5 if possible; it has some cool features like partitions. I only store my most critical stuff on tapes. Each weekend I do a tape swap and take the old tape to work on Monday and bring the oldest one back. I keep the tapes at work in case my house burns down or something (I lived through two house fires as a kid so I'm super neurotic about it, don't ask). The tapes are encrypted so nobody can recover from them if they get one. If my house did burn down I'd have to somehow procure another tape drive, but that would be the least of my problems. I just use a custom tar script to do it. I tried both Bacula and Amanda in the past but they were too complicated or had other issues. My own tar script did what I wanted. > How cost-effective is the overall setup compared to spinning rust? Since I got most of it for free and I already had experience with tape, it was pretty good. I've had it since 2008 so it's over ten years old and still going. I also have some UBS3 backup disks that I use for daily backups. In all this time, only one tape has died and I do a full validation of the previous backups every time I bring a tape back and re-use it. For new installs, go with some USB3 disks from Seagate or WD -- both are great, I have both. Avoid Toshiba due to some weaknesses in handling bumps and vibration during runtime due to firmware (Toshiba internal drives are great though) -- Don't ask me how I know this. Seal them into some kind of fire-proof/flood-proof safe and store somewhere that a fire/flood won't destroy it. > How is driver support in popular operating systems? I'm all linux. It works. All that matters is the SCSI card being supported. Make sure NOT to get a RAID card, regular SCSI only. The RAID cards often won't work with tape drives because the firmware is made for RAID drives, not tape. > Is there more you'd recommend taking into consideration or learning before jumping in? Especially beyond the obvious realities of extremely high latency, of course ;) So the only real advantage for me using tapes is that I got most of it for free, the tapes are more portable and lighter than a USB hard drive for carrying around. Almost everything else is a negative compared to USB hard drives. I use some UBS3 hard drives for daily backups via an rsync --link-dest script that de-duplicates everything and that works great. For $100 you can get a 4-6GB USB hard drive from WD or Seagate and for my important stuff, that works fine. I'm about to fill up my older 5GB drive -- it's down to about 200GB free space left. I already have a new 6TB drive I brought recently and when it gets down a little lower, I'll just replace it, do full backups, and put the old drive into storage for permanent archival. Data backups is a full occupational specialty, like being a DBA, network engineer, security, or web monkey. I did TSM for awhile. There's a ton of issues in the hardware, strategy, security, performance, and more. Do yourself a favor and just set up something simple that JUST WORKS and don't fuck with it too much. Do a practice recovery once and that'll teach you how good your setup is and improve.
oconnore 7y ago
For personal archival I think Blu-ray is still best. I think of it as a supplement to S3 or Dropbox. Even if you end up re-writing discs it is very cheap per GB, and 25 or 50 GB discs are comfortable for “normal” personal sizes (my entire file set that I care to keep around is around 100 GB — mostly because I’m not careful about deleting old node_modules from ancient projects I’ve done)
- jbverschoor 7y ago
  My photos/videos alone are over 300gb and growing. node_modules is shit anyway, and shouldn't be duplicated for every single project.
  - lostlogin 7y ago
    How do you store and view this? I have yet to find a good solution. My phone is crippled by the library and none of the Macs like it much either. I’d guess I was at 600gb.
    - jbverschoor 7y ago
      Very good question.. I have most of it in directory folders "yyyy-mm-dd NAME OF THE EVENT" on dropbox, which I only sync on my macmini due to the amount of storage. I'm trying to use icloud photos, because then I have them where they belong on my iPhone, but I just don't like the way it works. It's slow on my mac, but it's nice to have thumbnails locally, and originals available when needed. Althought that also doesn't work nicely when dragging. It's cool that photos groups things on event, but for me, I like to have my "camera roll", and manually make albums. That's possible, but it's ust easier to drag files around. Also, I'm looking for a solution to 'archive' photos (hide them, because I took 10 pictures of 1 thing). There was a time I wanted to write an app/service for this. I might still do that if there's enough demand. I also haven't figured out how to archive some video material. Recently I started shooting small clips, with the intention to make small 5 minute videos of my trips. As I don't want to delete the original material, I also need a place to store them, but I don't want to "browse" them.
  - 7y ago
ramraj07 7y ago
I setup a tape drive for my lab backup and in paper it was straightforward- we got a tape drive from Newegg for a few thousand bucks, put it in a machine, realized we needed some pci card, got it, realized we needed some cable, got it, realized we needed some custom drivers for some reason, got it, and then finally it was working. Started getting some tapes and backing up. Overall would not recommend unless you have hundreds of terabytes that you wanna backup yourself
mmaunder 7y ago
LTO is used in the film industry for long term archival. I own a LTO8 drive. Sourced from Maxx Digital in Orange County, CA. Talk to Matt Stone and he'll spec one for you. Tell him Mark Maunder sent you. :-) The drives are bloody expensive. Mine was around $5k. LTO8 tapes are still hard to come by even though the patent dispute is settled. This should change soon. The software to archive to tape is clunky and frustrating but it works. I use something free that integrates with Finder on Mac. I'm away from my office so dont remember what it's called. The main benefit is that the tapes last for half a century and are very cheap per TB once you have made the initial investment in the drive. Very good for long term archival of films and source footage which is what I use it for.
- Keyframe 7y ago
  Exactly what we’ve used it (when I was day-to-day in that business as well). Daily shot material stored in three copies and sent to three different physical locations and one additional copy for on-site archiving + online on hard drives. When you do a lot of those, drive price isn’t all that expensive. My major gripe with it is that you couldn’t hook it up to a USB3. Rare ones had thunderbolt, but most were that bloody external SAS thing. Not really all that great when you don’t have those controllers around. Beats the purpose of external drives.
  - mmaunder 7y ago
    I use TB3 and I'm quite happy with this. Also has a drive bay that'll fit 12TB drives I have. https://magstor.com/collections/magstor-thunderbolt-3-tape-drives/products/magstor-lto8-12tb-thunderbolt-3-tape-drive?variant=8913615093796 https://magstor.com/collections/magstor-thunderbolt-3-tape-d...
adrian_b 7y ago
I have been using in my home for several years a Quantum external LTO-7 drive, which is connected via a Serial-Atached-SCSI cable, so I also have a SAS HBA card in a computer. I am living in the European Union and I have bought the required hardware from an on-line shop from Germany, where it was cheaper. Now you can also buy it easily e.g. from Amazon. The cost was around 3000 EUR. The tape cost is about 10 EUR per terabyte. Previously I was using HDDs for archiving. I have not followed closely the evolution of the HDD price, but looking now on Amazon I see about 30 EUR per terabyte. I have a data archive of about 120 TB, but I am paranoid because of past data losses due to defective HDDs/CDs, so I store 3 copies of each tape (in different places). That means that by using tape instead of HDDs, I have already saved 20 * 360 - 3000 = 4200 EUR. Someone who would store just 2 copies of each tape (the minimum rational value), would need to have a data archive of at least 75 TB to recover the initial cost. Nevertheless saving money is not everything, you also gain peace-of-mind because the chances of the tape becoming defective just from storing are negligible compared to a HDD, which if not used for a very long time might never start spinning again. With tapes you must transfer the data only when you can no longer buy drives that would read them. With HDDs you should better replace them before their warranty expires, as I have seen too many that died immediately after that. The tapes work OK both in Linux and in FreeBSD, but the FreeBSD utilities are more convenient, so I have connected the tape drive to a FreeBSD server and I transfer the data to be archived through 10 Gb/s Ethernet, which is faster than the 6 Gb/s SAS, so it is not a bottleneck. The writing speed of LTO-7 is 3 Gb/s, so that is the bottleneck anyway, not the interfaces. That speed is much faster than of any HDD of up to 8 TB that I have ever used, but I have not checked the latest HDD models to see if they have improved their speed to levels comparable to tapes. Using tape is very simple and retrieving any file (I have a tape index on my SSDs) takes at most 3 minutes, including the time to manually start the drive and inserting a tape in the drive. The maximum seek time is about 2 minutes. When I had a HDD archive, there was still the same time for manually selecting a HDD and inserting it in the HDD rack. For retrieving very large files, the tape can even be faster, due to the faster sequential speed. The conclusion is that if you have data of at least several tens of terabytes and/or if you value that data enough to want to avoid any chance of losing it, than magnetic tape remains the best solution for now.
Annatar 7y ago
Got it on ebay. There are great deals to be had if one knows the technology, partly because there are very few of us left now who know it, so extremely expensive equipment goes for pennies on the dollar. I use self-system engineered (read: self OS packaged) Bacula software for backups with a custom developed shell program which interfaces with Oracle RMAN to do backups. I use Solaris 10 on sparc, which also goes for pennies on the dollar nowadays since there are few of us left who understand how to run it and even fewer who understand how to squeeze the most out of that hardware + software combination. The tape library shows up as character and block files under /dev/rmt/ and the robotic arm with the barcode scanner under /dev/changer/, nothing special or fancy about it. There is another piece of freeware I compiled and packaged (whose name I forgot) which controls the robot arm and the scanner, also by issuing standard SCSI commands to /dev/changer/ files, nothing fancy or special about it. The tape library has enough capacity for full backups with three month retention policy, so the only thing I have to do is replace a worn out tape cartridge roughly once in three years. Backups are daily. The entire solution ran me around $800 (server + differential SCSI controller + tape library) and the tapes ran me $500, over the span of 12 years.
TheCondor 7y ago
LTO6 here, I bought it at Newegg. It’s pricy but having a 2.5TB and fairly durable cartridge is nice for my needs. I just dropped one earlier this week, no big deal though. I use LTFS on it, it has latency but it’s faster than you expect and once you’re streaming on or off of it, it’s downright quick. Historically, I burned optical media for this but the size of tape makes it much easier. It’s on a SAS card in a linux server, I literally plugged it in and it worked with open SUSE. I forget if the tape tools were already there or not but the old school Unix stuff just worked. I installed the ltfs tools and it all worked without any tricks. It’s a quantum drive. I did experiment with tar initially but ltfs makes it radically more usable, radically. Pre-tape, I used disks, rotated them and kept them in a safe. Burned optical discs of my most important stuff. I had two experiences that caused me to change: I had a failure and went to do a recovery and in either haste or anxiety (I put this on me, I should have paused and took more time but I was rushing) I knocked the most recent drive off my desk and effectively killed it. The other was a bit more subtle (oddly, I have only had 3 backup recovery situations and I messed up 2 of them) but my “backup” was rsync to disk on cron and a backup started during restore and began to “remove” my good backup because the master didn’t have that restored yet. There are a lot of lessons in there, most significantly, “backup” is a different thing than live access to data, you want it versioned and somewhat fixed. Rsync isnt it. You don’t want to accidentally delete your backup. Tape switches your mindset on that stuff. I’m not going to pretend it’s cost effective for my needs (although I have a lot of hd+ video of my kids) but it’s durable, it’s purpose built and cold storage can’t really be hacked. There’s no monthly fees, no risk of a cloud provider sunsetting my tier of service, and 25T of tape is like $250. Drive MTBF numbers are big enough to be abstract and yet you will likely encounter it at some point, tape has these practical numbers that are ultra conservative and come from decades of experience; you sort of keep it in mind when you’ve used a tape a dozen times or so. That being said, when the drive flakes, I’m in for an expensive upgrade/replacement and when I do upgrade, I should copy my tapes to new ones with the new format which will take time and money. With SMR, it’s almost like discs are taking on tape like qualities and it’s not hard to find stories about people getting crap performance on their new disks. It’s not sexy but I feel very confident that I’ll be able to get my data if I need it. It’s really easy to store a tape in your safe deposit box or at your parents house too, they won’t accidentally kill it by knocking it off a shelf.
tomatocracy 7y ago
I also run my own tape setup for personal backups. On where to source etc - I’d agree with others that eBay is a good option for both drives and media (but as always check what exactly you’re buying and make sure you’re comfortable with the seller). You’ll almost certainly need a SCSI or FC interface too so don’t forget that. I’d recommend going with external standalone drives rather than either internal or sled mounted for this type of setup unless you’ve got a good reason to do otherwise as that means you only need to power the drive when you’re using it which can cut down on wear etc. I’ve only ever read bad things about mixing generations despite the fact that the drives are capable of this (eg LTO5 tape in LTO6/7 drives) though I’ve not done this personally - but I wouldn’t recommend doing so based on this. Don’t be tempted to buy really old generations as if you ever need to restore in a disaster where you only have the tapes, you then have the nightmare of sourcing a new old drive too which by that point might be a lot harder. You’ll want to think carefully about software setup. You’re probably going to wind up using some form of custom scripts unless you can put in the time to set up bacula properly and are sure it works for you (bare metal restore can be a tricky aspect of any system to verify it works properly). If you script your own then make sure you also understand the drawbacks of tar. tar as a tool is highly compatible which is good but is poor at recording incremental backups (there are some custom extensions which gnu tar has but this reduces compatibility) if you want to go with a more complex schedule than full backup every time. Tar also makes it difficult to seek to just the part of the archive that you need to get a single file. Again there are gnu extensions for this but this isn’t the best solution. Finally, tar then gzip means if your gzip stream gets corrupted then you might lose quite a bit of the tar archive so it’s not very resilient. You could rely on the drive compression but you might also consider other solutions, especially ones which also include checksumming. Consider either scripting around these or better what other tools could take its place (eg I have used dar in some of my setup as it handles this better but then you need to think about keeping a working binary around which can unpack your archives - I put a static compiled binary in my archive sets just in case). Finally if you go down the tape route then consider strongly spooling your stream which will be written out to tape onto disk first, to give yourself maximum chance of getting the more consistent throughput onto tape which has benefits in terms of tape reliability (and may indeed wind up being quicker overall). That also allows you to easily write multiple copies of the same tape if you intend to send one or more offsite and keep one locally. Where this setup works well for me is that when you start wanting many copies of your backups (offsite, multiple point in time full backups etc) then the cost effectiveness becomes a lot better than the equivalent with disk. I also trust tape to survive longer term on a shelf than disk.

netrap 7y ago

eBay. 1-1 HDD backups is probably close in price even in the used tape market. I use LTO-4 with FC. Works fine in Windows with Retrospect. I'd expect it to work in Linux as well. The hard part is backup software. I guess with LTO-5+ you can just directly access the tapes with LTFS, but I haven't tried that yet.

mc32 7y ago
Anything which supports ndmp should be good. Both bacula and Amanda among other SW support NDMP. I mean of course enterprise solutions support it too.

klodolph 7y ago

> Where did you get the gear? I never ended up using it personally, but when I was considering it, used gear on eBay was top of the list. > How cost-effective is the overall setup compared to spinning rust? For personal use, it is not even remotely cost-effective unless you are a data hoarder. The cost of the drive is simply too high. Also keep in mind that the drive can become damaged or need servicing. It only makes sense if you are dealing with e.g. large quantities of video. The article cites a minimum 140TB where tape becomes cheaper but this involves some dubious decision-making. > How is driver support in popular operating systems? I have only ever used tape with Linux + custom software so I can’t speak of software support in general. > Is there more you'd recommend taking into consideration or learning before jumping in? Especially beyond the obvious realities of extremely high latency, of course ;) The obvious reality to me here is that it just doesn’t make sense unless you’re in spitting distance of a petabyte. The only thing I can imagine someone doing personally is video, and it would have to be a lot of video like some kind of long-running documentary project that you are producing in your spare time. The other factor is to be more paranoid about mixing tape generations than the vendors suggest. Ignore the specs and stick to same-generation media and drives. Efficiently writing to a tape means filling e.g. a 20 MB/s write channel for LTO-6, or 750 MB/s for LTO-8. Can you sustain 750 MB/s? It’s not so easy in practice. You throw away tape capacity if you get buffer underruns.

jolmg 7y ago
> Can you sustain 750 MB/s? It’s not so easy in practice. You throw away tape capacity if you get buffer underruns. The drive doesn't pause while the buffer fills again? Considering the price of the drives, I find it very surprising that they don't do that.
- namibj 7y ago
  No, they stop, rewind, and accelerate again, causing massive mechanical wear.
  - Yeroc 7y ago
    The higher-end tape devices are capable of running at slower speeds to better match the rate you're feeding data to it to avoid this.
    - namibj 7y ago
      Yes. But it was argued that they could just stop and wait for the buffer. As said, they do, but you don't want them doing so.
- klodolph 7y ago
  > The drive doesn't pause while the buffer fills again? The drive does stop, and it runs at different speeds depending on how fast you are feeding it data. However, this process wastes tape capacity as you switch speeds.
  - AstralStorm 7y ago
    Thus you should pick one speed that's fast enough and stick to it.
  - jolmg 7y ago
    I don't understand how stopping or switching speeds wastes tape capacity. Are you not talking about storage capacity but some other type of capacity?

ijiiijji1 7y ago

I have, back in the day.. both floppy-controller and IDE based... with a whopping range of storage from 220 MiB to 4 GiB. Since I don't have enough karma to write another comment anonymously: Restoring from backups is a last resort for very important files deleted more than a few days ago (beyond what hourly/daily snapshots contain) or for business continuity (BCP) / disaster recovery (DR) of an entire system that cannot be recovered via automated configuration management rebuild. Snapshots to nearline storage reduce RTO for most common use-cases, and most users and systems don't generate much churn of data in a day. For recovery of important files, sending a courier out to retrieve a backup tape set from the vaulting provider implies the RTO would accept in a matter of several hours. An RTO of under an hour would imply backups would need to be stored on nearline media (commodity spinning rust) or at a backup cloud vendor like AWS Glacier or Tarsnap, which is more expensive than tape. The deciding factor of how much to rationally spend on DR/BCP solutions follows from a Business Impact Analysis (BIA) to quantify the loss of a service over a certain time, which drives RTO and RPO goals. (A common factoid of DR/BCP is "about half of businesses who lose their data go out of business within six months.") RTO - Recovery Time Objective - desire/need to recover within a certain time window RPO - Recovery Point Objective - how close to the original is the recovery

TwoNineFive 7y ago

> Where did you get the gear? I have an old Quantum DLT S4 drive and something like 18 tapes. Each tape is 800GB uncompressed and I get about 1.8x compression on average. It's an external unit connected via SCSI ultra 320. The ultra 320 SCSI card is something I picked up from ebay for like $20. The drive and tapes I got from a former consulting client who was going to trash it. It was the last of the DLT line and is similar to LTO4. Get an LTO5 if possible; it has some cool features like partitions. I only store my most critical stuff on tapes. Each weekend I do a tape swap and take the old tape to work on Monday and bring the oldest one back. I keep the tapes at work in case my house burns down or something (I lived through two house fires as a kid so I'm super neurotic about it, don't ask). The tapes are encrypted so nobody can recover from them if they get one. If my house did burn down I'd have to somehow procure another tape drive, but that would be the least of my problems. I just use a custom tar script to do it. I tried both Bacula and Amanda in the past but they were too complicated or had other issues. My own tar script did what I wanted. > How cost-effective is the overall setup compared to spinning rust? Since I got most of it for free and I already had experience with tape, it was pretty good. I've had it since 2008 so it's over ten years old and still going. I also have some UBS3 backup disks that I use for daily backups. In all this time, only one tape has died and I do a full validation of the previous backups every time I bring a tape back and re-use it. For new installs, go with some USB3 disks from Seagate or WD -- both are great, I have both. Avoid Toshiba due to some weaknesses in handling bumps and vibration during runtime due to firmware (Toshiba internal drives are great though) -- Don't ask me how I know this. Seal them into some kind of fire-proof/flood-proof safe and store somewhere that a fire/flood won't destroy it. > How is driver support in popular operating systems? I'm all linux. It works. All that matters is the SCSI card being supported. Make sure NOT to get a RAID card, regular SCSI only. The RAID cards often won't work with tape drives because the firmware is made for RAID drives, not tape. > Is there more you'd recommend taking into consideration or learning before jumping in? Especially beyond the obvious realities of extremely high latency, of course ;) So the only real advantage for me using tapes is that I got most of it for free, the tapes are more portable and lighter than a USB hard drive for carrying around. Almost everything else is a negative compared to USB hard drives. I use some UBS3 hard drives for daily backups via an rsync --link-dest script that de-duplicates everything and that works great. For $100 you can get a 4-6GB USB hard drive from WD or Seagate and for my important stuff, that works fine. I'm about to fill up my older 5GB drive -- it's down to about 200GB free space left. I already have a new 6TB drive I brought recently and when it gets down a little lower, I'll just replace it, do full backups, and put the old drive into storage for permanent archival. Data backups is a full occupational specialty, like being a DBA, network engineer, security, or web monkey. I did TSM for awhile. There's a ton of issues in the hardware, strategy, security, performance, and more. Do yourself a favor and just set up something simple that JUST WORKS and don't fuck with it too much. Do a practice recovery once and that'll teach you how good your setup is and improve.

oconnore 7y ago

For personal archival I think Blu-ray is still best. I think of it as a supplement to S3 or Dropbox. Even if you end up re-writing discs it is very cheap per GB, and 25 or 50 GB discs are comfortable for “normal” personal sizes (my entire file set that I care to keep around is around 100 GB — mostly because I’m not careful about deleting old node_modules from ancient projects I’ve done)

jbverschoor 7y ago
My photos/videos alone are over 300gb and growing. node_modules is shit anyway, and shouldn't be duplicated for every single project.
- lostlogin 7y ago
  How do you store and view this? I have yet to find a good solution. My phone is crippled by the library and none of the Macs like it much either. I’d guess I was at 600gb.
  - jbverschoor 7y ago
    Very good question.. I have most of it in directory folders "yyyy-mm-dd NAME OF THE EVENT" on dropbox, which I only sync on my macmini due to the amount of storage. I'm trying to use icloud photos, because then I have them where they belong on my iPhone, but I just don't like the way it works. It's slow on my mac, but it's nice to have thumbnails locally, and originals available when needed. Althought that also doesn't work nicely when dragging. It's cool that photos groups things on event, but for me, I like to have my "camera roll", and manually make albums. That's possible, but it's ust easier to drag files around. Also, I'm looking for a solution to 'archive' photos (hide them, because I took 10 pictures of 1 thing). There was a time I wanted to write an app/service for this. I might still do that if there's enough demand. I also haven't figured out how to archive some video material. Recently I started shooting small clips, with the intention to make small 5 minute videos of my trips. As I don't want to delete the original material, I also need a place to store them, but I don't want to "browse" them.
- 7y ago

ramraj07 7y ago

I setup a tape drive for my lab backup and in paper it was straightforward- we got a tape drive from Newegg for a few thousand bucks, put it in a machine, realized we needed some pci card, got it, realized we needed some cable, got it, realized we needed some custom drivers for some reason, got it, and then finally it was working. Started getting some tapes and backing up. Overall would not recommend unless you have hundreds of terabytes that you wanna backup yourself

mmaunder 7y ago

LTO is used in the film industry for long term archival. I own a LTO8 drive. Sourced from Maxx Digital in Orange County, CA. Talk to Matt Stone and he'll spec one for you. Tell him Mark Maunder sent you. :-) The drives are bloody expensive. Mine was around $5k. LTO8 tapes are still hard to come by even though the patent dispute is settled. This should change soon. The software to archive to tape is clunky and frustrating but it works. I use something free that integrates with Finder on Mac. I'm away from my office so dont remember what it's called. The main benefit is that the tapes last for half a century and are very cheap per TB once you have made the initial investment in the drive. Very good for long term archival of films and source footage which is what I use it for.

Keyframe 7y ago
Exactly what we’ve used it (when I was day-to-day in that business as well). Daily shot material stored in three copies and sent to three different physical locations and one additional copy for on-site archiving + online on hard drives. When you do a lot of those, drive price isn’t all that expensive. My major gripe with it is that you couldn’t hook it up to a USB3. Rare ones had thunderbolt, but most were that bloody external SAS thing. Not really all that great when you don’t have those controllers around. Beats the purpose of external drives.
- mmaunder 7y ago
  I use TB3 and I'm quite happy with this. Also has a drive bay that'll fit 12TB drives I have. https://magstor.com/collections/magstor-thunderbolt-3-tape-drives/products/magstor-lto8-12tb-thunderbolt-3-tape-drive?variant=8913615093796 https://magstor.com/collections/magstor-thunderbolt-3-tape-d...

adrian_b 7y ago

I have been using in my home for several years a Quantum external LTO-7 drive, which is connected via a Serial-Atached-SCSI cable, so I also have a SAS HBA card in a computer. I am living in the European Union and I have bought the required hardware from an on-line shop from Germany, where it was cheaper. Now you can also buy it easily e.g. from Amazon. The cost was around 3000 EUR. The tape cost is about 10 EUR per terabyte. Previously I was using HDDs for archiving. I have not followed closely the evolution of the HDD price, but looking now on Amazon I see about 30 EUR per terabyte. I have a data archive of about 120 TB, but I am paranoid because of past data losses due to defective HDDs/CDs, so I store 3 copies of each tape (in different places). That means that by using tape instead of HDDs, I have already saved 20 * 360 - 3000 = 4200 EUR. Someone who would store just 2 copies of each tape (the minimum rational value), would need to have a data archive of at least 75 TB to recover the initial cost. Nevertheless saving money is not everything, you also gain peace-of-mind because the chances of the tape becoming defective just from storing are negligible compared to a HDD, which if not used for a very long time might never start spinning again. With tapes you must transfer the data only when you can no longer buy drives that would read them. With HDDs you should better replace them before their warranty expires, as I have seen too many that died immediately after that. The tapes work OK both in Linux and in FreeBSD, but the FreeBSD utilities are more convenient, so I have connected the tape drive to a FreeBSD server and I transfer the data to be archived through 10 Gb/s Ethernet, which is faster than the 6 Gb/s SAS, so it is not a bottleneck. The writing speed of LTO-7 is 3 Gb/s, so that is the bottleneck anyway, not the interfaces. That speed is much faster than of any HDD of up to 8 TB that I have ever used, but I have not checked the latest HDD models to see if they have improved their speed to levels comparable to tapes. Using tape is very simple and retrieving any file (I have a tape index on my SSDs) takes at most 3 minutes, including the time to manually start the drive and inserting a tape in the drive. The maximum seek time is about 2 minutes. When I had a HDD archive, there was still the same time for manually selecting a HDD and inserting it in the HDD rack. For retrieving very large files, the tape can even be faster, due to the faster sequential speed. The conclusion is that if you have data of at least several tens of terabytes and/or if you value that data enough to want to avoid any chance of losing it, than magnetic tape remains the best solution for now.

Annatar 7y ago

Got it on ebay. There are great deals to be had if one knows the technology, partly because there are very few of us left now who know it, so extremely expensive equipment goes for pennies on the dollar. I use self-system engineered (read: self OS packaged) Bacula software for backups with a custom developed shell program which interfaces with Oracle RMAN to do backups. I use Solaris 10 on sparc, which also goes for pennies on the dollar nowadays since there are few of us left who understand how to run it and even fewer who understand how to squeeze the most out of that hardware + software combination. The tape library shows up as character and block files under /dev/rmt/ and the robotic arm with the barcode scanner under /dev/changer/, nothing special or fancy about it. There is another piece of freeware I compiled and packaged (whose name I forgot) which controls the robot arm and the scanner, also by issuing standard SCSI commands to /dev/changer/ files, nothing fancy or special about it. The tape library has enough capacity for full backups with three month retention policy, so the only thing I have to do is replace a worn out tape cartridge roughly once in three years. Backups are daily. The entire solution ran me around $800 (server + differential SCSI controller + tape library) and the tapes ran me $500, over the span of 12 years.

TheCondor 7y ago

LTO6 here, I bought it at Newegg. It’s pricy but having a 2.5TB and fairly durable cartridge is nice for my needs. I just dropped one earlier this week, no big deal though. I use LTFS on it, it has latency but it’s faster than you expect and once you’re streaming on or off of it, it’s downright quick. Historically, I burned optical media for this but the size of tape makes it much easier. It’s on a SAS card in a linux server, I literally plugged it in and it worked with open SUSE. I forget if the tape tools were already there or not but the old school Unix stuff just worked. I installed the ltfs tools and it all worked without any tricks. It’s a quantum drive. I did experiment with tar initially but ltfs makes it radically more usable, radically. Pre-tape, I used disks, rotated them and kept them in a safe. Burned optical discs of my most important stuff. I had two experiences that caused me to change: I had a failure and went to do a recovery and in either haste or anxiety (I put this on me, I should have paused and took more time but I was rushing) I knocked the most recent drive off my desk and effectively killed it. The other was a bit more subtle (oddly, I have only had 3 backup recovery situations and I messed up 2 of them) but my “backup” was rsync to disk on cron and a backup started during restore and began to “remove” my good backup because the master didn’t have that restored yet. There are a lot of lessons in there, most significantly, “backup” is a different thing than live access to data, you want it versioned and somewhat fixed. Rsync isnt it. You don’t want to accidentally delete your backup. Tape switches your mindset on that stuff. I’m not going to pretend it’s cost effective for my needs (although I have a lot of hd+ video of my kids) but it’s durable, it’s purpose built and cold storage can’t really be hacked. There’s no monthly fees, no risk of a cloud provider sunsetting my tier of service, and 25T of tape is like $250. Drive MTBF numbers are big enough to be abstract and yet you will likely encounter it at some point, tape has these practical numbers that are ultra conservative and come from decades of experience; you sort of keep it in mind when you’ve used a tape a dozen times or so. That being said, when the drive flakes, I’m in for an expensive upgrade/replacement and when I do upgrade, I should copy my tapes to new ones with the new format which will take time and money. With SMR, it’s almost like discs are taking on tape like qualities and it’s not hard to find stories about people getting crap performance on their new disks. It’s not sexy but I feel very confident that I’ll be able to get my data if I need it. It’s really easy to store a tape in your safe deposit box or at your parents house too, they won’t accidentally kill it by knocking it off a shelf.

tomatocracy 7y ago

I also run my own tape setup for personal backups. On where to source etc - I’d agree with others that eBay is a good option for both drives and media (but as always check what exactly you’re buying and make sure you’re comfortable with the seller). You’ll almost certainly need a SCSI or FC interface too so don’t forget that. I’d recommend going with external standalone drives rather than either internal or sled mounted for this type of setup unless you’ve got a good reason to do otherwise as that means you only need to power the drive when you’re using it which can cut down on wear etc. I’ve only ever read bad things about mixing generations despite the fact that the drives are capable of this (eg LTO5 tape in LTO6/7 drives) though I’ve not done this personally - but I wouldn’t recommend doing so based on this. Don’t be tempted to buy really old generations as if you ever need to restore in a disaster where you only have the tapes, you then have the nightmare of sourcing a new old drive too which by that point might be a lot harder. You’ll want to think carefully about software setup. You’re probably going to wind up using some form of custom scripts unless you can put in the time to set up bacula properly and are sure it works for you (bare metal restore can be a tricky aspect of any system to verify it works properly). If you script your own then make sure you also understand the drawbacks of tar. tar as a tool is highly compatible which is good but is poor at recording incremental backups (there are some custom extensions which gnu tar has but this reduces compatibility) if you want to go with a more complex schedule than full backup every time. Tar also makes it difficult to seek to just the part of the archive that you need to get a single file. Again there are gnu extensions for this but this isn’t the best solution. Finally, tar then gzip means if your gzip stream gets corrupted then you might lose quite a bit of the tar archive so it’s not very resilient. You could rely on the drive compression but you might also consider other solutions, especially ones which also include checksumming. Consider either scripting around these or better what other tools could take its place (eg I have used dar in some of my setup as it handles this better but then you need to think about keeping a working binary around which can unpack your archives - I put a static compiled binary in my archive sets just in case). Finally if you go down the tape route then consider strongly spooling your stream which will be written out to tape onto disk first, to give yourself maximum chance of getting the more consistent throughput onto tape which has benefits in terms of tape reliability (and may indeed wind up being quicker overall). That also allows you to easily write multiple copies of the same tape if you intend to send one or more offsite and keep one locally. Where this setup works well for me is that when you start wanting many copies of your backups (offsite, multiple point in time full backups etc) then the cost effectiveness becomes a lot better than the equivalent with disk. I also trust tape to survive longer term on a shelf than disk.