11 ms·

AWS Tape Gateway

140 points by brudgers 4y ago

bob1029 4y ago
AWS doing tapes like this feels weird to me. The physicality of the thing is kind of the point in my mind. Virtualizing the last line of defense seems wrong, but perhaps I could be convinced otherwise. Why doesn't Iron Mountain or some other competitor offer a service like this?
- sithadmin 4y ago
  Iron Mountain does have a service that will pick up physical tape and offload to other storage, but nothing like AWS's virtual tape capabilities. FWIW, Amazon isn't 'doing tapes' with this product. It's just a compatibility shim laid on top of object storage so that legacy tape backup/archival systems can eliminate physical tape libraries.
SCdF 4y ago
Dumb question, and I'm guessing this is a "if you don't know it's not for you" situation, but what is the point of a virtual tape? Isn't the point of a tape that it's not virtual? Or is this more replicating tape software apis (WINE/proton style) so you can get rid of physical tapes (because you no longer care about their physicality) without having to change your backup strategy?
- mbutler4 4y ago
  From the perspective of the AWS solution, this is a way of giving on-premises hardware an option versus Iron Mountain or similar for archiving to offsite for DR/BC purposes. Since AWS' storage options are typically pretty insane SLAs, this is acceptable (and certainly not much worse than having old LTO tapes in a storage locker). VTLs have been around for a long, long time. Initially, they were a way of speeding up tape by writing to a much faster media, and allowing for deduplication on the media to reduce storage costs. This was, of course, in the days of on-prem arrays and storage. Over time, LTO (the dominant tape format, I think we're on LTO-9?) became denser and faster, and the speeds at which you could write to a single drive would outpace the line speeds. Given that, methods were introduced to interleave multiple writers to a single drive/cart, and to leverage the drives speed. One thing you don't want on a tape drive is "shoeshining", this is the effect when the drive has no data to write, but the momentum of the reels in the drive mechanism carry the media forward, past the last written segment, past the write head. The drive must then backup and re index to find where it left off while queuing whatever data is incoming. Well, as VTLs were built initially to handle speed of writes and dedup, once the tape drives outpaced those issues, VTL became problematic. It was basically taking a randomly writeable media and turning it into a serial storage mechanism. This really wasn't much use, and VTL started to fade. Many solutions were built, one from NetBackup was OpenStorage; this was an API that a storage vendor could write a plugin for and allow NetBackup to write to their storage as intended: multiple writer, multiple access. Allowing for treatment of a deduplicable storage media as a filesystem versus an emulated tape drive. This API approach allowed for plugins to be constructed to talk to any storage, including cloud. AWS spoke with NetBackup at one point, intending to build a storage mechanism for a very solid solution: allow on-prem customers to export their data to AWS storage, AND to recover that data BACK from AWS storage (think of the data transfer fees you would get out of that!!). When they looked at the options, they decided to build the VTL gateway. I'm not sure of how or where the decision was made, but it seemed a bit odd at the time. As VTLs were fading and the limitations of emulating tape would lead to a caching requirement that might be bad, but with line speeds for interconnect to AWS making the caching requirements even worse. I'm not quire sure how that was done, but I will say, a VTL will always be a stable, solid thing, that tends to simply run and do what was intended; much like a traditional tape drive. As for some of the other discussion here, about reading old tapes and storage longevity; I've seen tapes kept for decades that were readable, Iron Mountain is still in business for a reason. Note that THEY don't have a solid cloud offering, but really, if you had everyones old data sitting around on tapes, and could charge not only for data delivery but media conversion at DR time, wouldn't you? And the issue with old tapes isn't so much the storage in that situation, it's the hardware. LTO only reads 2 generations backward (IIRC), and can only write one gen forwards. So, for your LTO-6 drive, it can read LTO-4 to 6, and can write LTO-6 to 7, again if memory serves. So, finding the Apollo code tapes on a format tape where the only possible drives available to read it live in a museum and may or may not actually work, make the fact that they recovered that source code amazing. Not to mention the software needed to be built to interface to hardware that the last person to build a "driver" for likely is sitting in a nice retirement joint in Sarasota.
- deleted 4y ago
  [deleted]
- thedougd 4y ago
  It's the latter. In a large enterprise, the backup configuration can be wildly complicated, a matrix of systems, schedules, slas, etc. Just reconfigure your backup software to use this new virtual tape device and you're on your way.
  - logifail 4y ago
    > Just reconfigure your backup software to use this new virtual tape device and you're on your way Isn't the whole point of tape that it is a physical thing, and may be taken offsite in a truck / stored in a vault, and all that jazz. "Just reconfiguring" your backup software sounds like a business might just bypass all that without necessarily realising the consequences. Then "we got hacked and our local backups are gone" -> "restore from offsite tape" -> "oh, actually there are no tapes, it's all the cloud now" -> ... ??
    - orf 4y ago
      Then "we got hacked and our local backups are gone" -> "restore from offsite tape" -> "oh, actually all the tapes where stored at the wrong temperature or humidity and now they are fucked" -> ... ??
    - kube-system 4y ago
      AWS's hard disks are a physical thing that is offsite
      - logifail 4y ago
        > AWS's hard disks are a physical thing that is offsite Taking tapes to a secure offsite location means they're air-gapped from the source data, so even in a situation where an entire network is compromised, remotely-stored tapes can't be wiped/encrypted. I'm sure AWS has thought about this issue while designing Tape Gateway, but it's not clear to me how it could know whether a request to retrieve and overwrite a virtual tape was legitimate or not.
        4y ago
- sithadmin 4y ago
  The gateway appliance is fairly compatible with legacy backup solutions, which makes it a great drop-in replacement for physical tape library systems. There are certainly better backup methods available these days (though it's hard to beat the durability and cost of LTO tape for long-term archival), but I've seen AWS's virtual tape used as a good stopgap while other backup/recovery solutions are still a ways out an org's infrastructure roadmap.
  - tablespoon 4y ago
    > though it's hard to beat the durability and cost of LTO tape for long-term archival I'm not entirely comfortable with the trend towards ever more esoteric and seemingly "all or nothing" technologies. Tape (and other physical things) may have their downsides, but I think there's something to be said for something that could (theoretically) be forgotten in a closet and read 50 years later. Likewise, AM radio may not be the best quality, but it seems like you can cover more area with a single installation than any other communication technology, which might come in handy after a serious disaster like a nuclear war (e.g. crank up the nighttime power of some undamaged countryside station running on generators to tell survivors where it's still safe).
    - vbezhenar 4y ago
      It's very unlikely that you'll be able to read modern tape from closet in 50 years. Tape requires very precise temperature and humidity for storage. Otherwise all bets are off.
      - linksnapzz 4y ago
        Only one datapoint; but I’ve seen an FPS/Celerity minicomputer boot off of a Qic-20 tape that’d been stored in a non climate controlled Albuquerque storage unit for 19 years…
- brudgers 4y ago
  Even for a new strategy, using a tape abstraction might simplify the design and/or implementation because backup tools and culture have their roots in tape. There's a sense in which tape is water in which backups swim.
- shrike 4y ago
  The VTL was invented because, at the time (2012ish), none of the largest enterprise backup solutions had a good S3 interface and none supported Glacier. After talking with the backup software vendors (Spectrum, Tivoli, Symantec, Commvault, etc) it became clear that adding another backup target wasn't something we (AWS) could get them to prioritize, for perfectly reasonable reasons. We could (and did) apply pressure via our shared customers, even then they estimated it would take years. The fastest way to enable large enterprise access to S3 and Glacier for backups was to meet them where they were. We did this by virtualizing a tape library. Background: I'm one of the original inventors - https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/10013166 https://image-ppubs.uspto.gov/dirsearch-public/print/downloa.... I am no longer with AWS.
  - deleted 4y ago
    [deleted]
  - sshagent 4y ago
    You did such a great job they still try and steer people this route. I've multiple times mentioned "you know we don't need to emulate tape drives any more"
  - free-ideas 4y ago
    VTL has been around since the 90’s https://en.m.wikipedia.org/wiki/Virtual_tape_library https://en.m.wikipedia.org/wiki/Virtual_tape_library. This is a fantastic example of how broken the patent system is. This is not an invention it’s a good implementation of a well understood problem. It is also priced ridiculously - it is at least an order of magnitude more expensive than operating a physical library and remote physical storage that is beyond cyber threat by virtue of it being disconnected. A real backup is offline and offsite.
  - Twirrim 4y ago
    You did such a good job, that when I worked on Glacier (2013-2016), our interactions with the team running VTL was already pretty minimal. It was one of the least problematical things that interacted with Glacier. Some of the solutions that external vendors produced were nightmarish and left customers up the creek without a paddle in a disturbing number of situations.
glasss 4y ago
My first and only experience with tape backups was in 2014 working with the BBB of Chicago. They were using tape backups, and there I learned that almost no company that small should be using tapes. Unless you have one or two people dedicated to handling that plus whatever other backup solutions you have in place, it won't end well.
- robohoe 4y ago
  I hear you. I had to manage a Quantum LTO4 tape library with Bacula back in early 2010s at a small company. Yeah that was a fun experience (not) and I'm glad I don't have to deal with them. Good riddance. Just getting the labeling right and making sure that the tapes were writable was a pain.
  - guenthert 4y ago
    But that is Bacula fault isn't it? I used it myself @home and couldn't help thinking that there must be a better way. At work Commvault was used iirc, but others were in charge of that. I evaluated Tivoli SM, but that's more than 15 years ago and I barely remember. TSM and Bacula share some concepts, but the latter appears quite hackish. Not that confidence inspiring.
- jedberg 4y ago
  Interesting. I wonder if tapes got more complicated or if everything else just got easier. The last time I dealt with tapes was in the 1990s, and it was dead simple. We used tape backups at a company with only 150 people. I was the most junior admin so I had to be the tape monkey, but it was easy. Just run backups regularly, change the tape every couple of days and label it, ship a stack to the offsite storage and request the old stack back, and then do a test restore each time a stack came back. The whole thing took maybe two hours a week.
  - glasss 4y ago
    Everything else was definitely easier by comparison, but the situation was more about the one internal guy responsible for it was also doing all of the normal IT dept functions, so being the tape monkey was at the bottom of the list - for better or worse. When you're juggling 12 plates, a couple are gonna fall.
jmclnx 4y ago
Having to restore data from Mag Tapes over decades and getting CRC errors maybe 20% of the time, this cannot possibly be worse.
- Tepix 4y ago
  But vastly more expensive, that's for sure.
  - Nexxxeh 4y ago
    Depends on the cost of a potential 20%+ loss of data.
- shiftpgdn 4y ago
  You needed to run a cleaner through that tape device.
  - jmclnx 4y ago
    No kidding :) But the company that had the issue (I left a bit ago) believed in "cheaper the better". Cleaners ? That place would expect you to use soap from the restroom :)
api 4y ago
Anyone considering putting giant amounts of data into AWS or most other big clouds should be sure to do the numbers on retrieval. All the big clouds have "roach motel pricing" -- data is free or very cheap to put in, and expensive to get out. Outbound bandwidth costs from AWS are astronomical, so make sure you are not going to need to stream down all that data or move it any time soon or that you have compared that cost to your in-house solution.
- cube00 4y ago
  This kind of unbalanced pricing also acts as a disincentive to regularly test your disaster recovery procedures.
jurassic 4y ago
> Tape Gateway stores virtual tapes in Amazon S3, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive, protected by 99.999999999% of durability. That’s a lot of 9s.
- aftbit 4y ago
  Does than mean that you should expect to lose 1 byte for every 100 GB that you store?
  - primax 4y ago
    The statement is made at an object level typically for AWS
- kcb 4y ago
  I hate it. Once the number gets so small beyond comprehension it’s hard to believe the validity. It reminds me of the Feynman’s statements during the Space Shuttle Challenger disaster. > Feynman was disturbed by two aspects of this practice. First, NASA management assigned a probability of failure to each individual bolt, sometimes claiming a probability of 1 in 10^8, i.e. one in one hundred million. Feynman pointed out that it is impossible to calculate such a remote possibility with any scientific rigor. https://en.wikipedia.org/wiki/Rogers_Commission_Report https://en.wikipedia.org/wiki/Rogers_Commission_Report Unless S3 can survive nuclear war or an asteroid hitting the earth, I don’t buy it.
  - lopkeny12ko 4y ago
    There is rigor behind the claim. Check out this talk from a few years ago: https://www.youtube.com/watch?v=DzRyrvUF-C0 https://www.youtube.com/watch?v=DzRyrvUF-C0
    - dgacmu 4y ago
      There's rigor within the assumptions. The problem is when the assumptions are wrong - most notably around failure independence. Data center fire? Accounted for. Simultaneous terrorist attack / asteroid (as per GP), / nuclear war on all data centers? Maybe not. Where does insider threat fit in on that spectrum? We don't know. I'm personally fine with the claim as I understand the methodology and assumptions but it has to be interpreted carefully.
chrisshroba 4y ago
Tangent: What's the cheapest way to backup a few TB of personal data these days, pricing based on the premise that I probably will never need to retrieve it (due to local backups as well), but I don't want to pay thousands if I do have to (hundreds would be okay). Glacier Deep Archive?
- unethical_ban 4y ago
  Yes. You can use rclone to backup to an S3 bucket, and then have the S3 bucket set to "instantly" move files to deep archive. To retrieve them, you will need to run an extra command to "unarchive" them for some period of time. I find this easier than trying to interact directly with any Deep Archive API. It is very cheap, and AWS's durability guarantees are impressive. Main S3 page, linked to a note about Deep Archive https://rclone.org/s3/#glacier-and-glacier-deep-archive https://rclone.org/s3/#glacier-and-glacier-deep-archive The only thing I am not sure on is how to quickly restore a large number of files in a bucket from GDA to S3. It obviously can be scripted, but I don't have that handy. I only keep a small number of large, previously encrypted files in it, so I manually restore from the GUI. (By the way, rclone can transparently encrypt files and filenames client-side!)
- ghaff 4y ago
  Probably. Looks like about $1/TB/month. Although Backblaze personal backup isn't that much more for a few TB--about $6/month "unlimited" paid annually.
- floren 4y ago
  Backblaze B2 would cost you about $10/mo to store 2TB. I've been using them for years for a few tens of gigabytes worth of documents and photos. https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html edit: it integrates with TrueNAS (formerly FreeNAS) so it's been pretty much set-and-forget, but I can check in and see my stuff through their management interface. I do encrypt my docs before uploading, which TrueNAS also supports.
  - thomaslord 4y ago
mjlee 4y ago
I think it's worth pointing out that this service was first released in 2012.
- s_dev 4y ago
  So why is it relevant to the curious mind on HN in 2023? Was there a significant change made to the system?
zetazzed 4y ago
I think I'll point to this as one of the examples of how AWS has been scrappy and successful. (Yes, I'll get like 10 flames on this comment.) They made this nice simple API for S3 that has become a de facto standard. They added lots of features and security controls, and I can easily imagine an architect saying "everyone should just adopt S3, it's so easy!" But huge, paying customers were used to tape libraries so they said, damnit, fine we'll pretend to be a tape library even though that is super weird on some level... Meeting users where they are can be a superpower.
- chickenpotpie 4y ago
  > damnit, fine we'll pretend to be a tape library IIRC, they may not be pretending. It's not public knowledge what storage medium AWS glacier is built on, but many speculate it is actually tape.
  - capableweb 4y ago
    Well, they do call it "virtual tapes", would be weird (maybe even illegal? False/misleading advertisement?) if they ended up storing it on physical tapes anyways. Edit: found in another thread here, the inventor linked the patent (https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/10013166 https://image-ppubs.uspto.gov/dirsearch-public/print/downloa...) which confirms it's virtual tapes, not actual tapes.
    - dragonwriter 4y ago
      > Well, they do call it "virtual tapes", would be weird (maybe even illegal? False/misleading advertisement?) if they ended up storing it on physical tapes anyways. No, just like it is not weird and definitely not illegal for the things they call “virtual machines” to end up being run on actual machines.
      - capableweb 4y ago
        If you purchase a service describing it as "isolated virtual machines" and it ends up just being processes on a computer without any virtualization, wouldn't you call that at least misleading?
OliverJones 4y ago
HEY! Before you adopt this check the bandwidth egress costs! Seriously. Your disaster recovery plan will need to budget for extra egress costs if you ever need to restore something big from one of those virtual tapes. AWS charges for egress -- downbound -- bandwidth but they don't charge for ingress -- upbound -- bandwidth. (And I think it's funny their icons and images look like old school DECTapes.)
- jeffrallen 4y ago
  The roach motel of data: your data goes in, but it doesn't go out (for free).
- klabb3 4y ago
  > Before you adopt this check the bandwidth egress costs! Closed, works as intended: This is the industry-standard price model for all ransomware. /light sarcasm
jasoneckert 4y ago
It's been a long time since I've heard the word "tape" in my circles (I can't get John Cleese's Institute for Backup Trauma out of my head: https://www.youtube.com/watch?v=q_9wIupr9Hc https://www.youtube.com/watch?v=q_9wIupr9Hc). That being said, kudos to Amazon for having a solution for those who aren't in my circles and still use them.
cynicalsecurity 4y ago
[flagged]
deleted 4y ago
[deleted]
dzdt 4y ago
As always the problem seems to be the 'Hotel California' issue: you can check out but you can never leave. Once you have massive data in AWS there is no efficient and affordable solution to move that data back out; you are locked in forever subject to whatever future terms Amazon may chose to impose.
- thedougd 4y ago
  It is for backups of which you have another copy. To switch providers you would start shipping new backups to the new provider. Once your confident the provider has all the backups to meet your retention policy, you abandon AWS.
- WrtCdEvrydy 4y ago
  > Once you have massive data in AWS there is no efficient and affordable solution to move that data back out; you are locked in forever subject to whatever future terms Amazon may chose to impose. You can have your shit packed into a snowball and shipped to you.
  - pmw 4y ago
    But you pay the same egress rate.
    - WrtCdEvrydy 4y ago
      There's no egress for snowball, you just pay snowball pricing.
  - htrp 4y ago
    Can you? https://aws.amazon.com/snowmobile/faqs/ https://aws.amazon.com/snowmobile/faqs/ > Snowmobile does not support data export.
    - vbezhenar 4y ago
      It's confusing. https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/ Quickly retrieve data from the cloud whenever you need it. What does it mean?
johnklos 4y ago
"Tape Gateway supports all leading backup applications" Only huge companies like Amazon can be this dumb. They don't even mention tar or pax, the two most common tape backup applications. Also, how will this magic Amazon "Tape Gateway" back up petabytes over slow links? There are many data heavy businesses that don't necessarily have tons of Internet bandwidth. Fully saturating a 100 Mbps outgoing connection will only get you 1 TB a day, so what happens when you have a tape's worth of new data daily and an already heavily used Internet connection?
- jaywalk 4y ago
  > Also, how will this magic Amazon "Tape Gateway" back up petabytes over slow links? Easy: it won't. If you have more data than bandwidth, you won't use this service. Not sure why you seem to think otherwise.
  - brudgers 4y ago
    Amazon has a service to send a semi-trailer to the loading dock of your data warehouse, so you can forklift in pallets of physical media. Then hauls it all away for ingestion into its cloud. Or, if it will fit in your station wagon, you can haul your data down the highway to one of Amazon's clouds. Or put it in a box and ship it Fedex.
    - luhn 4y ago
      That's not quite how it works. Amazon will send you various storage appliances (including a semi hauling a 45ft shipping container full of hard drives), which you connect to your network, upload your data, and send it back to them. They don't accept pallets of arbitrary media devices. https://aws.amazon.com/snow/ https://aws.amazon.com/snow/
      - brudgers 4y ago
        Thanks. I started from Tanenbaum’s hurtling station wagon and used my poetic license without worrying too much that the parent commentor would buy a forklift before talking to AWS sales.

140 points by brudgers 4y ago

AWS doing tapes like this feels weird to me. The physicality of the thing is kind of the point in my mind. Virtualizing the last line of defense seems wrong, but perhaps I could be convinced otherwise. Why doesn't Iron Mountain or some other competitor offer a service like this?

sithadmin 4y ago
Iron Mountain does have a service that will pick up physical tape and offload to other storage, but nothing like AWS's virtual tape capabilities. FWIW, Amazon isn't 'doing tapes' with this product. It's just a compatibility shim laid on top of object storage so that legacy tape backup/archival systems can eliminate physical tape libraries.

SCdF 4y ago

Dumb question, and I'm guessing this is a "if you don't know it's not for you" situation, but what is the point of a virtual tape? Isn't the point of a tape that it's not virtual? Or is this more replicating tape software apis (WINE/proton style) so you can get rid of physical tapes (because you no longer care about their physicality) without having to change your backup strategy?

mbutler4 4y ago
From the perspective of the AWS solution, this is a way of giving on-premises hardware an option versus Iron Mountain or similar for archiving to offsite for DR/BC purposes. Since AWS' storage options are typically pretty insane SLAs, this is acceptable (and certainly not much worse than having old LTO tapes in a storage locker). VTLs have been around for a long, long time. Initially, they were a way of speeding up tape by writing to a much faster media, and allowing for deduplication on the media to reduce storage costs. This was, of course, in the days of on-prem arrays and storage. Over time, LTO (the dominant tape format, I think we're on LTO-9?) became denser and faster, and the speeds at which you could write to a single drive would outpace the line speeds. Given that, methods were introduced to interleave multiple writers to a single drive/cart, and to leverage the drives speed. One thing you don't want on a tape drive is "shoeshining", this is the effect when the drive has no data to write, but the momentum of the reels in the drive mechanism carry the media forward, past the last written segment, past the write head. The drive must then backup and re index to find where it left off while queuing whatever data is incoming. Well, as VTLs were built initially to handle speed of writes and dedup, once the tape drives outpaced those issues, VTL became problematic. It was basically taking a randomly writeable media and turning it into a serial storage mechanism. This really wasn't much use, and VTL started to fade. Many solutions were built, one from NetBackup was OpenStorage; this was an API that a storage vendor could write a plugin for and allow NetBackup to write to their storage as intended: multiple writer, multiple access. Allowing for treatment of a deduplicable storage media as a filesystem versus an emulated tape drive. This API approach allowed for plugins to be constructed to talk to any storage, including cloud. AWS spoke with NetBackup at one point, intending to build a storage mechanism for a very solid solution: allow on-prem customers to export their data to AWS storage, AND to recover that data BACK from AWS storage (think of the data transfer fees you would get out of that!!). When they looked at the options, they decided to build the VTL gateway. I'm not sure of how or where the decision was made, but it seemed a bit odd at the time. As VTLs were fading and the limitations of emulating tape would lead to a caching requirement that might be bad, but with line speeds for interconnect to AWS making the caching requirements even worse. I'm not quire sure how that was done, but I will say, a VTL will always be a stable, solid thing, that tends to simply run and do what was intended; much like a traditional tape drive. As for some of the other discussion here, about reading old tapes and storage longevity; I've seen tapes kept for decades that were readable, Iron Mountain is still in business for a reason. Note that THEY don't have a solid cloud offering, but really, if you had everyones old data sitting around on tapes, and could charge not only for data delivery but media conversion at DR time, wouldn't you? And the issue with old tapes isn't so much the storage in that situation, it's the hardware. LTO only reads 2 generations backward (IIRC), and can only write one gen forwards. So, for your LTO-6 drive, it can read LTO-4 to 6, and can write LTO-6 to 7, again if memory serves. So, finding the Apollo code tapes on a format tape where the only possible drives available to read it live in a museum and may or may not actually work, make the fact that they recovered that source code amazing. Not to mention the software needed to be built to interface to hardware that the last person to build a "driver" for likely is sitting in a nice retirement joint in Sarasota.
deleted 4y ago
[deleted]
thedougd 4y ago
It's the latter. In a large enterprise, the backup configuration can be wildly complicated, a matrix of systems, schedules, slas, etc. Just reconfigure your backup software to use this new virtual tape device and you're on your way.
- logifail 4y ago
  > Just reconfigure your backup software to use this new virtual tape device and you're on your way Isn't the whole point of tape that it is a physical thing, and may be taken offsite in a truck / stored in a vault, and all that jazz. "Just reconfiguring" your backup software sounds like a business might just bypass all that without necessarily realising the consequences. Then "we got hacked and our local backups are gone" -> "restore from offsite tape" -> "oh, actually there are no tapes, it's all the cloud now" -> ... ??
  - orf 4y ago
    Then "we got hacked and our local backups are gone" -> "restore from offsite tape" -> "oh, actually all the tapes where stored at the wrong temperature or humidity and now they are fucked" -> ... ??
  - kube-system 4y ago
    AWS's hard disks are a physical thing that is offsite
    - logifail 4y ago
      > AWS's hard disks are a physical thing that is offsite Taking tapes to a secure offsite location means they're air-gapped from the source data, so even in a situation where an entire network is compromised, remotely-stored tapes can't be wiped/encrypted. I'm sure AWS has thought about this issue while designing Tape Gateway, but it's not clear to me how it could know whether a request to retrieve and overwrite a virtual tape was legitimate or not.
      - 4y ago
sithadmin 4y ago
The gateway appliance is fairly compatible with legacy backup solutions, which makes it a great drop-in replacement for physical tape library systems. There are certainly better backup methods available these days (though it's hard to beat the durability and cost of LTO tape for long-term archival), but I've seen AWS's virtual tape used as a good stopgap while other backup/recovery solutions are still a ways out an org's infrastructure roadmap.
- tablespoon 4y ago
  > though it's hard to beat the durability and cost of LTO tape for long-term archival I'm not entirely comfortable with the trend towards ever more esoteric and seemingly "all or nothing" technologies. Tape (and other physical things) may have their downsides, but I think there's something to be said for something that could (theoretically) be forgotten in a closet and read 50 years later. Likewise, AM radio may not be the best quality, but it seems like you can cover more area with a single installation than any other communication technology, which might come in handy after a serious disaster like a nuclear war (e.g. crank up the nighttime power of some undamaged countryside station running on generators to tell survivors where it's still safe).
  - vbezhenar 4y ago
    It's very unlikely that you'll be able to read modern tape from closet in 50 years. Tape requires very precise temperature and humidity for storage. Otherwise all bets are off.
    - linksnapzz 4y ago
      Only one datapoint; but I’ve seen an FPS/Celerity minicomputer boot off of a Qic-20 tape that’d been stored in a non climate controlled Albuquerque storage unit for 19 years…
brudgers 4y ago
Even for a new strategy, using a tape abstraction might simplify the design and/or implementation because backup tools and culture have their roots in tape. There's a sense in which tape is water in which backups swim.
shrike 4y ago
The VTL was invented because, at the time (2012ish), none of the largest enterprise backup solutions had a good S3 interface and none supported Glacier. After talking with the backup software vendors (Spectrum, Tivoli, Symantec, Commvault, etc) it became clear that adding another backup target wasn't something we (AWS) could get them to prioritize, for perfectly reasonable reasons. We could (and did) apply pressure via our shared customers, even then they estimated it would take years. The fastest way to enable large enterprise access to S3 and Glacier for backups was to meet them where they were. We did this by virtualizing a tape library. Background: I'm one of the original inventors - https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/10013166 https://image-ppubs.uspto.gov/dirsearch-public/print/downloa.... I am no longer with AWS.
- deleted 4y ago
  [deleted]
- sshagent 4y ago
  You did such a great job they still try and steer people this route. I've multiple times mentioned "you know we don't need to emulate tape drives any more"
- free-ideas 4y ago
  VTL has been around since the 90’s https://en.m.wikipedia.org/wiki/Virtual_tape_library https://en.m.wikipedia.org/wiki/Virtual_tape_library. This is a fantastic example of how broken the patent system is. This is not an invention it’s a good implementation of a well understood problem. It is also priced ridiculously - it is at least an order of magnitude more expensive than operating a physical library and remote physical storage that is beyond cyber threat by virtue of it being disconnected. A real backup is offline and offsite.
- Twirrim 4y ago
  You did such a good job, that when I worked on Glacier (2013-2016), our interactions with the team running VTL was already pretty minimal. It was one of the least problematical things that interacted with Glacier. Some of the solutions that external vendors produced were nightmarish and left customers up the creek without a paddle in a disturbing number of situations.

glasss 4y ago

My first and only experience with tape backups was in 2014 working with the BBB of Chicago. They were using tape backups, and there I learned that almost no company that small should be using tapes. Unless you have one or two people dedicated to handling that plus whatever other backup solutions you have in place, it won't end well.

robohoe 4y ago
I hear you. I had to manage a Quantum LTO4 tape library with Bacula back in early 2010s at a small company. Yeah that was a fun experience (not) and I'm glad I don't have to deal with them. Good riddance. Just getting the labeling right and making sure that the tapes were writable was a pain.
- guenthert 4y ago
  But that is Bacula fault isn't it? I used it myself @home and couldn't help thinking that there must be a better way. At work Commvault was used iirc, but others were in charge of that. I evaluated Tivoli SM, but that's more than 15 years ago and I barely remember. TSM and Bacula share some concepts, but the latter appears quite hackish. Not that confidence inspiring.
jedberg 4y ago
Interesting. I wonder if tapes got more complicated or if everything else just got easier. The last time I dealt with tapes was in the 1990s, and it was dead simple. We used tape backups at a company with only 150 people. I was the most junior admin so I had to be the tape monkey, but it was easy. Just run backups regularly, change the tape every couple of days and label it, ship a stack to the offsite storage and request the old stack back, and then do a test restore each time a stack came back. The whole thing took maybe two hours a week.
- glasss 4y ago
  Everything else was definitely easier by comparison, but the situation was more about the one internal guy responsible for it was also doing all of the normal IT dept functions, so being the tape monkey was at the bottom of the list - for better or worse. When you're juggling 12 plates, a couple are gonna fall.

jmclnx 4y ago

Having to restore data from Mag Tapes over decades and getting CRC errors maybe 20% of the time, this cannot possibly be worse.

Tepix 4y ago
But vastly more expensive, that's for sure.
- Nexxxeh 4y ago
  Depends on the cost of a potential 20%+ loss of data.
shiftpgdn 4y ago
You needed to run a cleaner through that tape device.
- jmclnx 4y ago
  No kidding :) But the company that had the issue (I left a bit ago) believed in "cheaper the better". Cleaners ? That place would expect you to use soap from the restroom :)

api 4y ago

Anyone considering putting giant amounts of data into AWS or most other big clouds should be sure to do the numbers on retrieval. All the big clouds have "roach motel pricing" -- data is free or very cheap to put in, and expensive to get out. Outbound bandwidth costs from AWS are astronomical, so make sure you are not going to need to stream down all that data or move it any time soon or that you have compared that cost to your in-house solution.

cube00 4y ago
This kind of unbalanced pricing also acts as a disincentive to regularly test your disaster recovery procedures.

jurassic 4y ago

> Tape Gateway stores virtual tapes in Amazon S3, Amazon S3 Glacier Flexible Retrieval, and Amazon S3 Glacier Deep Archive, protected by 99.999999999% of durability. That’s a lot of 9s.

aftbit 4y ago
Does than mean that you should expect to lose 1 byte for every 100 GB that you store?
- primax 4y ago
  The statement is made at an object level typically for AWS
kcb 4y ago
I hate it. Once the number gets so small beyond comprehension it’s hard to believe the validity. It reminds me of the Feynman’s statements during the Space Shuttle Challenger disaster. > Feynman was disturbed by two aspects of this practice. First, NASA management assigned a probability of failure to each individual bolt, sometimes claiming a probability of 1 in 10^8, i.e. one in one hundred million. Feynman pointed out that it is impossible to calculate such a remote possibility with any scientific rigor. https://en.wikipedia.org/wiki/Rogers_Commission_Report https://en.wikipedia.org/wiki/Rogers_Commission_Report Unless S3 can survive nuclear war or an asteroid hitting the earth, I don’t buy it.
- lopkeny12ko 4y ago
  There is rigor behind the claim. Check out this talk from a few years ago: https://www.youtube.com/watch?v=DzRyrvUF-C0 https://www.youtube.com/watch?v=DzRyrvUF-C0
  - dgacmu 4y ago
    There's rigor within the assumptions. The problem is when the assumptions are wrong - most notably around failure independence. Data center fire? Accounted for. Simultaneous terrorist attack / asteroid (as per GP), / nuclear war on all data centers? Maybe not. Where does insider threat fit in on that spectrum? We don't know. I'm personally fine with the claim as I understand the methodology and assumptions but it has to be interpreted carefully.

chrisshroba 4y ago

Tangent: What's the cheapest way to backup a few TB of personal data these days, pricing based on the premise that I probably will never need to retrieve it (due to local backups as well), but I don't want to pay thousands if I do have to (hundreds would be okay). Glacier Deep Archive?

unethical_ban 4y ago
Yes. You can use rclone to backup to an S3 bucket, and then have the S3 bucket set to "instantly" move files to deep archive. To retrieve them, you will need to run an extra command to "unarchive" them for some period of time. I find this easier than trying to interact directly with any Deep Archive API. It is very cheap, and AWS's durability guarantees are impressive. Main S3 page, linked to a note about Deep Archive https://rclone.org/s3/#glacier-and-glacier-deep-archive https://rclone.org/s3/#glacier-and-glacier-deep-archive The only thing I am not sure on is how to quickly restore a large number of files in a bucket from GDA to S3. It obviously can be scripted, but I don't have that handy. I only keep a small number of large, previously encrypted files in it, so I manually restore from the GUI. (By the way, rclone can transparently encrypt files and filenames client-side!)
ghaff 4y ago
Probably. Looks like about $1/TB/month. Although Backblaze personal backup isn't that much more for a few TB--about $6/month "unlimited" paid annually.
floren 4y ago
Backblaze B2 would cost you about $10/mo to store 2TB. I've been using them for years for a few tens of gigabytes worth of documents and photos. https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html edit: it integrates with TrueNAS (formerly FreeNAS) so it's been pretty much set-and-forget, but I can check in and see my stuff through their management interface. I do encrypt my docs before uploading, which TrueNAS also supports.
- thomaslord 4y ago

mjlee 4y ago

I think it's worth pointing out that this service was first released in 2012.

s_dev 4y ago
So why is it relevant to the curious mind on HN in 2023? Was there a significant change made to the system?

zetazzed 4y ago

I think I'll point to this as one of the examples of how AWS has been scrappy and successful. (Yes, I'll get like 10 flames on this comment.) They made this nice simple API for S3 that has become a de facto standard. They added lots of features and security controls, and I can easily imagine an architect saying "everyone should just adopt S3, it's so easy!" But huge, paying customers were used to tape libraries so they said, damnit, fine we'll pretend to be a tape library even though that is super weird on some level... Meeting users where they are can be a superpower.

chickenpotpie 4y ago
> damnit, fine we'll pretend to be a tape library IIRC, they may not be pretending. It's not public knowledge what storage medium AWS glacier is built on, but many speculate it is actually tape.
- capableweb 4y ago
  Well, they do call it "virtual tapes", would be weird (maybe even illegal? False/misleading advertisement?) if they ended up storing it on physical tapes anyways. Edit: found in another thread here, the inventor linked the patent (https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/10013166 https://image-ppubs.uspto.gov/dirsearch-public/print/downloa...) which confirms it's virtual tapes, not actual tapes.
  - dragonwriter 4y ago
    > Well, they do call it "virtual tapes", would be weird (maybe even illegal? False/misleading advertisement?) if they ended up storing it on physical tapes anyways. No, just like it is not weird and definitely not illegal for the things they call “virtual machines” to end up being run on actual machines.
    - capableweb 4y ago
      If you purchase a service describing it as "isolated virtual machines" and it ends up just being processes on a computer without any virtualization, wouldn't you call that at least misleading?

OliverJones 4y ago

HEY! Before you adopt this check the bandwidth egress costs! Seriously. Your disaster recovery plan will need to budget for extra egress costs if you ever need to restore something big from one of those virtual tapes. AWS charges for egress -- downbound -- bandwidth but they don't charge for ingress -- upbound -- bandwidth. (And I think it's funny their icons and images look like old school DECTapes.)

jeffrallen 4y ago
The roach motel of data: your data goes in, but it doesn't go out (for free).
klabb3 4y ago
> Before you adopt this check the bandwidth egress costs! Closed, works as intended: This is the industry-standard price model for all ransomware. /light sarcasm

jasoneckert 4y ago

It's been a long time since I've heard the word "tape" in my circles (I can't get John Cleese's Institute for Backup Trauma out of my head: https://www.youtube.com/watch?v=q_9wIupr9Hc https://www.youtube.com/watch?v=q_9wIupr9Hc). That being said, kudos to Amazon for having a solution for those who aren't in my circles and still use them.

cynicalsecurity 4y ago

[flagged]

deleted 4y ago

[deleted]

dzdt 4y ago

As always the problem seems to be the 'Hotel California' issue: you can check out but you can never leave. Once you have massive data in AWS there is no efficient and affordable solution to move that data back out; you are locked in forever subject to whatever future terms Amazon may chose to impose.

thedougd 4y ago
It is for backups of which you have another copy. To switch providers you would start shipping new backups to the new provider. Once your confident the provider has all the backups to meet your retention policy, you abandon AWS.
WrtCdEvrydy 4y ago
> Once you have massive data in AWS there is no efficient and affordable solution to move that data back out; you are locked in forever subject to whatever future terms Amazon may chose to impose. You can have your shit packed into a snowball and shipped to you.
- pmw 4y ago
  But you pay the same egress rate.
  - WrtCdEvrydy 4y ago
    There's no egress for snowball, you just pay snowball pricing.
- htrp 4y ago
  Can you? https://aws.amazon.com/snowmobile/faqs/ https://aws.amazon.com/snowmobile/faqs/ > Snowmobile does not support data export.
  - vbezhenar 4y ago
    It's confusing. https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/ Quickly retrieve data from the cloud whenever you need it. What does it mean?

johnklos 4y ago

"Tape Gateway supports all leading backup applications" Only huge companies like Amazon can be this dumb. They don't even mention tar or pax, the two most common tape backup applications. Also, how will this magic Amazon "Tape Gateway" back up petabytes over slow links? There are many data heavy businesses that don't necessarily have tons of Internet bandwidth. Fully saturating a 100 Mbps outgoing connection will only get you 1 TB a day, so what happens when you have a tape's worth of new data daily and an already heavily used Internet connection?

jaywalk 4y ago
> Also, how will this magic Amazon "Tape Gateway" back up petabytes over slow links? Easy: it won't. If you have more data than bandwidth, you won't use this service. Not sure why you seem to think otherwise.
- brudgers 4y ago
  Amazon has a service to send a semi-trailer to the loading dock of your data warehouse, so you can forklift in pallets of physical media. Then hauls it all away for ingestion into its cloud. Or, if it will fit in your station wagon, you can haul your data down the highway to one of Amazon's clouds. Or put it in a box and ship it Fedex.
  - luhn 4y ago
    That's not quite how it works. Amazon will send you various storage appliances (including a semi hauling a 45ft shipping container full of hard drives), which you connect to your network, upload your data, and send it back to them. They don't accept pallets of arbitrary media devices. https://aws.amazon.com/snow/ https://aws.amazon.com/snow/
    - brudgers 4y ago
      Thanks. I started from Tanenbaum’s hurtling station wagon and used my poetic license without worrying too much that the parent commentor would buy a forklift before talking to AWS sales.