32 ms·

Iron Mountain: It's Time to Talk About Hard Drives

197 points by severine 2y ago

alchemist1e9 2y ago
Does it mean LTO tape for the win then? We’re about to start a project to build an LTO-9 based in-house backup system. Any suggestions for DIY Linux based operation doing it “correctly” would be appreciated. Preliminary planning is to have one drive system on in our primary data center and another offsite at an office center where tapes are verified before storage in locked fireproof storage cabinet. Tips on good small business suppliers and gear models would be great help.
- antisthenes 2y ago
  What's your budget?
  - alchemist1e9 2y ago
    $50K if needed but it doesn’t look to need that. 2PB initial data. predicted 1PB/year with around 10-30%/year rate of growth of rate of growth (acceleration?)
- akira2501 2y ago
  > fireproof storage cabinet. Nothing is fire proof. Is the cabinet "fire suppression system liquid" proof? > Tips on good small business suppliers and gear models would be great help. Hire an auditor would be my advice. Every business is different. I am, just now, having flashbacks of when I was in a SOX environment and had to regularly contract with them... and while the experience can be somewhat unpleasant I've often found good auditors to be extremely knowledgeable about solutions and their practical implementation considerations.
  - hinkley 2y ago
    Pretty sure the pyramids at Giza count as fireproof boxes. But that’s out of most people’s price range.
    - burnished 2y ago
      Not if the fire is big enough.
      - hinkley 2y ago
        When a catastrophic event is catastrophic enough, your other problems cease to be your problem.
    - akira2501 2y ago
      They're not fully sealed. There are two shafts which connect the King's chamber to the exterior of the pyramid. The lower ritual congregation area is not fully sealed off from the upper chambers either. Which means bats are a constant problem in pyramids. You could easily get a fire going inside one.
      - dredmorbius 2y ago
        Fortunately, it actually is somewhat challenging to get a fire going inside a bat.
        akira2501 2y ago
        Famously, their feces are not at all challenging to set on fire.
        dredmorbius 2y ago
        Still on the outside of the bats ;-)
- hinkley 2y ago
  The advice I got long ago from an IT guy was: if you wait long enough, tape will be on top again. That was a long time ago but I’ve peeked in at backup systems in the intervening years and it does seem to hold true over time. But it really depends how much data you have. My ex dropped a single HDD in a safety deposit box at CoB, N times per week and fetched back the oldest disk. I don’t think she ever said how many were in there but I doubt it was more than three. I think the CTO took one home with him once per week. The silly thing about most of this set up is that the office, the bank, and the data center were all within half a kilometer of each other. If something bad happened to that part of town they only had the infrequent offsite backup.
  - katbyte 2y ago
    20tb and under is easy. 500tb is hard
  - exe34 2y ago
    the sort of bad that would render all three unreadable would probably melt the rest of the infrastructure the company relies on for business anyway.
  - marcosdumay 2y ago
    Every time I've looked, tapes were on top "again" for large scale archival. And I've been looking for ~20 years by now. I don't get where people get the impression that X was at the top right before tapes got that last innovation (where X here is most often HDDs, but not always). But that's always the impression, and tapes are always on top. People also have been working with 3D phase change drives since the 90s. Those always promise to replace tapes. But nobody ever got them robust enough to leave the labs.
    - hakfoo 2y ago
      Tape might win for large-scale systems, but it's basically dead for home-office scale. You used to be able to get modestly priced tape units for home use from the old "connects to the floppy controller" units with capacities in the tens of megabytes, up to some late-gen SCSI/IDE/parallel/early USB models that would be a couple of gigabytes, but still at home-friendly prices. What's today's answer? An enterprise-grade device that might put 10TB+ on a tape, but comes with a four-figure price tag and isn't really sold at Best Buy. If I want to back up the house today (maybe 4 active PCs, 5-6Tb of total space), affordable choices are pretty much disc-based. I could choose a cheap NAS (ended up doing that with an old fanless Atom machine and a used 12TB datacentre drive) or get a USB-attached external drive. Even if I used BD-XL media, even my modest needs would be dozens of discs, plus getting a writer in a shrinking market. There are plenty of datahoarders with much bigger needs, but even for them, tape is completely outside the addressable market.
- WaitWaitWha 2y ago
  Q: Why not archival M-Disc?
  - actionfromafar 2y ago
    A: Are there any "real" M-Discs for purchase anymore? (I.e. not just rebranded regular dye-discs.)
    - netrap 2y ago
      Possibly in DVDR.
  - alchemist1e9 2y ago
    Amount of data makes it less realistic. We have around 2PB data currently and expect to grow around 1PB next year with maybe 10-30% annual growth rate.
- bluedino 2y ago
  Tapes are fun. You can fit a petabyte of data in a bankers box! The problem quickly becomes: - Do we have a drive that can read this tape? - Do we have server we can connect it to? - Do we have storage we can extract it to? (go ask your internal IT team for 10TB of drivespace...) - What program did we create this tape with? Backup Exec, Veritas, ArcServe, SureStore - You have the encryption keys, right? - How much of this data already exists on the previous months backup? - Who's going to pay for the storage to move it to Glacier/etc? -How long is it going to take to upload?
  - aftbit 2y ago
    If you are having trouble getting 10TB of disk space from IT, you have bad IT. Not saying that's uncommon or anything, but 10TB fits on one external hard drive for $300 from Best Buy, or less than $1000/mo using EBS on AWS if you need some better guarantees and are all-in on the cloud.
  - DaiPlusPlus 2y ago
    > Tapes are fun. You can fit a petabyte of data in a bankers box! Yes, though those ultra-thin M.2 NVMe drives could probably top that now. > Do we have a drive that can read this tape? Don't let this be a problem in the first place: buy 4 tape drives and keep 2 of them in your cold/offsite/airgapped storage site (2 in case 1 fails, so you can use the remaining drive to transfer everything to a newer format). > Do we have server we can connect it to? Significant hardware should not (and is not) necessary: the LTO-7/8/9 drives I see for sale right now seem to be using either USB 3.0, Thunderbolt, or SAS connections; USB and Thunderbolt can be handled by any computer you can find at a PC recycler today; while any old desktop can handle SAS with a $80 HBA card. > Do we have storage we can extract it to? (go ask your internal IT team for 10TB of drivespace...) 10TB isn't a good example (case-in-point: I have a 3-year-old stack of unused 12TB WD drives less than 3 feet away from me). That said, if you're enterprise-y enough for a $4000 LTO-9 drive, then you probably also have a SAN that's chock full of drives, so being able to provision a 10TB+ LUN should be implicit. > What program did we create this tape with? Backup Exec, Veritas, ArcServe, SureStore Ideally, none of those; instead, good ol' Perl and `dd`. > You have the encryption keys, right? I don't encrypt my backups to avoid this problem. My old data archives have little exploitable value for any potential attacker; and I imagine I'd store backup tapes this important in a fireproof safe in my parents' house or something. I'd only encrypt the entire tape if the tape were to leave my custody. I appreciate that this is not for everyone, and it's probably illegal for some people/orgs to not encrypt backups too anyway (HIPAA, etc). > How much of this data already exists on the previous months backup? Incremental/Differential backups are still a thing. > Who's going to pay for the storage to move it to Glacier/etc? No-one should. Cold data backups should/must always be in the custody of a designated responsible officer of the company. BUUUT, I guess there's nothing wrong with storing an encrypted copy in Cloud storage (as in S3/Glacier/AzureBlobs - not OneDrive...). I actually do this right now thanks to the smooth and painless integration in my Synology NAS. It costs me about $15/mo to store all these TBs in S3. > How long is it going to take to upload? Consider that it's 2024 - a company with LTO-9 and SAN is probably going to have a metro-ethernet IP connection at 10Gbps or even faster. At home I have a 10Gbps symmetric connection from Ziply (it's $300/mo and they give you an SFP module, which I put into my Ubiquiti UDM): so the limiting factor here is not my upload speed, but my drive read speed (LTO-9 drives seem to read at about 2-3Gbps raw/uncompressed?)
- gosub100 2y ago
  Make sure the bandwidth exists to keep up with the write speed of the LTO drive. For instance, the write speed for LTO-6 (which I own as a hobbyist) is around 300MB/s, but line speed of gigabit Ethernet is about 100MB/s. Translate those numbers to LTO-9 and make sure that the NAS, network, or local storage can keep up. It's not a deal-breaker to underflow the drive, but it causes the tape to stop, rewind, and re-buffer (called shoe-shining) which takes more time and causes unnecessary wear on the drive and cartridges.
- adrian_b 2y ago
  LTO-9 tapes can be easily found on Amazon in many countries, made by IBM, HP, Quantum or Fuji. The vendor does not matter, whichever happens to be cheaper at the moment is fine. For the tape drives, the internal drives can be cheaper by around 10%, but I prefer the tabletop drives, because they are less prone to accumulate dust, especially if you switch them on only when doing a backup or a retrieval. The tape drives have usually very noisy fans, because they are expected to be used in isolated server rooms. I believe that the cheapest tape drives from a reputable manufacturer are those from Quantum. I have been using a Quantum LTO-7 tape drive for about 7 or 8 years and I have been content with it. Looking now at the prices, it should be possible to find a tabletop LTO-9 drive for no more than $5000. Unfortunately, the prices for tape drives have been increasing. When I have bought an LTO-7 tabletop drive many years ago it was only slightly more than $3000. The tapes are much cheaper and much more reliable than hard disks, but because of the very expensive tape drive you need to store a few hundred TB to begin to save money over hard disks. You should normally make at least two copies of any tape that is intended for long-term archiving (to be stored in different places), which will shorten the time until reaching the threshold of breaking even with HDDs. Even if there are applications that simulate the existence of a file system on a tape, which can be used even by a naive user to just copy files on a tape, like copying files between disks, they are quite slow and inefficient in comparison to just using raw tape commands with the traditional UNIX utility "mt". It is possible to write some very simple scripts that use "mt" and which allow the appending of a number of files to a tape or the reading of a number of consecutive files from a tape, starting from the nth file since the beginning of a tape. So if you are using only raw "mt" commands, you can identify the archived files only by their ordinal number since the beginning of the tape. This is enough for me, because I prepare the files for backup by copying them in some directory, making an index of that directory, then compressing it and encrypting it. I send to the tape only encrypted and compressed archive files, so I disable the internal compression of the tape drive, which would be useless. I store the information about the content of the archives stored on tapes (which includes all relevant file metadata for each file contained in the compressed archives, including file name, path name, file length, modification time, a hash of the file content) in a database. Whenever I need archived data, I search the database, to determine that it can be found, for instance in tape 63, file 102. Then I can insert the corresponding cartridge in the drive and I give the command to retrieve file 102. I consider much better the utility "mt" of FreeBSD than that of Linux. The Linux magnetic drive utilities have seen little maintenance for many years. Because of that, when I make backups or retrievals they go to a server that runs FreeBSD, on which the SAS HBA card is installed. When a tabletop drive is used, the SAS HBA card must have external SAS connectors, to allow the use of an appropriate cable. I actually reboot that server into FreeBSD for doing backups or retrievals, which is easy because I boot it from Ethernet with PXE, so I can select remotely what OS to be booted. One could also use a FreeBSD VM on a Linux server, with pass-through of the SAS HBA card, but I have not tried to do this. My servers are connected with 10 Gb/s Ethernet links, which does not differ much from the SAS speed, so they do not slow much the backup/retrieval speed. I transfer the archive files with rsync over ssh. On slow computers and internal networks one can use rsync without ssh. I give the commands for the tape drive from the computer that is backed up, as one line commands executed remotely by ssh. The archive that is transferred is stored in a RAMdisk before being written on the tape, to ensure that the tape is written at the maximum speed. I write to the tape archive files that have usually a size of up to about 60 GB (I split any files bigger than that; e.g. there are BluRay movies of up to 100 GB). The server has a memory of 128 GB, so I can configure on it a RAMDdisk of up to 80 GB without problems. This method can be used even with a slow 1 Gb/s or 2.5 Gb/s network, but then uploading a file through Ethernet would take much more time than writing or reading the tape. There is one weird feature of the raw "mt" commands, which is poorly documented, so it took me some time to discover it, during which I have wasted some tape space. When you append files to a partially written tape, you first give a command to go to the end of the written part of the tape. However, you must not start writing, because the head is not positioned correctly. You must go 2 file marks backwards, then 1 file mark forwards. Only then is the head positioned correctly and you can write the next archived file. Otherwise there would be 1 empty file intercalated at each point where you have finished appending a number of files and then you have rewound the tape and then you have appended again other files at the end.
- wazoox 2y ago
  Archival is more of a process than only a question of media. First you must create a proper database of your archived data. Maybe you want to do 3 copies, not two. Maybe you want to use two different archival formats such as tar and LTFS, just in case. Maybe you want to source your media from both available producers (Sony and Fuji) because in the long run, maybe one or the other may grow some funky error mode or corruption problem. Etc. Also check my tape management primer: https://blogs.intellique.com/tech/2022/01/27#TapeCLI https://blogs.intellique.com/tech/2022/01/27#TapeCLI
antisthenes 2y ago
What's the scenario where you cannot take the old 1990's hard drive and back up its data in multiple cloud service providers cold storage (Azure/AWS/GCP) and have to keep the obsolete physical media on hand? I'm struggling to understand why these miles of shelves filled with essentially hardware junk haven't been digitized at the time when this media worked and didn't experience read issues. The article doesn't really provide an explanation for this other than incompetence and the business biting off more than it can reasonably chew. I'd be furious if I paid for a service that promised to archive my data, and 10-15 years later told me 25% of it was unreadable. I mean it's not like it was a surprise either. These workflows became digital 2-3 decades ago. There was plenty of time to prepare and convert this. That's kind of what I'm paying you for. As always, seems like the simple folk of /r/datahoarder and other archivist communities are more competent than a legacy industry behemoth.
- akira2501 2y ago
  > the business biting off more than it can reasonably chew It's hoarding behavior. They paid "a lot" of money for it, have no idea how to further exploit it, but can't shake the feeling that it might be massively valuable one day. The only difference is they pay someone to hold their hoard for them.
  - eitally 2y ago
    Alternatively ... they are forced to maintain it for compliance reasons, especially when it comes to healthcare, finance and other regulated industries (defense, in particular). This even applies to manufacturing companies assembly medical devices and defense products: all the data about the supply chain, the engineering designs & changes, the manufacturing and quality testing, and shipment needs to be kept for XX years and is subject to both audit by regulatory agencies and to legal discovery.
    - akira2501 2y ago
      Those are all things which can be printed out and stored in alternative forms and possibly even recreated from other data. It's also the case that much of that data will never be permanently at rest and so several archive copies of the data exist. Recordings of performances are an entirely different category of artefact.
akira2501 2y ago
> It may sound like a sales pitch, but it’s not; it’s a call for action Your entire article sounds like a sales pitch. Your solution is, well, it's bad, but trust us, we can maybe recover it anyways. Otherwise your article fails to convey anything meaningful.
- derefr 2y ago
  No, the call-to-action being referenced in the article is "stop archiving to hard drives" (and use tape instead, every other industry does.)
  - ganoushoreilly 2y ago
    I think it is or at least tries to be more than that. Not only stop archiving to HD's but understand the dependency requirements which is a whole secondary problem.
  - natch 2y ago
    The last time I used a tape it broke right off the reel the first time I used it. Good name brand tape drive, good name brand tape. I felt pretty burned by that experience after the money spent and the result.
  - shiroiushi 2y ago
    Tape drives aren't economical. If you're a big company, sure, they make sense, but for individuals and small companies, they really don't'. Hard drives are absolutely the only realistic and affordable way to back up data. They're not bulletproof though, so you need multiple hard drives, and you need a backup strategy that rotates them, so even if one fails, you haven't lost too much.
    - aidenn0 2y ago
      They are already paying another company to warehouse the drives; surely that company could offer to rent LTO drives going back several generations?
    - 2y ago
MisterTea 2y ago
Makes me wish we didn't stop advancing optical media technology to where we have cheap and reliable archival quality 1TB discs for a few bucks each. I guess LTO is the best option for personally controlled archival.
- 0cf8612b2e1e 2y ago
  We haven’t, but sadly the technology is locked to big tech. Microsoft has demoed some cool technology where they store data in glass, Project Silica. Sadly, it seems unlikely this will ever be available to consumers. One neat aspect of the design is that writing data is significantly higher power than reading. So you can keep your writing devices physically separated from the readers and have no fear that malicious code could ever overwrite existing data plates. Some blurbs Project Silica is developing the world’s first storage technology designed and built from the media up to address humanity’s need for a long-term, sustainable storage technology. We store data in quartz glass: a low-cost, durable WORM media that is EMF-proof, and offers lifetimes of tens to hundreds of thousands of years. This has huge consequences for sustainability, as it means we can leave data in situ, and eliminate the costly cycle of periodically copying data to a new media generation. We’re re-thinking how large-scale storage systems are built in order to fully exploit the properties of the glass media and create a sustainable and secure storage system to support archival storage for decades to come! We are co-designing the hardware and software stacks from scratch, from the media all the way up to the cloud user API. This includes a novel, low-power design for the media library that challenges what the robotics and mechanics of archival storage systems look like. https://www.microsoft.com/en-us/research/project/project-silica/ https://www.microsoft.com/en-us/research/project/project-sil...
  - actionfromafar 2y ago
    Now this is proper Sci-Fi tech! Data crystals, like 1960s Star Trek!
  - yellow_postit 2y ago
    I'm partial to their DNA work: https://www.microsoft.com/en-us/research/project/dna-storage/ https://www.microsoft.com/en-us/research/project/dna-storage...
simonw 2y ago
My understanding is that the only reliable way of long-term digital archival storage is to refresh the media you are storing things on every few years, copying the previous archives to the fresh storage. Since storage constantly gets cheaper, 100GB first stored in 2001 can be stored on updated media for a fraction of that original cost in 2024.
- abracadaniel 2y ago
  Pretty much. You see hobbyists getting data off of 30+ year old hard drives for the novelty of it, but I can’t imagine relying on that as a preservation copy. Optical media rots, magnetic media rots and loses magnetic charge, bearings seize, flash storage loses charge, etc. Entropy wins, sometimes much faster than you’d expect.
  - animal531 2y ago
    Sometimes they fail for other reasons as well, such as improper storage. Back in the 90's to 00's a friend had a collection of cd's that he'd written, but he stored them in a big sleeved folder container. The container itself caused them to warp slightly, which made them unusable. I took a few for testing and managed to unbend them after some time, which turned them back into a working state. [Note: That's the most apostrophes I've ever used in a sentence, it feels dirty]
    - aimor 2y ago
      A little dirty according to CMOS. Ought to be '90s not 90's, cds not cd's. https://www.chicagomanualofstyle.org/book/ed18/part2/ch09/psec035.html https://www.chicagomanualofstyle.org/book/ed18/part2/ch09/ps... https://www.chicagomanualofstyle.org/qanda/data/faq/topics/Punctuation/faq0051.html https://www.chicagomanualofstyle.org/qanda/data/faq/topics/P...
      - animal531 2y ago
        Thanks for the tip!
mercurialuser 2y ago
There are several articles about film preservation in digital format. Every X years all the data is "upgraded", from LTOn to LTOn+1 or +2. So it may sound like a sales pitch but I consider it more a warning notice
- esafak 2y ago
  That means the hardware and the file format.
Clamchop 2y ago
They have lots of problems: 1. Incomplete copies with missing dependencies. 2. Old software and their file formats with a poor virtualization story. 3. Poor cataloging. 4. Obsolete physical interfaces, file systems, etc. 5. Long-term cold storage on media neither proven nor marketed for the task. Managing archives is just a cost center until it isn't, and it's hard to predict what will have value. The worst part of this is that TFA discusses mostly music industry materials. Outside parties and the public would have a huge interest in preserving all this, but of course it's impossible. All private, proprietary, copyrighted, and likely doomed to be lost one way or another. Oh well.
- cookiengineer 2y ago
  Related documentary that comes to mind: Digital Amnesia (2014) [1] It broke my heart seeing those librarians in disbelief when their national library was sold off to the highest bidder. When they said "It seems our country does not value our own culture anymore". Books lasted hundreds of years. Good luck trying to read a floppy from the 90s, or even DVDs that are already beyond their lifetime and are a very recent medium. It gets worse when you read the fine print of the SSD specifications, wherein they state that an SSD may lose all its data after 2 weeks without power, and data retention rates are at less than 99%, meaning they will degrade after the first year of use. And don't get me started on SMR HDDs, I lost enough drives already :D Humanity has a backup problem. We surely live in Orwellian times because of it. [1] https://youtube.com/watch?v=NdZxI3nFVJs https://youtube.com/watch?v=NdZxI3nFVJs
  - h4ck_th3_pl4n3t 2y ago
    I was curious whether these claims were true or not, so I looked it up. https://images.samsung.com/is/content/samsung/assets/pl/memory-storage/PM893_2.5_SSD_Datasheet_v1.41.pdf https://images.samsung.com/is/content/samsung/assets/pl/memo... Damn. 3 months for my SSD.
    - stogot 2y ago
      I’ve had SSDs sitting for years thought it was safe. Should I be worried or is this a liability warning?
lizknope 2y ago
Tape doesn't last forever either. https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations LTO-1 started in 2000 and the current LTO-9 spec is from 2021. But it only has backwards compatibility for 1 to 2 generations. You can't read an LTO-6 tape in an LTO-9 drive. https://en.wikipedia.org/wiki/Sticky-shed_syndrome https://en.wikipedia.org/wiki/Sticky-shed_syndrome > Sticky-shed syndrome is a condition created by the deterioration of the binders in a magnetic tape, which hold the ferric oxide magnetizable coating to its plastic carrier, or which hold the thinner back-coating on the outside of the tape.[1] This deterioration renders the tape unusable. Stiction Reversal Treatment for Magnetic Tape Media https://katalystdm.com/digital-transformation/tape-transcription/stiction-reversal-treatment/ https://katalystdm.com/digital-transformation/tape-transcrip... > Stiction can, in many cases, be reversed to a sufficient degree, allowing data to be recovered from previously unreadable tapes. This stiction reversal method involves heating tapes over a period of 24 or more hours at specific temperatures (depending on the brand of tape involved). This process hardens the binder and will provide a window of opportunity during which data recovery can be performed. The process is by no means a permanent cure nor is it effective on all brands of tape. Certain brands of tape (eg. Memorex Green- see picture below) respond very well to this treatment. Others such as Mira 1000 appear to be largely unaffected by it. Data migration and periodic verification is the answer but it requires more money to hire people to actually do it. I've got files from 1992 but I didn't just leave them on a 3.5" floppy disk. They have migrated from floppy disk -> hard drive -> PD phase change optical disk -> CD-R -> DVD-R -> back to hard drive I verify all checksums twice a year and have 2 independent backups.
- adrian_b 2y ago
  While tape does not last forever, the LTO tapes are specified for at least 30 years. The more serious problem is as you say that the older drives become obsolete. Even so, if you start using an up-to-date LTO format you can expect that suitable new tape drives will be available for buying at least 10 years in the future. For HDDs, the most that you can hope is a lifetime of 5 years, if you buy the HDDs with the longest warranties.
  - 2y ago
bogwog 2y ago
This article is too vague. It sounds like they're talking about the physical drive not working, but they're giving examples where you can't playback because you need to install the correct old software, plugins, etc... Which doesn't have anything to do with hard drives. So what's actually wrong with hard drives for archival? Do they deteriorate? Do they "rot" like DVDs/blurays/etc have been known to do? Or is this just an ad for their archival service?
- wmf 2y ago
  Hard drives are known to suffer "sticktion" where the heads get stuck to the platter and either the drive won't spin up or it spins up and the heads damage the platters. I imagine hard drives could also have bad capacitors but I haven't heard of that happening.
  - shiroiushi 2y ago
    >I imagine hard drives could also have bad capacitors but I haven't heard of that happening. That's very unlikely. If you're thinking of the "capacitor plague" of the 2000s, that only affected electrolytic capacitors, since it was caused by the Chinese poorly copying the formula for capacitor electrolyte. I don't believe hard drives used electrolytic capacitors in that time period, simply due to their size, though I could be wrong.
  - bogwog 2y ago
    That seems like it could be solved by (carefully) disassembling the drive for long term storage, adding a thin piece of paper or tape under the heads, etc.
    - wmf 2y ago
      Disassembling a drive just allows dust to get in and cause more damage. Come to think of it, in recent drives the heads fully unload onto a ramp so they're probably less likely to stick.
  - zh3 2y ago
    In the old drives of 5" HDDs the head stepper motor shaft was external, and if a drive got stuck a slight twist of the stepper shaft would unstick the heads after which the drive would spin up (well, as long as it didn't rip heads off the HDA so it was always a calculated risk). Happened to me when I got a call out to a large UK outfit who'd have an extended power cut and knew recovery was going to be fun. First stop was a particularly critical PC which had exactly this problem, so open the case, touch the HDD just right and off it went - happy with that, and to the next item. Anyway, the recovery operation went well, and this particular incident came floating back by way of a hushed comment from a manager a few years later about this tech who'd come in to help with the recovery, and who'd "...laid his hands on the PC, and it came back to life!" :)
kkfx 2y ago
People should learn a thing: data are not tied to the physical media hosting them, like words on paper, and the sole way to preserve data is migrating them from a physical support to another regularly, also converting their formats sometimes, because things changes and an old format could end up unreadable in the future. We can't preserve bits like books.
- Thrymr 2y ago
  > We can't preserve bits like books. The only reason we have any copies of "books" (i.e. long written works) from the ancient world is that they were painstakingly copied over centuries from one medium to another, by hand for most of that time.
  - kkfx 2y ago
    Definitively right, above I intend we can't preserve bits simply collecting them in libraries of physical supports like we do with shelves full of books or folders of archived sheets. With bits there is no "original" and "copy", any copy is still "original", we do not loose information or introduce changes because of that.
hulitu 2y ago
> Of the thousands and thousands of archived hard disk drives from the 1990s that clients ask the company to work on, around one-fifth are unreadable Some 25 years ago, the hardest part in booting some Apollo workstations, was to make hard drives spin.
bell-cot 2y ago
Policy at $Job - all important data is backed up to a rotation of high-quality hard drives. Which are stored off-site, powered down. Every N weeks, each one of them is powered up (in an off-line system) and checked - both with the SMART long test, and `zfs scan` (which verifies ZFS's additional anti-bit-rot checksums for the data). Yes, it's a bit of a PITA. OTOH, modern HD's are huge, so a relative few are needed. And we've lost 0 bits of our off-site data in our >25 years of using that system.
nayuki 2y ago
I had to do a double take on this, as I associate Iron Mountain as the brand that shreds papers and hard drives as their most common service.
- RobRivera 2y ago
  The second M doesn't help. Almost like the title was purposely crafted to mislead you to draw eyeballs.
- 1oooqooq 2y ago
  Well, this one is storing spinners POWERED DOWN... So it's petty much a slower data wiping service :shrug_emoji
- dredmorbius 2y ago
  Same firm. "Data management" sometimes means destruction, sometimes means preservation. <https://en.wikipedia.org/wiki/Iron_Mountain_(company) https://en.wikipedia.org/wiki/Iron_Mountain_(company)>
thomassmith65 2y ago
Relevant: "Sony Is Killing the Blu-ray" https://news.ycombinator.com/item?id=40880077 https://news.ycombinator.com/item?id=40880077 Ah, this stupid industry.
sgarland 2y ago
Related, CD-Rs. When I left my submarine in 2013, they (by which I mean the entire Virginia class) were still using them to store archived logs, despite my explanation that they’d be lucky to get a decade out of them. The first chosen storage location was literally the hottest part of the engine room, right in between the main engines. Easily 120+ F at all times. After protest, we moved ours to a somewhat cooler location. Still hot, and still with atmospheric oil and other fun chemicals floating around. I look forward to the first time logs from a few decades ago are required, and the media is absolutely dead. EDIT: they weren’t even Azo dye, they were phthalocyanine. A decade was probably generous.
- hunter-gatherer 2y ago
  Knowing nothing of submarines or seafaring, I'm genuinely curious as to what is logged on a ship that may be necessary a decade later?
  - Mountain_Skies 2y ago
    No idea what they actually log but it seems like performance data from the propulsion system under a wide range of conditions would be useful when designing the next generation of such systems.
  - sgarland 2y ago
    Everything. Someone could bring a lawsuit up years later, and logs would be necessary to determine if they had standing. The aforementioned optical media storage was specific to the nuclear reactor and electric plant; I think everyone else’s were stored differently. Not positive. EDIT: sibling comment below mentions performance data. Yes, that too. I graphed (nuclear) fuel consumption on one underway, and was surprised to find it didn’t match expected. My Captain was also surprised, and thrilled, because it meant he got to be more important (fair enough; who doesn’t want to be listened to by their boss?)
    - bushbaba 2y ago
      If lawsuit related. Maybe they wanted the backups to fail?
crazygringo 2y ago
As counterintuitive as it may be, it seems to me like the only reliable long-term storage for data is with commercial cloud providers. Any time you're physically warehousing old hard drives and whatnot, they're going to be turning into bricks. Whereas with cloud providers, they're keeping highly redundant copies and every time a hard drive fails, data gets copied to another one. And you can achieve extreme redundancy and guard against engineering errors by archiving data simultaneously with two cloud providers. Is there any situation where it makes sense to be physically hosting backups yourself, for long-term archival purposes? Purely from the perspective of preserving data, it seems worse in every way.
- thadt 2y ago
  ^ This. Physical media is continuously degrading. Large storage systems work by regularly reading, verifying, and replicating data - it is always doing backups and restores. If this isn't happening actively and regularly, your data will cease to exist at some point in time. Whether we collectively need to store all these things is another question entirely. But if we want to keep it - we'll have to do the work to keep it maintained.
- otabdeveloper4 2y ago
  > they're keeping highly redundant copies and every time a hard drive fails, data gets copied to another one Or so they say. It's not like you can double-check. > Is there any situation where it makes sense to be physically hosting backups yourself, for long-term archival purposes? Yes, political and legal risks. There's no guarantee your cloud won't terminate your account for any of a thousand reasons in the future.
  - crazygringo 2y ago
    > It's not like you can double-check. Don't they publish SLA numbers? The reliability of the major cloud providers seems quite well-established. > There's no guarantee your cloud won't terminate your account Two cloud providers pretty much guarantees against that -- the idea that two would terminate it simultaneously is vanishingly small.
TZubiri 2y ago
I'm launching a competitor for Iron Mountain. It's called DevNull LLC. Just send us your files! We'll take care of it don't worry.
jwsteigerwalt 2y ago
I grew up in Pittsburgh. When I was flying in and out of the Pittsburgh airport (usually to Atlanta) during and after college, I would often see Iron Mountain uniformed employees waited for standby seats carrying their little pelicans cases…
steve3055 2y ago
It sounds like the only reliable backup media is punched paper tape.
steve3055 2y ago
I once pressed my boss into having off-premises storage of documents so we could still manufacture product in a new facility if the current facility burnt down. Unfortunately, someone started the habit of sending the primary documents to the same facility if the product was deprecated. One day, that off-premises facility burnt down and all the contents was lost. I think it was a regular self-storage space. That aside, this sounds extremely old-fashioned, but it seems to me that the only media that is acceptable for long-term storage is going to be punched paper tape. How long does paper last? How long do the holes in it remain readable? Can it be spliced and repaired?
elzbardico 2y ago
The solution is metallic punch cards.
- exhilaration 2y ago
  How about QR codes instead?
somat 2y ago
For long term archiving, the fundamental hard problem is the storage density, The further the storage unit size drifts below human scale the harder it is to long term archive. I think for the average person, the best thing to do, for long term archives is to take advantage of sturgeons law, "90 percent of everything is crap". triage the things you want to archive to a minimum, then print them out, at human scale, on paper. have physical copies of the photos you want to keep, listings of the code you are proud of, correspondence that is dear to you. This will last, with no intervention, a very long time. Because as is increasingly becoming obvious, once the format drifts below human scale the best way to preserve data is to manage the data separate from the medium it is stored on with a constant effort to move it to a current medium. where it easily evaporates once vigilance drifts.
hilbert42 2y ago
"Of the thousands and thousands of archived hard disk drives from the 1990s that clients ask the company to work on, around one-fifth are unreadable." Why is this surprising? It's been known for decades that magnetic media loses remanence at several percent a year. It's why old sound tape recordings sound noisy or why one's family videotapes of say a wedding are either very noisy or unreadable 20 or so years later. Given that and the fact that hard disks are already on the margin of noise when working properly it's hardly surprising. The designers of hard disks go to inordinate lengths to design efficient data separators. These circuits just manage to separate the hardly-recognizable data signal from the noise when the drive is new and working well so the margin for deterioration is very small. The solution is simple, as the data is digital it should be regenerated every few years. Frankly I'm amazed that such a lax situation can exist in a professional storage facility. Edit: has this situation developed because the digital world doesn't know or has forgotten that storing data on magnetic media is an analog process and such signals are deteriorated by analog mechanisms?
- Nemo_bis 2y ago
  I'm amazed too. The mere fact that they accept storing hard disks powered off for years is a big red flag. It's in everyone's list of top things to avoid doing, not just in archiving circles: https://www.backblaze.com/blog/top-5-not-so-obvious-backup-archive-tips-for-creative-agencies/ https://www.backblaze.com/blog/top-5-not-so-obvious-backup-a... The final warning was the 2019 news of the 2008 Universal fire. https://en.wikipedia.org/wiki/2008_Universal_Studios_fire https://en.wikipedia.org/wiki/2008_Universal_Studios_fire
jcpham2 2y ago
A hard drive from 1995 will most likely be formatted FAT16/or FAT32 and the last Windows OS that reads that filesystem by default is Windows XP - I keep an old XP workstation operational as a test rig for reading data on old hard drives. Or I have to mount them in another OS that isn't Windows. It's more than just adjusting DAW settings and updating plugins at this point you need to know that around 2000 the filesystems completely changed with NTFS and added security that wasn't present before. By the time Vista/7 FAT hard drive support is gone from Microsoft land. There are of course add-ons and such but you still need to _know_ this happened and FAT drives look unformatted in modern Windows.
- iamtedd 2y ago
  What are you talking about? I'm able to create and read FAT-formatted USB drives today on Windows 11. In fact, Microsoft is adding features to Windows' FAT driver: https://www.bleepingcomputer.com/news/microsoft/microsoft-removes-fat32-partition-size-limit-in-windows-11/ https://www.bleepingcomputer.com/news/microsoft/microsoft-re...
deleted 2y ago
[deleted]
0ID 2y ago
An interesting topic would be the debate on how to store data for extremely long periods of time, several hundred thousand years, for example documentation of nuclear waste sites from power plants... Any ideas?

197 points by severine 2y ago

Does it mean LTO tape for the win then? We’re about to start a project to build an LTO-9 based in-house backup system. Any suggestions for DIY Linux based operation doing it “correctly” would be appreciated. Preliminary planning is to have one drive system on in our primary data center and another offsite at an office center where tapes are verified before storage in locked fireproof storage cabinet. Tips on good small business suppliers and gear models would be great help.

antisthenes 2y ago
What's your budget?
- alchemist1e9 2y ago
  $50K if needed but it doesn’t look to need that. 2PB initial data. predicted 1PB/year with around 10-30%/year rate of growth of rate of growth (acceleration?)
akira2501 2y ago
> fireproof storage cabinet. Nothing is fire proof. Is the cabinet "fire suppression system liquid" proof? > Tips on good small business suppliers and gear models would be great help. Hire an auditor would be my advice. Every business is different. I am, just now, having flashbacks of when I was in a SOX environment and had to regularly contract with them... and while the experience can be somewhat unpleasant I've often found good auditors to be extremely knowledgeable about solutions and their practical implementation considerations.
- hinkley 2y ago
  Pretty sure the pyramids at Giza count as fireproof boxes. But that’s out of most people’s price range.
  - burnished 2y ago
    Not if the fire is big enough.
    - hinkley 2y ago
      When a catastrophic event is catastrophic enough, your other problems cease to be your problem.
  - akira2501 2y ago
    They're not fully sealed. There are two shafts which connect the King's chamber to the exterior of the pyramid. The lower ritual congregation area is not fully sealed off from the upper chambers either. Which means bats are a constant problem in pyramids. You could easily get a fire going inside one.
    - dredmorbius 2y ago
      Fortunately, it actually is somewhat challenging to get a fire going inside a bat.
      - akira2501 2y ago
        Famously, their feces are not at all challenging to set on fire.
        dredmorbius 2y ago
        Still on the outside of the bats ;-)
hinkley 2y ago
The advice I got long ago from an IT guy was: if you wait long enough, tape will be on top again. That was a long time ago but I’ve peeked in at backup systems in the intervening years and it does seem to hold true over time. But it really depends how much data you have. My ex dropped a single HDD in a safety deposit box at CoB, N times per week and fetched back the oldest disk. I don’t think she ever said how many were in there but I doubt it was more than three. I think the CTO took one home with him once per week. The silly thing about most of this set up is that the office, the bank, and the data center were all within half a kilometer of each other. If something bad happened to that part of town they only had the infrequent offsite backup.
- katbyte 2y ago
  20tb and under is easy. 500tb is hard
- exe34 2y ago
  the sort of bad that would render all three unreadable would probably melt the rest of the infrastructure the company relies on for business anyway.
- marcosdumay 2y ago
  Every time I've looked, tapes were on top "again" for large scale archival. And I've been looking for ~20 years by now. I don't get where people get the impression that X was at the top right before tapes got that last innovation (where X here is most often HDDs, but not always). But that's always the impression, and tapes are always on top. People also have been working with 3D phase change drives since the 90s. Those always promise to replace tapes. But nobody ever got them robust enough to leave the labs.
  - hakfoo 2y ago
    Tape might win for large-scale systems, but it's basically dead for home-office scale. You used to be able to get modestly priced tape units for home use from the old "connects to the floppy controller" units with capacities in the tens of megabytes, up to some late-gen SCSI/IDE/parallel/early USB models that would be a couple of gigabytes, but still at home-friendly prices. What's today's answer? An enterprise-grade device that might put 10TB+ on a tape, but comes with a four-figure price tag and isn't really sold at Best Buy. If I want to back up the house today (maybe 4 active PCs, 5-6Tb of total space), affordable choices are pretty much disc-based. I could choose a cheap NAS (ended up doing that with an old fanless Atom machine and a used 12TB datacentre drive) or get a USB-attached external drive. Even if I used BD-XL media, even my modest needs would be dozens of discs, plus getting a writer in a shrinking market. There are plenty of datahoarders with much bigger needs, but even for them, tape is completely outside the addressable market.
WaitWaitWha 2y ago
Q: Why not archival M-Disc?
- actionfromafar 2y ago
  A: Are there any "real" M-Discs for purchase anymore? (I.e. not just rebranded regular dye-discs.)
  - netrap 2y ago
    Possibly in DVDR.
- alchemist1e9 2y ago
  Amount of data makes it less realistic. We have around 2PB data currently and expect to grow around 1PB next year with maybe 10-30% annual growth rate.
bluedino 2y ago
Tapes are fun. You can fit a petabyte of data in a bankers box! The problem quickly becomes: - Do we have a drive that can read this tape? - Do we have server we can connect it to? - Do we have storage we can extract it to? (go ask your internal IT team for 10TB of drivespace...) - What program did we create this tape with? Backup Exec, Veritas, ArcServe, SureStore - You have the encryption keys, right? - How much of this data already exists on the previous months backup? - Who's going to pay for the storage to move it to Glacier/etc? -How long is it going to take to upload?
- aftbit 2y ago
  If you are having trouble getting 10TB of disk space from IT, you have bad IT. Not saying that's uncommon or anything, but 10TB fits on one external hard drive for $300 from Best Buy, or less than $1000/mo using EBS on AWS if you need some better guarantees and are all-in on the cloud.
- DaiPlusPlus 2y ago
  > Tapes are fun. You can fit a petabyte of data in a bankers box! Yes, though those ultra-thin M.2 NVMe drives could probably top that now. > Do we have a drive that can read this tape? Don't let this be a problem in the first place: buy 4 tape drives and keep 2 of them in your cold/offsite/airgapped storage site (2 in case 1 fails, so you can use the remaining drive to transfer everything to a newer format). > Do we have server we can connect it to? Significant hardware should not (and is not) necessary: the LTO-7/8/9 drives I see for sale right now seem to be using either USB 3.0, Thunderbolt, or SAS connections; USB and Thunderbolt can be handled by any computer you can find at a PC recycler today; while any old desktop can handle SAS with a $80 HBA card. > Do we have storage we can extract it to? (go ask your internal IT team for 10TB of drivespace...) 10TB isn't a good example (case-in-point: I have a 3-year-old stack of unused 12TB WD drives less than 3 feet away from me). That said, if you're enterprise-y enough for a $4000 LTO-9 drive, then you probably also have a SAN that's chock full of drives, so being able to provision a 10TB+ LUN should be implicit. > What program did we create this tape with? Backup Exec, Veritas, ArcServe, SureStore Ideally, none of those; instead, good ol' Perl and `dd`. > You have the encryption keys, right? I don't encrypt my backups to avoid this problem. My old data archives have little exploitable value for any potential attacker; and I imagine I'd store backup tapes this important in a fireproof safe in my parents' house or something. I'd only encrypt the entire tape if the tape were to leave my custody. I appreciate that this is not for everyone, and it's probably illegal for some people/orgs to not encrypt backups too anyway (HIPAA, etc). > How much of this data already exists on the previous months backup? Incremental/Differential backups are still a thing. > Who's going to pay for the storage to move it to Glacier/etc? No-one should. Cold data backups should/must always be in the custody of a designated responsible officer of the company. BUUUT, I guess there's nothing wrong with storing an encrypted copy in Cloud storage (as in S3/Glacier/AzureBlobs - not OneDrive...). I actually do this right now thanks to the smooth and painless integration in my Synology NAS. It costs me about $15/mo to store all these TBs in S3. > How long is it going to take to upload? Consider that it's 2024 - a company with LTO-9 and SAN is probably going to have a metro-ethernet IP connection at 10Gbps or even faster. At home I have a 10Gbps symmetric connection from Ziply (it's $300/mo and they give you an SFP module, which I put into my Ubiquiti UDM): so the limiting factor here is not my upload speed, but my drive read speed (LTO-9 drives seem to read at about 2-3Gbps raw/uncompressed?)
gosub100 2y ago
Make sure the bandwidth exists to keep up with the write speed of the LTO drive. For instance, the write speed for LTO-6 (which I own as a hobbyist) is around 300MB/s, but line speed of gigabit Ethernet is about 100MB/s. Translate those numbers to LTO-9 and make sure that the NAS, network, or local storage can keep up. It's not a deal-breaker to underflow the drive, but it causes the tape to stop, rewind, and re-buffer (called shoe-shining) which takes more time and causes unnecessary wear on the drive and cartridges.
adrian_b 2y ago
LTO-9 tapes can be easily found on Amazon in many countries, made by IBM, HP, Quantum or Fuji. The vendor does not matter, whichever happens to be cheaper at the moment is fine. For the tape drives, the internal drives can be cheaper by around 10%, but I prefer the tabletop drives, because they are less prone to accumulate dust, especially if you switch them on only when doing a backup or a retrieval. The tape drives have usually very noisy fans, because they are expected to be used in isolated server rooms. I believe that the cheapest tape drives from a reputable manufacturer are those from Quantum. I have been using a Quantum LTO-7 tape drive for about 7 or 8 years and I have been content with it. Looking now at the prices, it should be possible to find a tabletop LTO-9 drive for no more than $5000. Unfortunately, the prices for tape drives have been increasing. When I have bought an LTO-7 tabletop drive many years ago it was only slightly more than $3000. The tapes are much cheaper and much more reliable than hard disks, but because of the very expensive tape drive you need to store a few hundred TB to begin to save money over hard disks. You should normally make at least two copies of any tape that is intended for long-term archiving (to be stored in different places), which will shorten the time until reaching the threshold of breaking even with HDDs. Even if there are applications that simulate the existence of a file system on a tape, which can be used even by a naive user to just copy files on a tape, like copying files between disks, they are quite slow and inefficient in comparison to just using raw tape commands with the traditional UNIX utility "mt". It is possible to write some very simple scripts that use "mt" and which allow the appending of a number of files to a tape or the reading of a number of consecutive files from a tape, starting from the nth file since the beginning of a tape. So if you are using only raw "mt" commands, you can identify the archived files only by their ordinal number since the beginning of the tape. This is enough for me, because I prepare the files for backup by copying them in some directory, making an index of that directory, then compressing it and encrypting it. I send to the tape only encrypted and compressed archive files, so I disable the internal compression of the tape drive, which would be useless. I store the information about the content of the archives stored on tapes (which includes all relevant file metadata for each file contained in the compressed archives, including file name, path name, file length, modification time, a hash of the file content) in a database. Whenever I need archived data, I search the database, to determine that it can be found, for instance in tape 63, file 102. Then I can insert the corresponding cartridge in the drive and I give the command to retrieve file 102. I consider much better the utility "mt" of FreeBSD than that of Linux. The Linux magnetic drive utilities have seen little maintenance for many years. Because of that, when I make backups or retrievals they go to a server that runs FreeBSD, on which the SAS HBA card is installed. When a tabletop drive is used, the SAS HBA card must have external SAS connectors, to allow the use of an appropriate cable. I actually reboot that server into FreeBSD for doing backups or retrievals, which is easy because I boot it from Ethernet with PXE, so I can select remotely what OS to be booted. One could also use a FreeBSD VM on a Linux server, with pass-through of the SAS HBA card, but I have not tried to do this. My servers are connected with 10 Gb/s Ethernet links, which does not differ much from the SAS speed, so they do not slow much the backup/retrieval speed. I transfer the archive files with rsync over ssh. On slow computers and internal networks one can use rsync without ssh. I give the commands for the tape drive from the computer that is backed up, as one line commands executed remotely by ssh. The archive that is transferred is stored in a RAMdisk before being written on the tape, to ensure that the tape is written at the maximum speed. I write to the tape archive files that have usually a size of up to about 60 GB (I split any files bigger than that; e.g. there are BluRay movies of up to 100 GB). The server has a memory of 128 GB, so I can configure on it a RAMDdisk of up to 80 GB without problems. This method can be used even with a slow 1 Gb/s or 2.5 Gb/s network, but then uploading a file through Ethernet would take much more time than writing or reading the tape. There is one weird feature of the raw "mt" commands, which is poorly documented, so it took me some time to discover it, during which I have wasted some tape space. When you append files to a partially written tape, you first give a command to go to the end of the written part of the tape. However, you must not start writing, because the head is not positioned correctly. You must go 2 file marks backwards, then 1 file mark forwards. Only then is the head positioned correctly and you can write the next archived file. Otherwise there would be 1 empty file intercalated at each point where you have finished appending a number of files and then you have rewound the tape and then you have appended again other files at the end.
wazoox 2y ago
Archival is more of a process than only a question of media. First you must create a proper database of your archived data. Maybe you want to do 3 copies, not two. Maybe you want to use two different archival formats such as tar and LTFS, just in case. Maybe you want to source your media from both available producers (Sony and Fuji) because in the long run, maybe one or the other may grow some funky error mode or corruption problem. Etc. Also check my tape management primer: https://blogs.intellique.com/tech/2022/01/27#TapeCLI https://blogs.intellique.com/tech/2022/01/27#TapeCLI

antisthenes 2y ago

What's the scenario where you cannot take the old 1990's hard drive and back up its data in multiple cloud service providers cold storage (Azure/AWS/GCP) and have to keep the obsolete physical media on hand? I'm struggling to understand why these miles of shelves filled with essentially hardware junk haven't been digitized at the time when this media worked and didn't experience read issues. The article doesn't really provide an explanation for this other than incompetence and the business biting off more than it can reasonably chew. I'd be furious if I paid for a service that promised to archive my data, and 10-15 years later told me 25% of it was unreadable. I mean it's not like it was a surprise either. These workflows became digital 2-3 decades ago. There was plenty of time to prepare and convert this. That's kind of what I'm paying you for. As always, seems like the simple folk of /r/datahoarder and other archivist communities are more competent than a legacy industry behemoth.

akira2501 2y ago
> the business biting off more than it can reasonably chew It's hoarding behavior. They paid "a lot" of money for it, have no idea how to further exploit it, but can't shake the feeling that it might be massively valuable one day. The only difference is they pay someone to hold their hoard for them.
- eitally 2y ago
  Alternatively ... they are forced to maintain it for compliance reasons, especially when it comes to healthcare, finance and other regulated industries (defense, in particular). This even applies to manufacturing companies assembly medical devices and defense products: all the data about the supply chain, the engineering designs & changes, the manufacturing and quality testing, and shipment needs to be kept for XX years and is subject to both audit by regulatory agencies and to legal discovery.
  - akira2501 2y ago
    Those are all things which can be printed out and stored in alternative forms and possibly even recreated from other data. It's also the case that much of that data will never be permanently at rest and so several archive copies of the data exist. Recordings of performances are an entirely different category of artefact.

akira2501 2y ago

> It may sound like a sales pitch, but it’s not; it’s a call for action Your entire article sounds like a sales pitch. Your solution is, well, it's bad, but trust us, we can maybe recover it anyways. Otherwise your article fails to convey anything meaningful.

derefr 2y ago
No, the call-to-action being referenced in the article is "stop archiving to hard drives" (and use tape instead, every other industry does.)
- ganoushoreilly 2y ago
  I think it is or at least tries to be more than that. Not only stop archiving to HD's but understand the dependency requirements which is a whole secondary problem.
- natch 2y ago
  The last time I used a tape it broke right off the reel the first time I used it. Good name brand tape drive, good name brand tape. I felt pretty burned by that experience after the money spent and the result.
- shiroiushi 2y ago
  Tape drives aren't economical. If you're a big company, sure, they make sense, but for individuals and small companies, they really don't'. Hard drives are absolutely the only realistic and affordable way to back up data. They're not bulletproof though, so you need multiple hard drives, and you need a backup strategy that rotates them, so even if one fails, you haven't lost too much.
  - aidenn0 2y ago
    They are already paying another company to warehouse the drives; surely that company could offer to rent LTO drives going back several generations?
  - 2y ago

MisterTea 2y ago

Makes me wish we didn't stop advancing optical media technology to where we have cheap and reliable archival quality 1TB discs for a few bucks each. I guess LTO is the best option for personally controlled archival.

0cf8612b2e1e 2y ago
We haven’t, but sadly the technology is locked to big tech. Microsoft has demoed some cool technology where they store data in glass, Project Silica. Sadly, it seems unlikely this will ever be available to consumers. One neat aspect of the design is that writing data is significantly higher power than reading. So you can keep your writing devices physically separated from the readers and have no fear that malicious code could ever overwrite existing data plates. Some blurbs Project Silica is developing the world’s first storage technology designed and built from the media up to address humanity’s need for a long-term, sustainable storage technology. We store data in quartz glass: a low-cost, durable WORM media that is EMF-proof, and offers lifetimes of tens to hundreds of thousands of years. This has huge consequences for sustainability, as it means we can leave data in situ, and eliminate the costly cycle of periodically copying data to a new media generation. We’re re-thinking how large-scale storage systems are built in order to fully exploit the properties of the glass media and create a sustainable and secure storage system to support archival storage for decades to come! We are co-designing the hardware and software stacks from scratch, from the media all the way up to the cloud user API. This includes a novel, low-power design for the media library that challenges what the robotics and mechanics of archival storage systems look like. https://www.microsoft.com/en-us/research/project/project-silica/ https://www.microsoft.com/en-us/research/project/project-sil...
- actionfromafar 2y ago
  Now this is proper Sci-Fi tech! Data crystals, like 1960s Star Trek!
- yellow_postit 2y ago
  I'm partial to their DNA work: https://www.microsoft.com/en-us/research/project/dna-storage/ https://www.microsoft.com/en-us/research/project/dna-storage...

simonw 2y ago

My understanding is that the only reliable way of long-term digital archival storage is to refresh the media you are storing things on every few years, copying the previous archives to the fresh storage. Since storage constantly gets cheaper, 100GB first stored in 2001 can be stored on updated media for a fraction of that original cost in 2024.

abracadaniel 2y ago
Pretty much. You see hobbyists getting data off of 30+ year old hard drives for the novelty of it, but I can’t imagine relying on that as a preservation copy. Optical media rots, magnetic media rots and loses magnetic charge, bearings seize, flash storage loses charge, etc. Entropy wins, sometimes much faster than you’d expect.
- animal531 2y ago
  Sometimes they fail for other reasons as well, such as improper storage. Back in the 90's to 00's a friend had a collection of cd's that he'd written, but he stored them in a big sleeved folder container. The container itself caused them to warp slightly, which made them unusable. I took a few for testing and managed to unbend them after some time, which turned them back into a working state. [Note: That's the most apostrophes I've ever used in a sentence, it feels dirty]
  - aimor 2y ago
    A little dirty according to CMOS. Ought to be '90s not 90's, cds not cd's. https://www.chicagomanualofstyle.org/book/ed18/part2/ch09/psec035.html https://www.chicagomanualofstyle.org/book/ed18/part2/ch09/ps... https://www.chicagomanualofstyle.org/qanda/data/faq/topics/Punctuation/faq0051.html https://www.chicagomanualofstyle.org/qanda/data/faq/topics/P...
    - animal531 2y ago
      Thanks for the tip!

mercurialuser 2y ago

There are several articles about film preservation in digital format. Every X years all the data is "upgraded", from LTOn to LTOn+1 or +2. So it may sound like a sales pitch but I consider it more a warning notice

esafak 2y ago
That means the hardware and the file format.

Clamchop 2y ago

They have lots of problems: 1. Incomplete copies with missing dependencies. 2. Old software and their file formats with a poor virtualization story. 3. Poor cataloging. 4. Obsolete physical interfaces, file systems, etc. 5. Long-term cold storage on media neither proven nor marketed for the task. Managing archives is just a cost center until it isn't, and it's hard to predict what will have value. The worst part of this is that TFA discusses mostly music industry materials. Outside parties and the public would have a huge interest in preserving all this, but of course it's impossible. All private, proprietary, copyrighted, and likely doomed to be lost one way or another. Oh well.

cookiengineer 2y ago
Related documentary that comes to mind: Digital Amnesia (2014) [1] It broke my heart seeing those librarians in disbelief when their national library was sold off to the highest bidder. When they said "It seems our country does not value our own culture anymore". Books lasted hundreds of years. Good luck trying to read a floppy from the 90s, or even DVDs that are already beyond their lifetime and are a very recent medium. It gets worse when you read the fine print of the SSD specifications, wherein they state that an SSD may lose all its data after 2 weeks without power, and data retention rates are at less than 99%, meaning they will degrade after the first year of use. And don't get me started on SMR HDDs, I lost enough drives already :D Humanity has a backup problem. We surely live in Orwellian times because of it. [1] https://youtube.com/watch?v=NdZxI3nFVJs https://youtube.com/watch?v=NdZxI3nFVJs
- h4ck_th3_pl4n3t 2y ago
  I was curious whether these claims were true or not, so I looked it up. https://images.samsung.com/is/content/samsung/assets/pl/memory-storage/PM893_2.5_SSD_Datasheet_v1.41.pdf https://images.samsung.com/is/content/samsung/assets/pl/memo... Damn. 3 months for my SSD.
  - stogot 2y ago
    I’ve had SSDs sitting for years thought it was safe. Should I be worried or is this a liability warning?

lizknope 2y ago

Tape doesn't last forever either. https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations LTO-1 started in 2000 and the current LTO-9 spec is from 2021. But it only has backwards compatibility for 1 to 2 generations. You can't read an LTO-6 tape in an LTO-9 drive. https://en.wikipedia.org/wiki/Sticky-shed_syndrome https://en.wikipedia.org/wiki/Sticky-shed_syndrome > Sticky-shed syndrome is a condition created by the deterioration of the binders in a magnetic tape, which hold the ferric oxide magnetizable coating to its plastic carrier, or which hold the thinner back-coating on the outside of the tape.[1] This deterioration renders the tape unusable. Stiction Reversal Treatment for Magnetic Tape Media https://katalystdm.com/digital-transformation/tape-transcription/stiction-reversal-treatment/ https://katalystdm.com/digital-transformation/tape-transcrip... > Stiction can, in many cases, be reversed to a sufficient degree, allowing data to be recovered from previously unreadable tapes. This stiction reversal method involves heating tapes over a period of 24 or more hours at specific temperatures (depending on the brand of tape involved). This process hardens the binder and will provide a window of opportunity during which data recovery can be performed. The process is by no means a permanent cure nor is it effective on all brands of tape. Certain brands of tape (eg. Memorex Green- see picture below) respond very well to this treatment. Others such as Mira 1000 appear to be largely unaffected by it. Data migration and periodic verification is the answer but it requires more money to hire people to actually do it. I've got files from 1992 but I didn't just leave them on a 3.5" floppy disk. They have migrated from floppy disk -> hard drive -> PD phase change optical disk -> CD-R -> DVD-R -> back to hard drive I verify all checksums twice a year and have 2 independent backups.

adrian_b 2y ago
While tape does not last forever, the LTO tapes are specified for at least 30 years. The more serious problem is as you say that the older drives become obsolete. Even so, if you start using an up-to-date LTO format you can expect that suitable new tape drives will be available for buying at least 10 years in the future. For HDDs, the most that you can hope is a lifetime of 5 years, if you buy the HDDs with the longest warranties.
- 2y ago

bogwog 2y ago

This article is too vague. It sounds like they're talking about the physical drive not working, but they're giving examples where you can't playback because you need to install the correct old software, plugins, etc... Which doesn't have anything to do with hard drives. So what's actually wrong with hard drives for archival? Do they deteriorate? Do they "rot" like DVDs/blurays/etc have been known to do? Or is this just an ad for their archival service?

wmf 2y ago
Hard drives are known to suffer "sticktion" where the heads get stuck to the platter and either the drive won't spin up or it spins up and the heads damage the platters. I imagine hard drives could also have bad capacitors but I haven't heard of that happening.
- shiroiushi 2y ago
  >I imagine hard drives could also have bad capacitors but I haven't heard of that happening. That's very unlikely. If you're thinking of the "capacitor plague" of the 2000s, that only affected electrolytic capacitors, since it was caused by the Chinese poorly copying the formula for capacitor electrolyte. I don't believe hard drives used electrolytic capacitors in that time period, simply due to their size, though I could be wrong.
- bogwog 2y ago
  That seems like it could be solved by (carefully) disassembling the drive for long term storage, adding a thin piece of paper or tape under the heads, etc.
  - wmf 2y ago
    Disassembling a drive just allows dust to get in and cause more damage. Come to think of it, in recent drives the heads fully unload onto a ramp so they're probably less likely to stick.
- zh3 2y ago
  In the old drives of 5" HDDs the head stepper motor shaft was external, and if a drive got stuck a slight twist of the stepper shaft would unstick the heads after which the drive would spin up (well, as long as it didn't rip heads off the HDA so it was always a calculated risk). Happened to me when I got a call out to a large UK outfit who'd have an extended power cut and knew recovery was going to be fun. First stop was a particularly critical PC which had exactly this problem, so open the case, touch the HDD just right and off it went - happy with that, and to the next item. Anyway, the recovery operation went well, and this particular incident came floating back by way of a hushed comment from a manager a few years later about this tech who'd come in to help with the recovery, and who'd "...laid his hands on the PC, and it came back to life!" :)

kkfx 2y ago

People should learn a thing: data are not tied to the physical media hosting them, like words on paper, and the sole way to preserve data is migrating them from a physical support to another regularly, also converting their formats sometimes, because things changes and an old format could end up unreadable in the future. We can't preserve bits like books.

Thrymr 2y ago
> We can't preserve bits like books. The only reason we have any copies of "books" (i.e. long written works) from the ancient world is that they were painstakingly copied over centuries from one medium to another, by hand for most of that time.
- kkfx 2y ago
  Definitively right, above I intend we can't preserve bits simply collecting them in libraries of physical supports like we do with shelves full of books or folders of archived sheets. With bits there is no "original" and "copy", any copy is still "original", we do not loose information or introduce changes because of that.

hulitu 2y ago

> Of the thousands and thousands of archived hard disk drives from the 1990s that clients ask the company to work on, around one-fifth are unreadable Some 25 years ago, the hardest part in booting some Apollo workstations, was to make hard drives spin.

bell-cot 2y ago

Policy at $Job - all important data is backed up to a rotation of high-quality hard drives. Which are stored off-site, powered down. Every N weeks, each one of them is powered up (in an off-line system) and checked - both with the SMART long test, and `zfs scan` (which verifies ZFS's additional anti-bit-rot checksums for the data). Yes, it's a bit of a PITA. OTOH, modern HD's are huge, so a relative few are needed. And we've lost 0 bits of our off-site data in our >25 years of using that system.

nayuki 2y ago

I had to do a double take on this, as I associate Iron Mountain as the brand that shreds papers and hard drives as their most common service.

RobRivera 2y ago
The second M doesn't help. Almost like the title was purposely crafted to mislead you to draw eyeballs.
1oooqooq 2y ago
Well, this one is storing spinners POWERED DOWN... So it's petty much a slower data wiping service :shrug_emoji
dredmorbius 2y ago
Same firm. "Data management" sometimes means destruction, sometimes means preservation. <https://en.wikipedia.org/wiki/Iron_Mountain_(company) https://en.wikipedia.org/wiki/Iron_Mountain_(company)>

thomassmith65 2y ago

Relevant: "Sony Is Killing the Blu-ray" https://news.ycombinator.com/item?id=40880077 https://news.ycombinator.com/item?id=40880077 Ah, this stupid industry.

sgarland 2y ago

Related, CD-Rs. When I left my submarine in 2013, they (by which I mean the entire Virginia class) were still using them to store archived logs, despite my explanation that they’d be lucky to get a decade out of them. The first chosen storage location was literally the hottest part of the engine room, right in between the main engines. Easily 120+ F at all times. After protest, we moved ours to a somewhat cooler location. Still hot, and still with atmospheric oil and other fun chemicals floating around. I look forward to the first time logs from a few decades ago are required, and the media is absolutely dead. EDIT: they weren’t even Azo dye, they were phthalocyanine. A decade was probably generous.

hunter-gatherer 2y ago
Knowing nothing of submarines or seafaring, I'm genuinely curious as to what is logged on a ship that may be necessary a decade later?
- Mountain_Skies 2y ago
  No idea what they actually log but it seems like performance data from the propulsion system under a wide range of conditions would be useful when designing the next generation of such systems.
- sgarland 2y ago
  Everything. Someone could bring a lawsuit up years later, and logs would be necessary to determine if they had standing. The aforementioned optical media storage was specific to the nuclear reactor and electric plant; I think everyone else’s were stored differently. Not positive. EDIT: sibling comment below mentions performance data. Yes, that too. I graphed (nuclear) fuel consumption on one underway, and was surprised to find it didn’t match expected. My Captain was also surprised, and thrilled, because it meant he got to be more important (fair enough; who doesn’t want to be listened to by their boss?)
  - bushbaba 2y ago
    If lawsuit related. Maybe they wanted the backups to fail?

crazygringo 2y ago

As counterintuitive as it may be, it seems to me like the only reliable long-term storage for data is with commercial cloud providers. Any time you're physically warehousing old hard drives and whatnot, they're going to be turning into bricks. Whereas with cloud providers, they're keeping highly redundant copies and every time a hard drive fails, data gets copied to another one. And you can achieve extreme redundancy and guard against engineering errors by archiving data simultaneously with two cloud providers. Is there any situation where it makes sense to be physically hosting backups yourself, for long-term archival purposes? Purely from the perspective of preserving data, it seems worse in every way.

thadt 2y ago
^ This. Physical media is continuously degrading. Large storage systems work by regularly reading, verifying, and replicating data - it is always doing backups and restores. If this isn't happening actively and regularly, your data will cease to exist at some point in time. Whether we collectively need to store all these things is another question entirely. But if we want to keep it - we'll have to do the work to keep it maintained.
otabdeveloper4 2y ago
> they're keeping highly redundant copies and every time a hard drive fails, data gets copied to another one Or so they say. It's not like you can double-check. > Is there any situation where it makes sense to be physically hosting backups yourself, for long-term archival purposes? Yes, political and legal risks. There's no guarantee your cloud won't terminate your account for any of a thousand reasons in the future.
- crazygringo 2y ago
  > It's not like you can double-check. Don't they publish SLA numbers? The reliability of the major cloud providers seems quite well-established. > There's no guarantee your cloud won't terminate your account Two cloud providers pretty much guarantees against that -- the idea that two would terminate it simultaneously is vanishingly small.

TZubiri 2y ago

I'm launching a competitor for Iron Mountain. It's called DevNull LLC. Just send us your files! We'll take care of it don't worry.

jwsteigerwalt 2y ago

I grew up in Pittsburgh. When I was flying in and out of the Pittsburgh airport (usually to Atlanta) during and after college, I would often see Iron Mountain uniformed employees waited for standby seats carrying their little pelicans cases…

steve3055 2y ago

It sounds like the only reliable backup media is punched paper tape.

steve3055 2y ago

I once pressed my boss into having off-premises storage of documents so we could still manufacture product in a new facility if the current facility burnt down. Unfortunately, someone started the habit of sending the primary documents to the same facility if the product was deprecated. One day, that off-premises facility burnt down and all the contents was lost. I think it was a regular self-storage space. That aside, this sounds extremely old-fashioned, but it seems to me that the only media that is acceptable for long-term storage is going to be punched paper tape. How long does paper last? How long do the holes in it remain readable? Can it be spliced and repaired?

elzbardico 2y ago

The solution is metallic punch cards.

exhilaration 2y ago
How about QR codes instead?

somat 2y ago

For long term archiving, the fundamental hard problem is the storage density, The further the storage unit size drifts below human scale the harder it is to long term archive. I think for the average person, the best thing to do, for long term archives is to take advantage of sturgeons law, "90 percent of everything is crap". triage the things you want to archive to a minimum, then print them out, at human scale, on paper. have physical copies of the photos you want to keep, listings of the code you are proud of, correspondence that is dear to you. This will last, with no intervention, a very long time. Because as is increasingly becoming obvious, once the format drifts below human scale the best way to preserve data is to manage the data separate from the medium it is stored on with a constant effort to move it to a current medium. where it easily evaporates once vigilance drifts.

hilbert42 2y ago

"Of the thousands and thousands of archived hard disk drives from the 1990s that clients ask the company to work on, around one-fifth are unreadable." Why is this surprising? It's been known for decades that magnetic media loses remanence at several percent a year. It's why old sound tape recordings sound noisy or why one's family videotapes of say a wedding are either very noisy or unreadable 20 or so years later. Given that and the fact that hard disks are already on the margin of noise when working properly it's hardly surprising. The designers of hard disks go to inordinate lengths to design efficient data separators. These circuits just manage to separate the hardly-recognizable data signal from the noise when the drive is new and working well so the margin for deterioration is very small. The solution is simple, as the data is digital it should be regenerated every few years. Frankly I'm amazed that such a lax situation can exist in a professional storage facility. Edit: has this situation developed because the digital world doesn't know or has forgotten that storing data on magnetic media is an analog process and such signals are deteriorated by analog mechanisms?

Nemo_bis 2y ago
I'm amazed too. The mere fact that they accept storing hard disks powered off for years is a big red flag. It's in everyone's list of top things to avoid doing, not just in archiving circles: https://www.backblaze.com/blog/top-5-not-so-obvious-backup-archive-tips-for-creative-agencies/ https://www.backblaze.com/blog/top-5-not-so-obvious-backup-a... The final warning was the 2019 news of the 2008 Universal fire. https://en.wikipedia.org/wiki/2008_Universal_Studios_fire https://en.wikipedia.org/wiki/2008_Universal_Studios_fire

jcpham2 2y ago

A hard drive from 1995 will most likely be formatted FAT16/or FAT32 and the last Windows OS that reads that filesystem by default is Windows XP - I keep an old XP workstation operational as a test rig for reading data on old hard drives. Or I have to mount them in another OS that isn't Windows. It's more than just adjusting DAW settings and updating plugins at this point you need to know that around 2000 the filesystems completely changed with NTFS and added security that wasn't present before. By the time Vista/7 FAT hard drive support is gone from Microsoft land. There are of course add-ons and such but you still need to _know_ this happened and FAT drives look unformatted in modern Windows.

iamtedd 2y ago
What are you talking about? I'm able to create and read FAT-formatted USB drives today on Windows 11. In fact, Microsoft is adding features to Windows' FAT driver: https://www.bleepingcomputer.com/news/microsoft/microsoft-removes-fat32-partition-size-limit-in-windows-11/ https://www.bleepingcomputer.com/news/microsoft/microsoft-re...

deleted 2y ago

[deleted]

0ID 2y ago

An interesting topic would be the debate on how to store data for extremely long periods of time, several hundred thousand years, for example documentation of nuclear waste sites from power plants... Any ideas?