8 ms·

I'm more interested in the lifetime of the medium. Durable backup media for consumers is still a holy grail as I understand. M-DISC didn't hold up to their 1000

by sedatk 6y ago

I'm more interested in the lifetime of the medium. Durable backup media for consumers is still a holy grail as I understand. M-DISC didn't hold up to their 1000 year lifetime promise. Archival-grade DVD's are also not good enough as I understand. Syylex went bankrupt. I want a consumer-grade backup medium that can provide at least a 100 years of lifetime. That said, I was able to recover 90% of the voice recordings my father made between 1959-1963 on reel-to-reel tapes 60 years later. Tape can be very durable but what I recovered was analog voice very tolerant of errors. I'm not so sure about gigabytes snuck into an inch-square.

dudus 6y ago
piql is the company that did the github arctic vault. And they offer it as a service. Maybe not for general public but still . https://www.piql.com/arctic-world-archive/ https://www.piql.com/arctic-world-archive/
- sedatk 6y ago
  Arctic vault is VERY expensive, about $100 per gigabyte.
elorant 6y ago
The lifetime of the medium is half the equation. The biggest problem imho is the lifetime of the device. Say I give you an old floppy drive for 8" floppies from the seventies. Where would you connect it?
- sedatk 6y ago
  You're right but not all media are the same. Arctic Vault used an optical film with QR codes. You can theoretically even take its photo and decode it by hand if you want. They even added a Rosetta Stone to the entrance so, even if all the knowledge is lost, one can hypothetically decode the data stored there. For magnetic media, you need some more specialized equipment for sure.
- kmeisthax 6y ago
  Tapes are uniquely terrible at this. I'd argue it's the Achilles' heel of the medium, even moreso than the actual limitations of linear tape. First off, the drives themselves are going to be expensive brand-new. You're going to be paying thousands of dollars for a drive, and it's probably going to have some weird interconnect that you'll have to spend even more money and waste a PCIe slot on an adapter in order to use the drive. Most common is SAS or Fibre Channel; although there's at least one company selling drives in Thunderbolt enclosures for the Mac market. (Aside from all that, SAS is actually pretty cool for things like hard drive enclosures, since it has SATA compatibility. I have a jury-rigged disk shelf built out of an old HP SAS expander, a slightly-modified CD duplicator case, and some 5 1/4" hard drive bays.) Second, tape formats are constantly moving. LTO and IBM 3592 come out with a new format every 2-3 years and backwards compatibility is limited. Generally speaking you can only write one generation behind on LTO and read two generations back. So, if you want a format that's got drives still being made for it, you'll need to migrate every 5-7 years. Sure, the actual tape is shelf-stable for longer, but you're going to be buying spare drives or jumping on eBay if you want to keep old tapes around that long. (eBay is actually not a bad place to buy used tape drives, but the pricing varies wildly. It's perfect for hobbyists and small-fry IT outfits looking for cheap backup media. Absolutely terrible if you're a large outfit with reliability guarantees and support agreements to maintain.) Third, actually using a tape drive is a nightmare. First off, Windows hasn't shipped tape software since 2003 (I think?), so you'll be in the market for proprietary backup solutions. Second, if you're just writing data directly to the tape, you will shoe-shine for days. Common IT practice is to actually have a second disk array sitting in front of the tapes as a write cache and custom software to copy data to the tapes at full speed once all the slow nonsequential IO is done. Reading from tape doesn't have to worry about this as much, but the fact that you had to use custom software just to archive your files means that you now have proprietary archive formats to deal with. So you can have tapes that rely on both having access to working drives and licenses for proprietary backup utilities. (Of course, if you had decently fast SSDs and a parallel archival utility, you could sustain decent write speeds on tape. I actually wrote this myself as an experiment: https://github.com/kmeisthax/rapidtar https://github.com/kmeisthax/rapidtar and it can saturate the LTO-5 drive I tested this with.)
- userbinator 6y ago
  Floppies are an interesting case because the protocols and physical specifications are all documented publicly, which means that one could literally build a drive from scratch today --- the trickiest part being the heads, but considering that they are many orders of magnitude lower density than HDDs, it would not be a big obstacle in the future. (I believe 8" floppy drives have the same interface as 5.25" ones --- and there's no shortage of adapters from the retrocomputing community for those, some of them even open-source.) Tape is far more closed, AFAIK most of the common formats are proprietary and the specs are behind NDAs and other walls.
  - crististm 6y ago
    I have hobby books from 30 years ago that teach you how to build magnetic heads for cassette tapes. No pictures or anything but with enough patience you definitely can build one today, at home (even then). Mind you, the size of the gap is not that big of the problem if you put your mind at polishing.
cm2187 6y ago
Given the pace at which storage capacity increases, what is the rationale for not copying over your data every 5-10 years onto the next cheapest consumer mass storage of the moment? You get all your data in one place and you don’t have to deal with standards disappearing (I read 2020’s game consoles can’t read CDs anymore, people should rip their CD collection right now).
- Max_aaa 6y ago
  The time needed to actually do the copying.
  - drdavid 6y ago
    It seems like there may be a market there. Every 10 to 15 years, you send in your archived (and new/interim) personal data and get it back on the current top-tech storage medium. That way it's not stored in the cloud and you can keep moving the stored data forward without having to deal with it all yourself.
  - cm2187 6y ago
    Not really. You can buy a 18TB hard drive now. Even if your data is humongous and needs several of those, it will likely fit on a single drive in 5-10y. So it takes an increasingly smaller amount of time to replicate (excluding the copying time which keeps the machine busy but not you).
- sedatk 6y ago
  The bookkeeping it requires for one, since you don't usually buy all your backup media at once, you acquire them over time, that gets unnecessarily complicated. It's riskier to copy the media periodically as you might increase the chances of data corruption due to the fault in the copying process (faulty RAM, faulty software, not concentrating good enough etc). You periodically introduce possibility of user/hardware/software errors to the longevity of your backups. Also, when others inherit the media they may not have proper equipment or skill to do it themselves as goal of the preservation is to get it 100 years ahead, not keep it always in a usable state per se. For example, I'd like my children to keep my backups until my grandchildren could access them 60 years later.
nichch 6y ago
Is there a source regarding M-DISC being unable to hold up to their lifetime promise? A quick google search brings up amateur-ish blog posts, and even those co sign the medium.
- sedatk 6y ago
  Yes, French national metrology and testing laboratory LNE: https://www.lne.fr/sites/default/files/inline-files/syylex-glass-dvd-accelerated-aging-report.pdf https://www.lne.fr/sites/default/files/inline-files/syylex-g...
  - nichch 6y ago
    Thank you!
  - labawi 6y ago
    > The objective of this study was to investigate the behavior of the GlassMasterDisc of Syylex under extreme climatic conditions (90°C and 85% relative humidity) and to demonstrate the potential of this technology for digital archiving. > The result of this study is that the GlassMasterDisc has a much longer lifetime in accelerated aging than other available DVD±R I wouldn't draw any other conclusions on normal ageing of other tested media. They did an accelerated aging test at 90°C and 85% RH, where most discs didn't last a single test cycle (of 10 days), two discs lasted a single cycle, and only syylex lasted all 4 cycles. Quote on a brand-name DVD > This DVD model had the longest lifetime (i.e. 1500h) at 80°C and 85% RH. At 90°C, it is destroyed after the first cycle of 250 hours. For an idea of what it does to the substrate: > [for measurement] DVDs have to be taken [out] .. To prevent the formation of water droplets in the polycarbonate, it is necessary to "purify" the polycarbonate from the water that was absorbed at high temperature. OTOH, I had CDs (Verbatim, upper middle class), of which about 1-2 of 50 had issues after 20 years storage (in dark, mostly room-temperature conditions).
    - sedatk 6y ago
      Yes, they did an accelerated test, and M-DISC has performed as good as archival grade DVDs. Syylex, which promises the same lifetime as M-DISC, performed significantly better. That clearly shows either M-DISC didn't live up to their promise, or Syylex and archival grade DVDs surpassed expectations. Either is bad news for M-DISC, isn't it? What am I missing?
julienb_sea 6y ago
There really isn't much market for it. You can pay Google or Apple or one of the large cloud providers a very reasonable and decreasing rate for a literal guarantee that your data is accessible. The only risk is the company goes under, which is extraordinarily unlikely for someone like Google / Apple and the shutdown would have advance notice. I realize for the hacker news audience there are multitudes of reasons the solution above doesn't fit your needs, but realize the consumer market is near nonexistent.
- eulers_secret 6y ago
  Wouldn’t the other risk be that after you’re dead and stop paying the data is deleted or inaccessible? Maybe if you could prepay 100 years it’d work?
  - bastawhiz 6y ago
    If nobody is "inheriting" your data (or rather—nobody cares enough to keep your data around), it seems kind of moot to ensure it hangs around. That is, if I put stuff in an S3 bucket and pre-pay for 100 years, if nobody is around to download it in 100 years then why bother? If you wanted to make a sort of digital time capsule and didn't care who discovered it, your next best bet would probably be the Internet Archive or some other archival community. If your data isn't appropriate for archival (i.e., can't be publicly consumed) and isn't interesting enough for your friends/family/etc. to keep around on your behalf, keeping the data is purposeless.
    - sedatk 6y ago
      I absolutely take inheritance into account when having backups lasting for a hundred years, but regardless of how uninteresting my data looks, we don't know if today's boring data would be invaluable for science in the future. We show slippers from 5000 years ago in museums today and they're invaluable. Consider the person who owned it, walking on a national treasure, unaware. Maybe, they didn't even like the slippers, found them boring. :)
giancarlostoro 6y ago
Hell, tell me it will last 20 years and I'm okay with that, if you can guarantee at least 15 years I can then buy replacements every decade and transfer files over...
- sedatk 6y ago
  You're right. That reminded me that I really don't know the state of my cold backups.
phonon 6y ago
Use https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive and add as much redundancy as you like. A lot cheaper to over-provision, than to create the uber-archive medium.
- sedatk 6y ago
  That's a good idea regardless of the medium, but if your flash disk becomes completely unreadable in 15 years, PAR doesn't help you much, does it?
  - phonon 6y ago
    Then don't use flash (or at least not MLC). Use an M-Disc with 3x PAR redundancy. Or RAID 1 enterprise magnetic HD with 3x PAR. Or all of the above :-) Are you trying to archive TBs or MBs? Likely the next most important consideration is a stable place for storage.
TedDoesntTalk 6y ago
> M-DISC didn't hold up Since I have a lot of stuff on M-DISC, you’ve now got me worried. Link? Any other info? I don’t need 1000 years but what is the lifespan?
- sedatk 6y ago
  Here it is, it basically performed as the same as archival grade DVDs: https://www.lne.fr/sites/default/files/inline-files/syylex-glass-dvd-accelerated-aging-report.pdf https://www.lne.fr/sites/default/files/inline-files/syylex-g...
  - jaclaz 6y ago
    Yep, but the reality is that "we don't know", an accelerated, simulated aging test may (or may not) have the same results as "real" aging. 250 hours @ 90 C°/85% humidity, how does that compare to - say - 100,000 hours at a "normal" 10-40 C°/30-50% humidity?
    - sedatk 6y ago
      That's right, but regardless of how "extreme" testing is, archival grade DVDs have performed as good as M-DISC, and Syylex has surpassed the rest by a huge margin. Syylex promised the same lifetime as M-DISC (unlike archival grade DVDs). I think the results are good enough to see M-DISC either doesn't live up to the expectations or archival grade DVDs exceed the expectations. Either way, bad news for M-DISC. If Syylex hadn't bankrupted, it would have been the best option of course.
      - jaclaz 6y ago
        Yes, what I mean is that - set aside the "glass" disc from Syylex - we don't know if both M-DISKs and archival grade DVDs suck or excel, let alone how long they actually last (readable) in the real world. IF my last guess in comparison is correct, 100,000 hours at "normal" temperature/humidity is roughly 11 years, but it may well be that without "cooking" them at 90 C°, the duration is for both 200,000 hours (or whatever) ... Single point anecdata, I had to dig i
noizejoy 6y ago
Brings back memories of a great remix contest in 2006 where the digital copies of analog tape tracks from Peter Gabriel’s “Shock the Monkey” were made available to remixers. To my surprise the pitch of the samples was a little lower (and varying ever so slightly over the duration of the song) than what you’d have expected with a440 tuning. It baffled me, since I expected some of the early digital synths used in the original sessions should habe been rock solid 440 tuning. And that’s how I learned about “tape stretch” where analog audio tape stretches just enough to make the pitch of everything a few cents lower over long period of time. p.s. I ended up applying digital pitch correction, so I could “jam along” with my own synths :-)
- sedatk 6y ago
  A different problem happened to me when digitizing my dad's tapes. Dad bought the player in USA and brought it to Turkey and made the recordings there. When I digitized them in USA, everything sounded in higher pitch. It later turned out that the voltage difference in AC (60hz/50hz) caused the rotation speed to change proportionally, so I slowed it down to 5/6, and it was perfect afterwards.
fy20 6y ago
That's probably not going to happen. Tapes have high longevities and you can buy used LTO drives on eBay for a few hundred bucks, but the biggest issue in 100 years is going to be finding a device to read the tape, and finding an adapter to hook it to the USB-H 12 quantum optical port on your motherboard. A better way would be to use something like that to store it for a decade or two, then copy the data onto whatever the newer version of archive medium is (LTO drives can typically read the last one or two versions as well). Rinse and repeat every decade, and it also lets you test if there has been any bitrot.
dannyw 6y ago
Ask HN: How can I create the most reliable and durable NAS today? I have a lot of very sentimental, very-important files, such as family photos and videos. And I simply like to hoard data. I currently have 8TB of data stored on a Synology DS218+ with RAID1, and monthly data scrubs (verifying checksums). It is backed up remotely to Google Drive (in encrypted form), and I also maintain an infrequently-updated, once-per-quarter disk clone with an external HDD. My biggest concern with my current setup is that the memory is non-ECC. Even though the files are checksummed, I am concerned that specific memory corruption / bit-flips could propagnate into the checksums, and hence result in data corruption. I am considering: * Building my own FreeNAS box using AMD Ryzen (which semi-officially support ECC memory). My concerns here are the semi-official nature of support: how do I know if ECC works, before a rare cosmic bit-flip? * Purchasing a Synology DS1621+. This is AUD$1400 which is a tough pill to swallow, for the equivalent of a first-gen Ryzen quad core and 4GB of memory. Any options or recommendations is appreciated!!
- wojciii 6y ago
  I just built my Truenas box using Ryzen. Let me know if you want a part list for inspiration or any tips.
  - dannyw 6y ago
    I would love your part list! Did you go with ECC memory? Did you have any way of verifying the ECC is working and actually detecting/correcting bitflips? The other thing I am interested in is minimizing idle power consumption. Just to be more environmentally friendly.
    - wojciii 6y ago
      I don't know if you want to build the same kind of system but at least you can get a list of parts that work together. I use my Truenas box for storage using ZFS, VMs and NFS server for different PCs. I bought ECC memory as I understand this is more or less a requirement for ZFS. I found out that FreeBSD which Truenas is based on can give you info about what type of RAM is present. The command is: # dmidecode -t memory According to this I have ECC RAM. :) I did a build with the following parts: Case: SST-CS380 V2 (space for 8 3.5" drives, 2 x ). Mainboard: ASRock X470D4U2-2T Power supply: Seasonic Focus Plus 550W Gold 80 Plus Full Modular Power Supply RAM: Kingston Server Premier, DDR4, 16 GB, KSM26ED8/5 M CPU: AMD Ryzen 5 3600X Wraith Spire CPU Cooler: Arctic Liquid Freezer II NVMe to PCI bridge: ASUS Hyper M.2 x16 Gen 4 (PCIe 4.0/3.0) supports 4X NVMe M.2 devices (2242/2260/2280/22110) up to 256 Gbps for AMD TRX40 / X570 PCIe 4.0 NVMe RAID and Intel® RAID platform - CPU features I bought all this from Amazon.de. The main board is a server main board with 10 GE Ethernet and a console for flashing bios and remoting into the machine - no graphics card needed. This was expensive and you can most likely save a lot using a consumer grade main board. The RAM was taken from the list provided at https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U2-2T#Memory https://www.asrockrack.com/general/productdetail.asp?Model=X... The CPU models supported are here: https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U2-2T#CPU https://www.asrockrack.com/general/productdetail.asp?Model=X... Be careful not to buy an AMD APU - this doesn't support ECC RAM for some insane reason. An APU would have build in graphics and the CPU. I use both SATA drives (long term storage) and SDD drives (for speed). I created two ZFS volumes (I don't remember the proper terms). One for rotating discs (which could sleep most of the time) and one using SDDs for fast storage which doesn't use much power. I have 6 KW solar cells with battery so I don't really care if the box uses a lot of power. During daylight its more or less free when sun is shining. I get next to nothing when selling the electricity that I generate and would like to use as much as possible locally.
deleted 6y ago
[deleted]

by sedatk 6y ago

dudus 6y ago

piql is the company that did the github arctic vault. And they offer it as a service. Maybe not for general public but still . https://www.piql.com/arctic-world-archive/ https://www.piql.com/arctic-world-archive/

sedatk 6y ago
Arctic vault is VERY expensive, about $100 per gigabyte.

elorant 6y ago

The lifetime of the medium is half the equation. The biggest problem imho is the lifetime of the device. Say I give you an old floppy drive for 8" floppies from the seventies. Where would you connect it?

sedatk 6y ago
You're right but not all media are the same. Arctic Vault used an optical film with QR codes. You can theoretically even take its photo and decode it by hand if you want. They even added a Rosetta Stone to the entrance so, even if all the knowledge is lost, one can hypothetically decode the data stored there. For magnetic media, you need some more specialized equipment for sure.
kmeisthax 6y ago
Tapes are uniquely terrible at this. I'd argue it's the Achilles' heel of the medium, even moreso than the actual limitations of linear tape. First off, the drives themselves are going to be expensive brand-new. You're going to be paying thousands of dollars for a drive, and it's probably going to have some weird interconnect that you'll have to spend even more money and waste a PCIe slot on an adapter in order to use the drive. Most common is SAS or Fibre Channel; although there's at least one company selling drives in Thunderbolt enclosures for the Mac market. (Aside from all that, SAS is actually pretty cool for things like hard drive enclosures, since it has SATA compatibility. I have a jury-rigged disk shelf built out of an old HP SAS expander, a slightly-modified CD duplicator case, and some 5 1/4" hard drive bays.) Second, tape formats are constantly moving. LTO and IBM 3592 come out with a new format every 2-3 years and backwards compatibility is limited. Generally speaking you can only write one generation behind on LTO and read two generations back. So, if you want a format that's got drives still being made for it, you'll need to migrate every 5-7 years. Sure, the actual tape is shelf-stable for longer, but you're going to be buying spare drives or jumping on eBay if you want to keep old tapes around that long. (eBay is actually not a bad place to buy used tape drives, but the pricing varies wildly. It's perfect for hobbyists and small-fry IT outfits looking for cheap backup media. Absolutely terrible if you're a large outfit with reliability guarantees and support agreements to maintain.) Third, actually using a tape drive is a nightmare. First off, Windows hasn't shipped tape software since 2003 (I think?), so you'll be in the market for proprietary backup solutions. Second, if you're just writing data directly to the tape, you will shoe-shine for days. Common IT practice is to actually have a second disk array sitting in front of the tapes as a write cache and custom software to copy data to the tapes at full speed once all the slow nonsequential IO is done. Reading from tape doesn't have to worry about this as much, but the fact that you had to use custom software just to archive your files means that you now have proprietary archive formats to deal with. So you can have tapes that rely on both having access to working drives and licenses for proprietary backup utilities. (Of course, if you had decently fast SSDs and a parallel archival utility, you could sustain decent write speeds on tape. I actually wrote this myself as an experiment: https://github.com/kmeisthax/rapidtar https://github.com/kmeisthax/rapidtar and it can saturate the LTO-5 drive I tested this with.)
userbinator 6y ago
Floppies are an interesting case because the protocols and physical specifications are all documented publicly, which means that one could literally build a drive from scratch today --- the trickiest part being the heads, but considering that they are many orders of magnitude lower density than HDDs, it would not be a big obstacle in the future. (I believe 8" floppy drives have the same interface as 5.25" ones --- and there's no shortage of adapters from the retrocomputing community for those, some of them even open-source.) Tape is far more closed, AFAIK most of the common formats are proprietary and the specs are behind NDAs and other walls.
- crististm 6y ago
  I have hobby books from 30 years ago that teach you how to build magnetic heads for cassette tapes. No pictures or anything but with enough patience you definitely can build one today, at home (even then). Mind you, the size of the gap is not that big of the problem if you put your mind at polishing.

cm2187 6y ago

Given the pace at which storage capacity increases, what is the rationale for not copying over your data every 5-10 years onto the next cheapest consumer mass storage of the moment? You get all your data in one place and you don’t have to deal with standards disappearing (I read 2020’s game consoles can’t read CDs anymore, people should rip their CD collection right now).

Max_aaa 6y ago
The time needed to actually do the copying.
- drdavid 6y ago
  It seems like there may be a market there. Every 10 to 15 years, you send in your archived (and new/interim) personal data and get it back on the current top-tech storage medium. That way it's not stored in the cloud and you can keep moving the stored data forward without having to deal with it all yourself.
- cm2187 6y ago
  Not really. You can buy a 18TB hard drive now. Even if your data is humongous and needs several of those, it will likely fit on a single drive in 5-10y. So it takes an increasingly smaller amount of time to replicate (excluding the copying time which keeps the machine busy but not you).
sedatk 6y ago
The bookkeeping it requires for one, since you don't usually buy all your backup media at once, you acquire them over time, that gets unnecessarily complicated. It's riskier to copy the media periodically as you might increase the chances of data corruption due to the fault in the copying process (faulty RAM, faulty software, not concentrating good enough etc). You periodically introduce possibility of user/hardware/software errors to the longevity of your backups. Also, when others inherit the media they may not have proper equipment or skill to do it themselves as goal of the preservation is to get it 100 years ahead, not keep it always in a usable state per se. For example, I'd like my children to keep my backups until my grandchildren could access them 60 years later.

nichch 6y ago

Is there a source regarding M-DISC being unable to hold up to their lifetime promise? A quick google search brings up amateur-ish blog posts, and even those co sign the medium.

sedatk 6y ago
Yes, French national metrology and testing laboratory LNE: https://www.lne.fr/sites/default/files/inline-files/syylex-glass-dvd-accelerated-aging-report.pdf https://www.lne.fr/sites/default/files/inline-files/syylex-g...
- nichch 6y ago
  Thank you!
- labawi 6y ago
  > The objective of this study was to investigate the behavior of the GlassMasterDisc of Syylex under extreme climatic conditions (90°C and 85% relative humidity) and to demonstrate the potential of this technology for digital archiving. > The result of this study is that the GlassMasterDisc has a much longer lifetime in accelerated aging than other available DVD±R I wouldn't draw any other conclusions on normal ageing of other tested media. They did an accelerated aging test at 90°C and 85% RH, where most discs didn't last a single test cycle (of 10 days), two discs lasted a single cycle, and only syylex lasted all 4 cycles. Quote on a brand-name DVD > This DVD model had the longest lifetime (i.e. 1500h) at 80°C and 85% RH. At 90°C, it is destroyed after the first cycle of 250 hours. For an idea of what it does to the substrate: > [for measurement] DVDs have to be taken [out] .. To prevent the formation of water droplets in the polycarbonate, it is necessary to "purify" the polycarbonate from the water that was absorbed at high temperature. OTOH, I had CDs (Verbatim, upper middle class), of which about 1-2 of 50 had issues after 20 years storage (in dark, mostly room-temperature conditions).
  - sedatk 6y ago
    Yes, they did an accelerated test, and M-DISC has performed as good as archival grade DVDs. Syylex, which promises the same lifetime as M-DISC, performed significantly better. That clearly shows either M-DISC didn't live up to their promise, or Syylex and archival grade DVDs surpassed expectations. Either is bad news for M-DISC, isn't it? What am I missing?

julienb_sea 6y ago

There really isn't much market for it. You can pay Google or Apple or one of the large cloud providers a very reasonable and decreasing rate for a literal guarantee that your data is accessible. The only risk is the company goes under, which is extraordinarily unlikely for someone like Google / Apple and the shutdown would have advance notice. I realize for the hacker news audience there are multitudes of reasons the solution above doesn't fit your needs, but realize the consumer market is near nonexistent.

eulers_secret 6y ago
Wouldn’t the other risk be that after you’re dead and stop paying the data is deleted or inaccessible? Maybe if you could prepay 100 years it’d work?
- bastawhiz 6y ago
  If nobody is "inheriting" your data (or rather—nobody cares enough to keep your data around), it seems kind of moot to ensure it hangs around. That is, if I put stuff in an S3 bucket and pre-pay for 100 years, if nobody is around to download it in 100 years then why bother? If you wanted to make a sort of digital time capsule and didn't care who discovered it, your next best bet would probably be the Internet Archive or some other archival community. If your data isn't appropriate for archival (i.e., can't be publicly consumed) and isn't interesting enough for your friends/family/etc. to keep around on your behalf, keeping the data is purposeless.
  - sedatk 6y ago
    I absolutely take inheritance into account when having backups lasting for a hundred years, but regardless of how uninteresting my data looks, we don't know if today's boring data would be invaluable for science in the future. We show slippers from 5000 years ago in museums today and they're invaluable. Consider the person who owned it, walking on a national treasure, unaware. Maybe, they didn't even like the slippers, found them boring. :)

giancarlostoro 6y ago

Hell, tell me it will last 20 years and I'm okay with that, if you can guarantee at least 15 years I can then buy replacements every decade and transfer files over...

sedatk 6y ago
You're right. That reminded me that I really don't know the state of my cold backups.

phonon 6y ago

Use https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive and add as much redundancy as you like. A lot cheaper to over-provision, than to create the uber-archive medium.

sedatk 6y ago
That's a good idea regardless of the medium, but if your flash disk becomes completely unreadable in 15 years, PAR doesn't help you much, does it?
- phonon 6y ago
  Then don't use flash (or at least not MLC). Use an M-Disc with 3x PAR redundancy. Or RAID 1 enterprise magnetic HD with 3x PAR. Or all of the above :-) Are you trying to archive TBs or MBs? Likely the next most important consideration is a stable place for storage.

TedDoesntTalk 6y ago

> M-DISC didn't hold up Since I have a lot of stuff on M-DISC, you’ve now got me worried. Link? Any other info? I don’t need 1000 years but what is the lifespan?

sedatk 6y ago
Here it is, it basically performed as the same as archival grade DVDs: https://www.lne.fr/sites/default/files/inline-files/syylex-glass-dvd-accelerated-aging-report.pdf https://www.lne.fr/sites/default/files/inline-files/syylex-g...
- jaclaz 6y ago
  Yep, but the reality is that "we don't know", an accelerated, simulated aging test may (or may not) have the same results as "real" aging. 250 hours @ 90 C°/85% humidity, how does that compare to - say - 100,000 hours at a "normal" 10-40 C°/30-50% humidity?
  - sedatk 6y ago
    That's right, but regardless of how "extreme" testing is, archival grade DVDs have performed as good as M-DISC, and Syylex has surpassed the rest by a huge margin. Syylex promised the same lifetime as M-DISC (unlike archival grade DVDs). I think the results are good enough to see M-DISC either doesn't live up to the expectations or archival grade DVDs exceed the expectations. Either way, bad news for M-DISC. If Syylex hadn't bankrupted, it would have been the best option of course.
    - jaclaz 6y ago
      Yes, what I mean is that - set aside the "glass" disc from Syylex - we don't know if both M-DISKs and archival grade DVDs suck or excel, let alone how long they actually last (readable) in the real world. IF my last guess in comparison is correct, 100,000 hours at "normal" temperature/humidity is roughly 11 years, but it may well be that without "cooking" them at 90 C°, the duration is for both 200,000 hours (or whatever) ... Single point anecdata, I had to dig i

noizejoy 6y ago

Brings back memories of a great remix contest in 2006 where the digital copies of analog tape tracks from Peter Gabriel’s “Shock the Monkey” were made available to remixers. To my surprise the pitch of the samples was a little lower (and varying ever so slightly over the duration of the song) than what you’d have expected with a440 tuning. It baffled me, since I expected some of the early digital synths used in the original sessions should habe been rock solid 440 tuning. And that’s how I learned about “tape stretch” where analog audio tape stretches just enough to make the pitch of everything a few cents lower over long period of time. p.s. I ended up applying digital pitch correction, so I could “jam along” with my own synths :-)

sedatk 6y ago
A different problem happened to me when digitizing my dad's tapes. Dad bought the player in USA and brought it to Turkey and made the recordings there. When I digitized them in USA, everything sounded in higher pitch. It later turned out that the voltage difference in AC (60hz/50hz) caused the rotation speed to change proportionally, so I slowed it down to 5/6, and it was perfect afterwards.

fy20 6y ago

That's probably not going to happen. Tapes have high longevities and you can buy used LTO drives on eBay for a few hundred bucks, but the biggest issue in 100 years is going to be finding a device to read the tape, and finding an adapter to hook it to the USB-H 12 quantum optical port on your motherboard. A better way would be to use something like that to store it for a decade or two, then copy the data onto whatever the newer version of archive medium is (LTO drives can typically read the last one or two versions as well). Rinse and repeat every decade, and it also lets you test if there has been any bitrot.

dannyw 6y ago

Ask HN: How can I create the most reliable and durable NAS today? I have a lot of very sentimental, very-important files, such as family photos and videos. And I simply like to hoard data. I currently have 8TB of data stored on a Synology DS218+ with RAID1, and monthly data scrubs (verifying checksums). It is backed up remotely to Google Drive (in encrypted form), and I also maintain an infrequently-updated, once-per-quarter disk clone with an external HDD. My biggest concern with my current setup is that the memory is non-ECC. Even though the files are checksummed, I am concerned that specific memory corruption / bit-flips could propagnate into the checksums, and hence result in data corruption. I am considering: * Building my own FreeNAS box using AMD Ryzen (which semi-officially support ECC memory). My concerns here are the semi-official nature of support: how do I know if ECC works, before a rare cosmic bit-flip? * Purchasing a Synology DS1621+. This is AUD$1400 which is a tough pill to swallow, for the equivalent of a first-gen Ryzen quad core and 4GB of memory. Any options or recommendations is appreciated!!

wojciii 6y ago
I just built my Truenas box using Ryzen. Let me know if you want a part list for inspiration or any tips.
- dannyw 6y ago
  I would love your part list! Did you go with ECC memory? Did you have any way of verifying the ECC is working and actually detecting/correcting bitflips? The other thing I am interested in is minimizing idle power consumption. Just to be more environmentally friendly.
  - wojciii 6y ago
    I don't know if you want to build the same kind of system but at least you can get a list of parts that work together. I use my Truenas box for storage using ZFS, VMs and NFS server for different PCs. I bought ECC memory as I understand this is more or less a requirement for ZFS. I found out that FreeBSD which Truenas is based on can give you info about what type of RAM is present. The command is: # dmidecode -t memory According to this I have ECC RAM. :) I did a build with the following parts: Case: SST-CS380 V2 (space for 8 3.5" drives, 2 x ). Mainboard: ASRock X470D4U2-2T Power supply: Seasonic Focus Plus 550W Gold 80 Plus Full Modular Power Supply RAM: Kingston Server Premier, DDR4, 16 GB, KSM26ED8/5 M CPU: AMD Ryzen 5 3600X Wraith Spire CPU Cooler: Arctic Liquid Freezer II NVMe to PCI bridge: ASUS Hyper M.2 x16 Gen 4 (PCIe 4.0/3.0) supports 4X NVMe M.2 devices (2242/2260/2280/22110) up to 256 Gbps for AMD TRX40 / X570 PCIe 4.0 NVMe RAID and Intel® RAID platform - CPU features I bought all this from Amazon.de. The main board is a server main board with 10 GE Ethernet and a console for flashing bios and remoting into the machine - no graphics card needed. This was expensive and you can most likely save a lot using a consumer grade main board. The RAM was taken from the list provided at https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U2-2T#Memory https://www.asrockrack.com/general/productdetail.asp?Model=X... The CPU models supported are here: https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U2-2T#CPU https://www.asrockrack.com/general/productdetail.asp?Model=X... Be careful not to buy an AMD APU - this doesn't support ECC RAM for some insane reason. An APU would have build in graphics and the CPU. I use both SATA drives (long term storage) and SDD drives (for speed). I created two ZFS volumes (I don't remember the proper terms). One for rotating discs (which could sleep most of the time) and one using SDDs for fast storage which doesn't use much power. I have 6 KW solar cells with battery so I don't really care if the box uses a lot of power. During daylight its more or less free when sun is shining. I get next to nothing when selling the electricity that I generate and would like to use as much as possible locally.

deleted 6y ago

[deleted]