11 ms·

Flash media longevity testing – 6 years later

181 points by 1970-01-01 8mo ago

jmakov 8mo ago
Powered all the time on or powered off?
- alnwlsn 8mo ago
  OP says powered off.
monster_truck 8mo ago
Rewriting the data each year hides the actual issue here. Have had plenty of "nice" flash drives rot to hell in 18+ months of dormancy
- benterris 8mo ago
  Does rewriting data help prevent bit rot? Does it mean powered drives can take advantage of it by periodically rewriting the same data over?
  - vel0city 8mo ago
    It depends on how the flash modules are maintained and their quality, but yes having freshly written data will imply better data consistency on flash media. Flash media relies on recharging, which may or many not happen often enough.
  - monster_truck 8mo ago
    It depends on the type of flash being used and the controller managing it. That he did not even identify the chips should inform you of the extent that these results can be trusted. All I can say for sure is that you should not trust any flash for long term storage, thumb drive or otherwise. In serious enough, high usage, high heat enviornments where everything working without problems or delay is part of what they are paying us to be responsible for, it is standard practice to clone fresh images to nvmes every time, with multiple spares that can be swapped out in minutes when they inevitably fail anyways.
- angry_albatross 8mo ago
  Did you miss that there are 10 different drives and so they have 10 different years of tests where they are testing a completely untouched drive?
  - monster_truck 8mo ago
    I don't think you're reading the results properly.
    - thinkling 8mo ago
      I think they are reading it correctly. Year 1, they touched one drive and left 9 untouched. Year 2, they read one additional drive and left 8 untouched. Etc.
      - monster_truck 8mo ago
        Those drives aren't being read
      - Springtime 8mo ago
        Yes, it's also confirmed on the OP's blog linked in the post.
    - angry_albatross 8mo ago
      What do you think I am I reading incorrectly? The post seems pretty clear: "I filled 10 32-GB Kingston flash drives with pseudo-random data." "The years where I'll first touch a new drive (assuming no errors) are: 1, 2, 3, 4, 6, 8, 11, 15, 20, 27" And from the blog: "Q: You know you powered the drive by reading it, right? A: Yes, that’s why I wrote 10 drives to begin with. We want to see how something works if left unpowered for 1 year, 2 years, etc."
ComputerGuru 8mo ago
Slightly related: I have a tool that writes random (incompressible) data to a disk and lets you verify it back without storing a copy (by using a csprng seed), initially developed for benchmarking SSDs that used to cheat to get better performance numbers but that can also be used for this purpose or to overwrite (“shred”) a disk: https://github.com/mqudsi/hddrand https://github.com/mqudsi/hddrand
- fhdkweig 8mo ago
  I haven't used badblocks https://en.wikipedia.org/wiki/Badblocks https://en.wikipedia.org/wiki/Badblocks in about 10 years, but I was annoyed that this exact feature wasn't available for testing accidental swapping of block locations. badblocks only writes the same data to each block and thus they are all indistinguishable.
  - champtar 8mo ago
    TIL `badblocks -t random` repeats the same random block over and over :(
    - jmb99 8mo ago
      You can however set the block size to something quite large, which means you write the same random pattern spread out over multiple blocks repeatedly. If you pick an "odd" block size (like say, your native block size multiplied by 47), it's highly unlikely your disk under test will be swapping around "groups of 47 blocks." (I usually just do a nice multiple, like 4K16, but if you're super paranoid a weird multiple should be pretty much good enough). You won't get reporting of which exact* blocks on the drive are failing, but these days, that isn't really useful information - if any blocks are failing, warranty or ditch the drive.
  - Superfud 8mo ago
    ddrescue can read from /dev/urandom and write to any device. This can come in handy when you have a device with bad blocks but still would like to "shred" all the still writable parts of.
nullorempty 8mo ago
What's the simplest way to rewrite the data without actually copying the data? Like in place rewrite - you write what you read.
- fhdkweig 8mo ago
  I've seen "dd if=/dev/removable of=/dev/removable" suggested. I don't know if it actually works or if the OS optimizes it to a no-op.
  - piyh 8mo ago
    the risk of catastrophic data loss from misuse of `dd` makes my hackles rise just looking at this. I will never forget when I mixed up `if` and `of` during a routine backup. `cat /dev/sda > /mnt/myDisk2` is so much safer, explicit, and in unix norms. It's also faster because you don't have to tune block size parameters. Plus you can also do `pv /dev/sda > /mnt/myDisk2` to get transfer speed details. Friends don't let friends use `dd` where `cat` can do the same job.
    - jmb99 8mo ago
      I stopped getting scared of `if` and `of` about a decade ago when I started explicitly saying (in my head) "input file" and "output file" rather than "if" and "of." You still can mess up the order, but imo no more easily than you can swap `cat in > out` for `cat out > in`. > Friends don't let friends use `dd` where `cat` can do the same job. Technically yes... but I like being able to explicitly set block sizes and force sync writes.
      - ogurechny 8mo ago
        I think you both are arguing about how to fight a bear with your bare hands. To win in that, you simply need to not fight with a bear. Let's say someone made an expansion board with a cool feature: there are 5 documented I/O addresses, but accessing any other address fries the stored firmware. What would you do? No, not leaving a lot of comments in code in CAPS LOCK. No, not printing the correct hexadecimal values in red to put the message on the wall. You make a driver that only allows access to the correct addresses, and configure the rest of the system to make sure that it can only work through that driver. Let's say there's a loading bay at the chemical plant with multiple flanges. If strong acid from the tanker is pumped into the main acid tank, everything is fine. If it is pumped into any other tank, the whole plant may explode and burn. What should be done? No, not promising that drivers will be fired, then shot by the firing squad if they make a mistake. Each connection is independently locked, and the driver only gets a single matching key. You have wonderful programmable devices that allow you to solve non-standard problems with non-standard tools. What should be done is making a wrapper for dd that just does not allow you to do anything you don't want to happen. Even the most basic script with checks and confirmation is enough.
01HNNWZ0MV43FF 8mo ago
That's good. I want to keep some institutional knowledge and photos in "cold storage" and cloud subscriptions with a credit card and password are completely inviable. I'll probably get a spinner and a flash drive and hope one of them survives the years.
- fhdkweig 8mo ago
  If privacy is your primary problem with cloud storage, I would suggest veracrypt containers. And if you aren't storing too much data, I would also suggest DVD/BluRay optical media with DVDisaster and PAR2 archives. I keep a DVD spindle in a safe deposit box that gets updated each year.
- rpcope1 8mo ago
  Unless the data is huge, you're probably going to be better off with M-Disc Blurays or DVDs, as they're explicitly designed for what you're trying to do.
ralferoo 8mo ago
I like the fact he's not just verifying all of them each year. AFAICR, reading the flash causes the row to be rewritten with the values just read. I remember years ago working on the Wii, and there was a restriction on how often you could read the flash to avoid premature wearing. Not sure if that was just the specific type of storage, as googling suggests that NAND is subject to this and NOR isn't. I think pretty much all USB drives now use NOR flash, so maybe this isn't actually an issue any more.
- wmf 8mo ago
  reading the flash causes the row to be rewritten with the values just read DRAM works that way but flash doesn't. Read disturb is a different issue. pretty much all USB drives now use NOR flash Nope, NOR flash is much more expensive than NAND so NOR is only used for firmware and everything else is NAND.
  - cyberax 8mo ago
    But the firmware might have the logic to rewrite the block when it reads it in case it hasn't been written in a while.
    - wmf 8mo ago
      SSDs should definitely rewrite static data if it has too many ECC errors. Unfortunately we don't know much about what's going on in SSDs. Some could have much better data integrity than others.
- zozbot234 8mo ago
  > reading the flash causes the row to be rewritten This only happens very rarely, though more frequently as NAND flash goes QLC and beyond. Besides, other experiments have shown that data remanence is way more of an issue with drives that are almost completely worn out (way beyond their specified TBW) and about to croak. Even then you only get rare bitrot that can be checked for and compensated quite cheaply in most cases. If you take fresh media, write it just once or a few times at most, use substantial overprovisioning to keep the drive in its fast pseudo-SLC mode, and reread the media periodically, NAND can be a good enough storage system for most casual needs.
digdugdirk 8mo ago
What is the best consumer friendly long-term storage medium? Are we still better off with high capacity dvd/Blu ray discs?
- 1970-01-01 8mo ago
  I've been a big fan of M-Disc BD-R.
- BoredPositron 8mo ago
  What's long-term? I have some dvd-rs that push 20-25 years and despite the plastic getting brittle they still work. I also have some ide drives that still work without problems after 40 years. I would rather aim for 20 years and upgrade the storage device if I still need to retain the data.
  - vel0city 8mo ago
    That's a thought I hadn't had. The plastic of the disk getting so brittle it shatters in the drive due to age. I wonder what's the embrittlement profile of polycarbonate stored in reasonable condition.
    - mrob 8mo ago
      I've personally never noticed brittleness in old optical discs (unlike the polystyrene jewel cases, which often turn brittle). I don't think shattering is likely, but if it's a concern some optical drives allow limiting the maximum spin speed. If the drive supports it you can temporarily set it with the -x option of the "eject" command from util-linux.
    - mapontosevenths 8mo ago
      Brittleness is not a concern. "Disk rot" is. The dyes used to make writable DVD's were organic (AZO usually), and break down starting at around the 17 year mark (some earlier, if they were poorly made). They have some measure of redundancy built-in, so you may not notice right away. The discs begin to look a bit "cloudy" at first. Eventually they become unreadable. Go with inorganic blu ray media if you want longevity. Most HTL blu rays made currently will last around 100 years if properly stored. If you need longer there are M-Disc's, but they are expensive and rumor has it that ALL verbatim 100Gb blu rays are essentially M-Discs with different labels these days. For all practical purposes any Blu ray larger than 25Gb is probably inorganic HTL, but if you worry a lot you can buy more expensive "archival grade" discs from Japan as well that have been vettted and tested.
somat 8mo ago
On a related subject, physical media, like a song album. I started by wondering if there were ever any solid state distribution options (One Company tried SD cards) and then started digging into the underlying storage tech to see if I could find a write once long term stable process. First the elephant in the room. Why solid state? because the drives to read the media are often the weak link. When the drives are no longer being manufactured how hard is it to make one? reading solid state drives is a relatively low precision electrical process compared to the high precision mechanical process needed for most media. First on the chopping block was bulk storage. It tends to be delicate and hard to read and short lifespans. But if I limited myself to small storage there are some interesting options. fusible proms were promising but top out at a few megabytes. Mask roms? does anyone offer a mask rom service anymore? Put a mask rom into a sd card... no, sd cards are too physically small. For a song album we want something bigger to put album art on. A thing the size of the original gameboy cartridge with a usb interface and a mask rom? My conclusion, for that specific goal, indefinite future storage of a song album. Vinyl records. low tech enough that it is easy to make a player for them.
jflshci 8mo ago
[dead]
deltoidmaximus 8mo ago
Interesting related thread that includes SSD firmware engineer fairfeather discussing refresh mechanism implementation: https://news.ycombinator.com/item?id=46038099 https://news.ycombinator.com/item?id=46038099
rambambram 8mo ago
I could google it, but I would rather ask HN: what are the best pens (or pen(cil)/paper-combination) for keeping written text as long as possible? I had some Stabilo pen which was very nice ergonomically, but the blue ink faded within a couple of years (laying on my window sil in the sun, but still). My guess is: regular graphite pencil on porous paper is best. Any ideas about further things I have to take into account?
- sandworm101 8mo ago
  Black ink on white paper, stored in a cool dark place, will last many decades. If may fade but will remain readable. Want centuries? Use skin parchment. Millenia? An engraving pen on glass. Going for longer? Take a grinder to a block of granite, but the real problem there is the lack of geologically-stable storage on this planet.
  - nine_k 8mo ago
    Granite is heavy and brittle. Instead, take a plate made of platinum or iridium, and engrave information on it. It offers excellent mechanical, chemical, and thermal durability. It can sink in volcanic lava and then hammered back out from the resulting rock, intact. (Expensive though.)
    - GJim 8mo ago
      > Instead, take a plate made of platinum or iridium No no no no no! Archival data should never be made on intrinsically valuable material; doing so makes it subject to theft or re-use for something "better". Example: There is a reason why more marble statues survive from antiquity than bronze statues.... Bronze has an intrinsic value (theft) and future artists would also melt down existing bronze statues to make something "better".
  - rambambram 8mo ago
    A couple of millennia might suffice. ;) Thanks for the input. The engraving pen on glass is a good one. Any experience with it?
jofla_net 8mo ago
Buyer be warned i think this is extremely brand-dependent. While i've had generally solid experience with sandisk for almost 20 years and had a few old drives (which i hear are slc-based so its not surprising) hold files for over 5 years no issue, i recently almost lost over 4 years of photos. I had purchased some lexar drives from costco since they were dual interface (usb A / usb C) about 2 years ago, and it was usefull to just get some pictures off my phone. I usually don't rely on such a setup for long term but as with all things I was delayed tending to it. I figured there were 2 per box so i just copied them twice, and diffed them several times to make sure they were exact copies. After 24 months, one of the drives had a %95 loss, almost every picture was lost cut-off bottom half or so. The other drive surprisingly seemed fine, though it had been plugged in every 6-9 months I recall, as I wanted to browse it a few times, it seems that this action saved the volume. Upon further inspection the good drive still lost 10 pictures in about 5 thousand, so it wasn't perfect. Lexar. https://www.ebay.com/itm/176810492981?chn=ps&_trkparms=ispr%3D1&amdata=enc%3A1tG4G9BXyRciKhEILAahbAQ14&norover=1&mkevt=1&mkrid=711-173151-913341-5&mkcid=2&mkscid=101&itemid=176810492981&targetid=&device=c&mktype=pla&googleloc=9191125&poi=&campaignid=23517806137&mkgroupid=&rlsatarget=&abcId=10414390&merchantid=5658200746&gad_source=1&gad_campaignid=23517810697&gclid=EAIaIQobChMIkvDgnceTkwMVXNDCBB2QrwMhEAQYASABEgLEgPD_BwE https://www.ebay.com/itm/176810492981?chn=ps&_trkparms=ispr%...
- pixl97 8mo ago
  Really who knows if you're getting a legit drive any longer... https://www.tomshardware.com/pc-components/ssds/fake-samsung-990-pro-passes-basic-checks-but-runs-slower-than-a-usb-2-0-drive-counterfeit-ssds-proliferate-as-nand-shortage-creates-the-perfect-storm-for-bogus-deals https://www.tomshardware.com/pc-components/ssds/fake-samsung...
  - Marsymars 8mo ago
    Well I just ordered a drive from sandisk.com, hopefully they’re not mailing out counterfeits.
- Giefo6ah 8mo ago
  > After 24 months, one of the drives had a %95 loss, almost every picture was lost cut-off bottom half or so. If these are JPEGs with a grey or green lower half, it's likely only a few 16x16 macroblocks are corrupted and you can recover the rest. This cannot be done programmatically because you have to guess what the average colour of the block was, but it can be worth it for precious pictures.
havaloc 8mo ago
I bought about 20 flash drives in 2019 at work to parcel out when needed, once or twice a quarter for users. I needed one last week, and had to throw most of them away, they had all died from presumably dormancy, even new in the package.
foxglacier 8mo ago
Beware that flash data lifetime is sensitive to temperature in the normal range people store stuff at. Store them in the roof space of your house that can exceed 40C each day and they might not last one year. Definitely not a medium to passively store anything long term without power! Use Hard drives or Blu-ray instead.
- jmb99 8mo ago
  The linked post currently has demonstrated that 6 years in some reasonable-ish condition is perfectly fine. > they might not last one year. > Definitely not a medium to passively store anything long term without power! Do you have any evidence to back up this claim? I'm much more interested in data than fear mongering.
  - foxglacier 8mo ago
    He stored them on a shelf which is probably 25C max. and that has an order of magnitude longer life than at 40C. [1] [1] https://www.ni.com/en/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html#section--1379857595 https://www.ni.com/en/support/documentation/supplemental/12/...
shiroiuma 8mo ago
We really need that holographic data storage that they've been working on for decades now.
ogurechny 8mo ago
I've been using an old Symbian phone with the same Class Not-That-Good SD card bought back then. In the early 2010s, I copied a lot of MP3 files and ebooks there, and used the camera to take photos occasionally. Then it was no longer used for music and other needs, and the files just rested there. After about 10 years, I've decided to play some music on the phone, and these tracks had a lot of skips and rattle. Images copied from the card showed a lot of damage, too. So when someone on the internet posts how SD cards are a cheap and compact long term storage, I am not impressed. You probably need to refresh all previously stored data with each monthly backup. It should be mentioned that the phone board often gets warm during operation or battery charging, and the temperature is stated an important harmful factor in a different comment. So if you have some old files on an old device, and assume that they are still there because their records in the file system still look fine, you might be surprised.
quilombodigital 8mo ago
I was trying to roast this guy because I figured a burn-in test should accomplish the same thing he’s doing. But no... I had to ask the great oracle, GPT (General Purpose Truth), and it told me the guy is actually doing something worthwhile. How are trolls supposed to survive in this inhospitable era? Apparently, multi-year storage tests are still valuable for validating whether those estimates match reality. Who knew...
Cyphase 8mo ago
I saw "reddit.com" and my first thought was r/DataHoarder.

181 points by 1970-01-01 8mo ago

jmakov 8mo ago

Powered all the time on or powered off?

alnwlsn 8mo ago
OP says powered off.

monster_truck 8mo ago

Rewriting the data each year hides the actual issue here. Have had plenty of "nice" flash drives rot to hell in 18+ months of dormancy

benterris 8mo ago
Does rewriting data help prevent bit rot? Does it mean powered drives can take advantage of it by periodically rewriting the same data over?
- vel0city 8mo ago
  It depends on how the flash modules are maintained and their quality, but yes having freshly written data will imply better data consistency on flash media. Flash media relies on recharging, which may or many not happen often enough.
- monster_truck 8mo ago
  It depends on the type of flash being used and the controller managing it. That he did not even identify the chips should inform you of the extent that these results can be trusted. All I can say for sure is that you should not trust any flash for long term storage, thumb drive or otherwise. In serious enough, high usage, high heat enviornments where everything working without problems or delay is part of what they are paying us to be responsible for, it is standard practice to clone fresh images to nvmes every time, with multiple spares that can be swapped out in minutes when they inevitably fail anyways.
angry_albatross 8mo ago
Did you miss that there are 10 different drives and so they have 10 different years of tests where they are testing a completely untouched drive?
- monster_truck 8mo ago
  I don't think you're reading the results properly.
  - thinkling 8mo ago
    I think they are reading it correctly. Year 1, they touched one drive and left 9 untouched. Year 2, they read one additional drive and left 8 untouched. Etc.
    - monster_truck 8mo ago
      Those drives aren't being read
    - Springtime 8mo ago
      Yes, it's also confirmed on the OP's blog linked in the post.
  - angry_albatross 8mo ago
    What do you think I am I reading incorrectly? The post seems pretty clear: "I filled 10 32-GB Kingston flash drives with pseudo-random data." "The years where I'll first touch a new drive (assuming no errors) are: 1, 2, 3, 4, 6, 8, 11, 15, 20, 27" And from the blog: "Q: You know you powered the drive by reading it, right? A: Yes, that’s why I wrote 10 drives to begin with. We want to see how something works if left unpowered for 1 year, 2 years, etc."

ComputerGuru 8mo ago

Slightly related: I have a tool that writes random (incompressible) data to a disk and lets you verify it back without storing a copy (by using a csprng seed), initially developed for benchmarking SSDs that used to cheat to get better performance numbers but that can also be used for this purpose or to overwrite (“shred”) a disk: https://github.com/mqudsi/hddrand https://github.com/mqudsi/hddrand

fhdkweig 8mo ago
I haven't used badblocks https://en.wikipedia.org/wiki/Badblocks https://en.wikipedia.org/wiki/Badblocks in about 10 years, but I was annoyed that this exact feature wasn't available for testing accidental swapping of block locations. badblocks only writes the same data to each block and thus they are all indistinguishable.
- champtar 8mo ago
  TIL `badblocks -t random` repeats the same random block over and over :(
  - jmb99 8mo ago
    You can however set the block size to something quite large, which means you write the same random pattern spread out over multiple blocks repeatedly. If you pick an "odd" block size (like say, your native block size multiplied by 47), it's highly unlikely your disk under test will be swapping around "groups of 47 blocks." (I usually just do a nice multiple, like 4K16, but if you're super paranoid a weird multiple should be pretty much good enough). You won't get reporting of which exact* blocks on the drive are failing, but these days, that isn't really useful information - if any blocks are failing, warranty or ditch the drive.
- Superfud 8mo ago
  ddrescue can read from /dev/urandom and write to any device. This can come in handy when you have a device with bad blocks but still would like to "shred" all the still writable parts of.

nullorempty 8mo ago

What's the simplest way to rewrite the data without actually copying the data? Like in place rewrite - you write what you read.

fhdkweig 8mo ago
I've seen "dd if=/dev/removable of=/dev/removable" suggested. I don't know if it actually works or if the OS optimizes it to a no-op.
- piyh 8mo ago
  the risk of catastrophic data loss from misuse of `dd` makes my hackles rise just looking at this. I will never forget when I mixed up `if` and `of` during a routine backup. `cat /dev/sda > /mnt/myDisk2` is so much safer, explicit, and in unix norms. It's also faster because you don't have to tune block size parameters. Plus you can also do `pv /dev/sda > /mnt/myDisk2` to get transfer speed details. Friends don't let friends use `dd` where `cat` can do the same job.
  - jmb99 8mo ago
    I stopped getting scared of `if` and `of` about a decade ago when I started explicitly saying (in my head) "input file" and "output file" rather than "if" and "of." You still can mess up the order, but imo no more easily than you can swap `cat in > out` for `cat out > in`. > Friends don't let friends use `dd` where `cat` can do the same job. Technically yes... but I like being able to explicitly set block sizes and force sync writes.
    - ogurechny 8mo ago
      I think you both are arguing about how to fight a bear with your bare hands. To win in that, you simply need to not fight with a bear. Let's say someone made an expansion board with a cool feature: there are 5 documented I/O addresses, but accessing any other address fries the stored firmware. What would you do? No, not leaving a lot of comments in code in CAPS LOCK. No, not printing the correct hexadecimal values in red to put the message on the wall. You make a driver that only allows access to the correct addresses, and configure the rest of the system to make sure that it can only work through that driver. Let's say there's a loading bay at the chemical plant with multiple flanges. If strong acid from the tanker is pumped into the main acid tank, everything is fine. If it is pumped into any other tank, the whole plant may explode and burn. What should be done? No, not promising that drivers will be fired, then shot by the firing squad if they make a mistake. Each connection is independently locked, and the driver only gets a single matching key. You have wonderful programmable devices that allow you to solve non-standard problems with non-standard tools. What should be done is making a wrapper for dd that just does not allow you to do anything you don't want to happen. Even the most basic script with checks and confirmation is enough.

01HNNWZ0MV43FF 8mo ago

That's good. I want to keep some institutional knowledge and photos in "cold storage" and cloud subscriptions with a credit card and password are completely inviable. I'll probably get a spinner and a flash drive and hope one of them survives the years.

fhdkweig 8mo ago
If privacy is your primary problem with cloud storage, I would suggest veracrypt containers. And if you aren't storing too much data, I would also suggest DVD/BluRay optical media with DVDisaster and PAR2 archives. I keep a DVD spindle in a safe deposit box that gets updated each year.
rpcope1 8mo ago
Unless the data is huge, you're probably going to be better off with M-Disc Blurays or DVDs, as they're explicitly designed for what you're trying to do.

ralferoo 8mo ago

I like the fact he's not just verifying all of them each year. AFAICR, reading the flash causes the row to be rewritten with the values just read. I remember years ago working on the Wii, and there was a restriction on how often you could read the flash to avoid premature wearing. Not sure if that was just the specific type of storage, as googling suggests that NAND is subject to this and NOR isn't. I think pretty much all USB drives now use NOR flash, so maybe this isn't actually an issue any more.

wmf 8mo ago
reading the flash causes the row to be rewritten with the values just read DRAM works that way but flash doesn't. Read disturb is a different issue. pretty much all USB drives now use NOR flash Nope, NOR flash is much more expensive than NAND so NOR is only used for firmware and everything else is NAND.
- cyberax 8mo ago
  But the firmware might have the logic to rewrite the block when it reads it in case it hasn't been written in a while.
  - wmf 8mo ago
    SSDs should definitely rewrite static data if it has too many ECC errors. Unfortunately we don't know much about what's going on in SSDs. Some could have much better data integrity than others.
zozbot234 8mo ago
> reading the flash causes the row to be rewritten This only happens very rarely, though more frequently as NAND flash goes QLC and beyond. Besides, other experiments have shown that data remanence is way more of an issue with drives that are almost completely worn out (way beyond their specified TBW) and about to croak. Even then you only get rare bitrot that can be checked for and compensated quite cheaply in most cases. If you take fresh media, write it just once or a few times at most, use substantial overprovisioning to keep the drive in its fast pseudo-SLC mode, and reread the media periodically, NAND can be a good enough storage system for most casual needs.

digdugdirk 8mo ago

What is the best consumer friendly long-term storage medium? Are we still better off with high capacity dvd/Blu ray discs?

1970-01-01 8mo ago
I've been a big fan of M-Disc BD-R.
BoredPositron 8mo ago
What's long-term? I have some dvd-rs that push 20-25 years and despite the plastic getting brittle they still work. I also have some ide drives that still work without problems after 40 years. I would rather aim for 20 years and upgrade the storage device if I still need to retain the data.
- vel0city 8mo ago
  That's a thought I hadn't had. The plastic of the disk getting so brittle it shatters in the drive due to age. I wonder what's the embrittlement profile of polycarbonate stored in reasonable condition.
  - mrob 8mo ago
    I've personally never noticed brittleness in old optical discs (unlike the polystyrene jewel cases, which often turn brittle). I don't think shattering is likely, but if it's a concern some optical drives allow limiting the maximum spin speed. If the drive supports it you can temporarily set it with the -x option of the "eject" command from util-linux.
  - mapontosevenths 8mo ago
    Brittleness is not a concern. "Disk rot" is. The dyes used to make writable DVD's were organic (AZO usually), and break down starting at around the 17 year mark (some earlier, if they were poorly made). They have some measure of redundancy built-in, so you may not notice right away. The discs begin to look a bit "cloudy" at first. Eventually they become unreadable. Go with inorganic blu ray media if you want longevity. Most HTL blu rays made currently will last around 100 years if properly stored. If you need longer there are M-Disc's, but they are expensive and rumor has it that ALL verbatim 100Gb blu rays are essentially M-Discs with different labels these days. For all practical purposes any Blu ray larger than 25Gb is probably inorganic HTL, but if you worry a lot you can buy more expensive "archival grade" discs from Japan as well that have been vettted and tested.

somat 8mo ago

On a related subject, physical media, like a song album. I started by wondering if there were ever any solid state distribution options (One Company tried SD cards) and then started digging into the underlying storage tech to see if I could find a write once long term stable process. First the elephant in the room. Why solid state? because the drives to read the media are often the weak link. When the drives are no longer being manufactured how hard is it to make one? reading solid state drives is a relatively low precision electrical process compared to the high precision mechanical process needed for most media. First on the chopping block was bulk storage. It tends to be delicate and hard to read and short lifespans. But if I limited myself to small storage there are some interesting options. fusible proms were promising but top out at a few megabytes. Mask roms? does anyone offer a mask rom service anymore? Put a mask rom into a sd card... no, sd cards are too physically small. For a song album we want something bigger to put album art on. A thing the size of the original gameboy cartridge with a usb interface and a mask rom? My conclusion, for that specific goal, indefinite future storage of a song album. Vinyl records. low tech enough that it is easy to make a player for them.

jflshci 8mo ago

[dead]

deltoidmaximus 8mo ago

Interesting related thread that includes SSD firmware engineer fairfeather discussing refresh mechanism implementation: https://news.ycombinator.com/item?id=46038099 https://news.ycombinator.com/item?id=46038099

rambambram 8mo ago

I could google it, but I would rather ask HN: what are the best pens (or pen(cil)/paper-combination) for keeping written text as long as possible? I had some Stabilo pen which was very nice ergonomically, but the blue ink faded within a couple of years (laying on my window sil in the sun, but still). My guess is: regular graphite pencil on porous paper is best. Any ideas about further things I have to take into account?

sandworm101 8mo ago
Black ink on white paper, stored in a cool dark place, will last many decades. If may fade but will remain readable. Want centuries? Use skin parchment. Millenia? An engraving pen on glass. Going for longer? Take a grinder to a block of granite, but the real problem there is the lack of geologically-stable storage on this planet.
- nine_k 8mo ago
  Granite is heavy and brittle. Instead, take a plate made of platinum or iridium, and engrave information on it. It offers excellent mechanical, chemical, and thermal durability. It can sink in volcanic lava and then hammered back out from the resulting rock, intact. (Expensive though.)
  - GJim 8mo ago
    > Instead, take a plate made of platinum or iridium No no no no no! Archival data should never be made on intrinsically valuable material; doing so makes it subject to theft or re-use for something "better". Example: There is a reason why more marble statues survive from antiquity than bronze statues.... Bronze has an intrinsic value (theft) and future artists would also melt down existing bronze statues to make something "better".
- rambambram 8mo ago
  A couple of millennia might suffice. ;) Thanks for the input. The engraving pen on glass is a good one. Any experience with it?

jofla_net 8mo ago

Buyer be warned i think this is extremely brand-dependent. While i've had generally solid experience with sandisk for almost 20 years and had a few old drives (which i hear are slc-based so its not surprising) hold files for over 5 years no issue, i recently almost lost over 4 years of photos. I had purchased some lexar drives from costco since they were dual interface (usb A / usb C) about 2 years ago, and it was usefull to just get some pictures off my phone. I usually don't rely on such a setup for long term but as with all things I was delayed tending to it. I figured there were 2 per box so i just copied them twice, and diffed them several times to make sure they were exact copies. After 24 months, one of the drives had a %95 loss, almost every picture was lost cut-off bottom half or so. The other drive surprisingly seemed fine, though it had been plugged in every 6-9 months I recall, as I wanted to browse it a few times, it seems that this action saved the volume. Upon further inspection the good drive still lost 10 pictures in about 5 thousand, so it wasn't perfect. Lexar. https://www.ebay.com/itm/176810492981?chn=ps&_trkparms=ispr%3D1&amdata=enc%3A1tG4G9BXyRciKhEILAahbAQ14&norover=1&mkevt=1&mkrid=711-173151-913341-5&mkcid=2&mkscid=101&itemid=176810492981&targetid=&device=c&mktype=pla&googleloc=9191125&poi=&campaignid=23517806137&mkgroupid=&rlsatarget=&abcId=10414390&merchantid=5658200746&gad_source=1&gad_campaignid=23517810697&gclid=EAIaIQobChMIkvDgnceTkwMVXNDCBB2QrwMhEAQYASABEgLEgPD_BwE https://www.ebay.com/itm/176810492981?chn=ps&_trkparms=ispr%...

pixl97 8mo ago
Really who knows if you're getting a legit drive any longer... https://www.tomshardware.com/pc-components/ssds/fake-samsung-990-pro-passes-basic-checks-but-runs-slower-than-a-usb-2-0-drive-counterfeit-ssds-proliferate-as-nand-shortage-creates-the-perfect-storm-for-bogus-deals https://www.tomshardware.com/pc-components/ssds/fake-samsung...
- Marsymars 8mo ago
  Well I just ordered a drive from sandisk.com, hopefully they’re not mailing out counterfeits.
Giefo6ah 8mo ago
> After 24 months, one of the drives had a %95 loss, almost every picture was lost cut-off bottom half or so. If these are JPEGs with a grey or green lower half, it's likely only a few 16x16 macroblocks are corrupted and you can recover the rest. This cannot be done programmatically because you have to guess what the average colour of the block was, but it can be worth it for precious pictures.

havaloc 8mo ago

I bought about 20 flash drives in 2019 at work to parcel out when needed, once or twice a quarter for users. I needed one last week, and had to throw most of them away, they had all died from presumably dormancy, even new in the package.

foxglacier 8mo ago

Beware that flash data lifetime is sensitive to temperature in the normal range people store stuff at. Store them in the roof space of your house that can exceed 40C each day and they might not last one year. Definitely not a medium to passively store anything long term without power! Use Hard drives or Blu-ray instead.

jmb99 8mo ago
The linked post currently has demonstrated that 6 years in some reasonable-ish condition is perfectly fine. > they might not last one year. > Definitely not a medium to passively store anything long term without power! Do you have any evidence to back up this claim? I'm much more interested in data than fear mongering.
- foxglacier 8mo ago
  He stored them on a shelf which is probably 25C max. and that has an order of magnitude longer life than at 40C. [1] [1] https://www.ni.com/en/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html#section--1379857595 https://www.ni.com/en/support/documentation/supplemental/12/...

shiroiuma 8mo ago

We really need that holographic data storage that they've been working on for decades now.

ogurechny 8mo ago

I've been using an old Symbian phone with the same Class Not-That-Good SD card bought back then. In the early 2010s, I copied a lot of MP3 files and ebooks there, and used the camera to take photos occasionally. Then it was no longer used for music and other needs, and the files just rested there. After about 10 years, I've decided to play some music on the phone, and these tracks had a lot of skips and rattle. Images copied from the card showed a lot of damage, too. So when someone on the internet posts how SD cards are a cheap and compact long term storage, I am not impressed. You probably need to refresh all previously stored data with each monthly backup. It should be mentioned that the phone board often gets warm during operation or battery charging, and the temperature is stated an important harmful factor in a different comment. So if you have some old files on an old device, and assume that they are still there because their records in the file system still look fine, you might be surprised.

quilombodigital 8mo ago

I was trying to roast this guy because I figured a burn-in test should accomplish the same thing he’s doing. But no... I had to ask the great oracle, GPT (General Purpose Truth), and it told me the guy is actually doing something worthwhile. How are trolls supposed to survive in this inhospitable era? Apparently, multi-year storage tests are still valuable for validating whether those estimates match reality. Who knew...

Cyphase 8mo ago

I saw "reddit.com" and my first thought was r/DataHoarder.