3 ms·

Curious as to how frequently bitrot shows up? Has anyone modeled this in terms of bits flipped per gigabyte per year or something like this? I have about half

by tomc1985 4y ago

Curious as to how frequently bitrot shows up? Has anyone modeled this in terms of bits flipped per gigabyte per year or something like this? I have about half a terabyte of media, including some original music I made a long time ago and lost the project files for. Backups are great but it will just back of the corrupted versions

joshstrange 4y ago
I don't have hard and fast numbers but with ~120TB of media I can tell you I'm pretty sure I've had it happen or I've suffered from raid issues, I have no way to tell which. I've seen multiple instances when media that I know played flawlessly suddenly has stopped working. Most this occurred when I was using UnRaid (I still am for a few things) and could have been 100% my fault (most likely reason). I now just use a Synology and it has a data integrity feature that I have enabled. Time will tell if I suffer from "bitrot" again. Either way, bitrot or raid issues, a tool like this is interesting to me. I'd love something that watched my collection, stored the hashes, and altered me when they had changed (I know that's not what this tool does). I'm not sure I'd be willing to give up the space for something like par2 files but even a heads up other than "Oh, this isn't playing" would be nice.
- mmastrac 4y ago
  I tried to make the tool easily pluggable so I wager someone could add hash support pretty easily. I might take a shot at it if I find some time as others seem to be interested in monitoring (but not paying the storage penalty) for bitrot.
jmillikin 4y ago
I'm currently migrating my media library from git-annex to a self-written replacement, and have encountered maybe ~300 instances of bitrot so far (out of ~5 TB). Detected through SHA256 checksum mismatch. Usually shows up as a run of zeros in an otherwise healthy file. So far all of the errors have been in files I could re-rip or recover from public sources, but I'm dreading finding an error in a file that's hard to replace. I'm cataloging those separately as I go along, and planning to do a full mirror in cloud storage.

joshstrange 4y ago

I don't have hard and fast numbers but with ~120TB of media I can tell you I'm pretty sure I've had it happen or I've suffered from raid issues, I have no way to tell which. I've seen multiple instances when media that I know played flawlessly suddenly has stopped working. Most this occurred when I was using UnRaid (I still am for a few things) and could have been 100% my fault (most likely reason). I now just use a Synology and it has a data integrity feature that I have enabled. Time will tell if I suffer from "bitrot" again. Either way, bitrot or raid issues, a tool like this is interesting to me. I'd love something that watched my collection, stored the hashes, and altered me when they had changed (I know that's not what this tool does). I'm not sure I'd be willing to give up the space for something like par2 files but even a heads up other than "Oh, this isn't playing" would be nice.

mmastrac 4y ago
I tried to make the tool easily pluggable so I wager someone could add hash support pretty easily. I might take a shot at it if I find some time as others seem to be interested in monitoring (but not paying the storage penalty) for bitrot.

jmillikin 4y ago

I'm currently migrating my media library from git-annex to a self-written replacement, and have encountered maybe ~300 instances of bitrot so far (out of ~5 TB). Detected through SHA256 checksum mismatch. Usually shows up as a run of zeros in an otherwise healthy file. So far all of the errors have been in files I could re-rip or recover from public sources, but I'm dreading finding an error in a file that's hard to replace. I'm cataloging those separately as I go along, and planning to do a full mirror in cloud storage.