4 ms·

Show HN: Automedia – a tool for managing bitrot and formats in media libraries

71 points by mmastrac 5y ago

yboris 4y ago
Awesome tool! Thank you :) May be a nice complement to my Video Hub App - browse, search, preview, and organize videos on Win, Mac, and Linux. https://videohubapp.com/ https://videohubapp.com/ & https://github.com/whyboris/Video-Hub-App https://github.com/whyboris/Video-Hub-App
mmastrac 5y ago
Hey all, a tiny project I wrote for myself over the last couple of months. I've been ripping my old physical audio and video media to a server running locally at home and have been worried about bitrot. I built this tool to check a large media library for correctness (by test-decoding it with FFMPEG), and to prevent future bitrot by automatically adding per-directory PAR2 files. The tool has another feature to transcode your entire library to a given format - I'm using this to keep a USB key in my car filled with AAC 64k files for listening when Spotify is out of range or I'm too lazy to connect via Bluetooth. It's designed to be somewhat easy to extend with media operations that consume a directory at a time. For example, you could add or check music metadata using a third-party service like MusicBrainz.
- chrisweekly 5y ago
  Great idea! Thanks for building and sharing it; looking fwd to looking closer at it after Mother's Day!
- sirius87 5y ago
  This is pretty good! Detecting bitrot is fine, can also be done by storing a MD5/SHA256 of the file content. `md5sum <file>` barfs if its unable to read the file from disk. Challenge is rebuilding the broken file. I was worried about bitrot on my old external drives too. I eventually went with btrfs[1] filesystem running RAID1 across 2 disks. [1] https://wiki.archlinux.org/title/btrfs https://wiki.archlinux.org/title/btrfs
  - mmastrac 5y ago
    Adding/verifying hashes on media might not be a bad feature to add for those who don't want to burn the extra space required for PAR2. I run w/two extra parity disks on unRAID, but I'm too paranoid to leave my data at rest without an extra file-level of parity.
- Wowfunhappy 5y ago
  Did you consider storing your media in a ZFS pool?
  - pronoiac 5y ago
    Checksum and PAR2 files are also useful alongside backup copies.
- aspenmayer 5y ago
  What’s a good method for making the original lossless rips, making parity files, and keeping them organized for use with Automedia? Is there something like Calibre for video/audio archives?
  - mmastrac 5y ago
    MusicBrainz Picard will do metadata tagging and some basic directory organization. It's what I use to create my music directories. With a great metadata manager like that, any software that rips to FLAC or Apple Lossless should be sufficient. Exact Audio Copy is one of the best ones on the Windows side.
    - aspenmayer 4y ago
      I appreciate these recommendations for audio content; I’m mainly looking for resources for archiving video content.
saamerm 5y ago
Great work! Thanks for sharing and making it Open Source!
daedalus2027 5y ago
hi, I don't understand why they use ffmpeg for bitrot checking and not a simple hash checksum or a merkle tree.
- arendtio 5y ago
  From my very limited understanding of the software I understand, that it doesn't compare an earlier version with a later version as you would with hash sums, but rather utilizes ffmpeg to check if something odd/irregular shows up. In addition, it can use PAR2 files which can be used to restore the original version which would not easily be possible with hash sums. But as I said, I just read the description, nothing more.
  - mmastrac 5y ago
    This is correct. I have a large media library and I wasn't sure how rotted it was. Test decoding won't catch every error, but it identified albums where there has clearly been a bitflip in a song (at which point I can try to find a lossless copy of that song to replace it). Once I'm happy with the state of the library, I use automedia to add on par2 files to avoid this going forward. Verification is for libraries that have no existing hashes or parity, essentially.
  - UncleEntity 5y ago
    I think par2 just uses a list of hashes to determine if it needs to do some error correction. Though I do see the value of checking the files when first ‘checked in’ to this recovery scheme so you start with a known good copy. The idea is solid and since it has transcoding build in it makes perfect sense to just use ffmpeg. —edit— The author beat me to the answer…
- fnordpiglet 5y ago
  In addition to the other comments I don’t think a merkle tree would help in this situation. Bitrot is a random event and wouldn’t trigger anything that would take advantage of the tree data structure.
arendtio 5y ago
Just to give you an early warning: Someone else seems to have a project with a similar name. At least the Arch Linux User Repositories (AUR) have this package and it seems to be a different project: https://aur.archlinux.org/packages/automedia-git https://aur.archlinux.org/packages/automedia-git
tomc1985 5y ago
Curious as to how frequently bitrot shows up? Has anyone modeled this in terms of bits flipped per gigabyte per year or something like this? I have about half a terabyte of media, including some original music I made a long time ago and lost the project files for. Backups are great but it will just back of the corrupted versions
- joshstrange 5y ago
  I don't have hard and fast numbers but with ~120TB of media I can tell you I'm pretty sure I've had it happen or I've suffered from raid issues, I have no way to tell which. I've seen multiple instances when media that I know played flawlessly suddenly has stopped working. Most this occurred when I was using UnRaid (I still am for a few things) and could have been 100% my fault (most likely reason). I now just use a Synology and it has a data integrity feature that I have enabled. Time will tell if I suffer from "bitrot" again. Either way, bitrot or raid issues, a tool like this is interesting to me. I'd love something that watched my collection, stored the hashes, and altered me when they had changed (I know that's not what this tool does). I'm not sure I'd be willing to give up the space for something like par2 files but even a heads up other than "Oh, this isn't playing" would be nice.
  - mmastrac 5y ago
    I tried to make the tool easily pluggable so I wager someone could add hash support pretty easily. I might take a shot at it if I find some time as others seem to be interested in monitoring (but not paying the storage penalty) for bitrot.
- jmillikin 5y ago
  I'm currently migrating my media library from git-annex to a self-written replacement, and have encountered maybe ~300 instances of bitrot so far (out of ~5 TB). Detected through SHA256 checksum mismatch. Usually shows up as a run of zeros in an otherwise healthy file. So far all of the errors have been in files I could re-rip or recover from public sources, but I'm dreading finding an error in a file that's hard to replace. I'm cataloging those separately as I go along, and planning to do a full mirror in cloud storage.

71 points by mmastrac 5y ago

Awesome tool! Thank you :) May be a nice complement to my Video Hub App - browse, search, preview, and organize videos on Win, Mac, and Linux. https://videohubapp.com/ https://videohubapp.com/ & https://github.com/whyboris/Video-Hub-App https://github.com/whyboris/Video-Hub-App

mmastrac 5y ago

Hey all, a tiny project I wrote for myself over the last couple of months. I've been ripping my old physical audio and video media to a server running locally at home and have been worried about bitrot. I built this tool to check a large media library for correctness (by test-decoding it with FFMPEG), and to prevent future bitrot by automatically adding per-directory PAR2 files. The tool has another feature to transcode your entire library to a given format - I'm using this to keep a USB key in my car filled with AAC 64k files for listening when Spotify is out of range or I'm too lazy to connect via Bluetooth. It's designed to be somewhat easy to extend with media operations that consume a directory at a time. For example, you could add or check music metadata using a third-party service like MusicBrainz.

chrisweekly 5y ago
Great idea! Thanks for building and sharing it; looking fwd to looking closer at it after Mother's Day!
sirius87 5y ago
This is pretty good! Detecting bitrot is fine, can also be done by storing a MD5/SHA256 of the file content. `md5sum <file>` barfs if its unable to read the file from disk. Challenge is rebuilding the broken file. I was worried about bitrot on my old external drives too. I eventually went with btrfs[1] filesystem running RAID1 across 2 disks. [1] https://wiki.archlinux.org/title/btrfs https://wiki.archlinux.org/title/btrfs
- mmastrac 5y ago
  Adding/verifying hashes on media might not be a bad feature to add for those who don't want to burn the extra space required for PAR2. I run w/two extra parity disks on unRAID, but I'm too paranoid to leave my data at rest without an extra file-level of parity.
Wowfunhappy 5y ago
Did you consider storing your media in a ZFS pool?
- pronoiac 5y ago
  Checksum and PAR2 files are also useful alongside backup copies.
aspenmayer 5y ago
What’s a good method for making the original lossless rips, making parity files, and keeping them organized for use with Automedia? Is there something like Calibre for video/audio archives?
- mmastrac 5y ago
  MusicBrainz Picard will do metadata tagging and some basic directory organization. It's what I use to create my music directories. With a great metadata manager like that, any software that rips to FLAC or Apple Lossless should be sufficient. Exact Audio Copy is one of the best ones on the Windows side.
  - aspenmayer 4y ago
    I appreciate these recommendations for audio content; I’m mainly looking for resources for archiving video content.

saamerm 5y ago

Great work! Thanks for sharing and making it Open Source!

daedalus2027 5y ago

hi, I don't understand why they use ffmpeg for bitrot checking and not a simple hash checksum or a merkle tree.

arendtio 5y ago
From my very limited understanding of the software I understand, that it doesn't compare an earlier version with a later version as you would with hash sums, but rather utilizes ffmpeg to check if something odd/irregular shows up. In addition, it can use PAR2 files which can be used to restore the original version which would not easily be possible with hash sums. But as I said, I just read the description, nothing more.
- mmastrac 5y ago
  This is correct. I have a large media library and I wasn't sure how rotted it was. Test decoding won't catch every error, but it identified albums where there has clearly been a bitflip in a song (at which point I can try to find a lossless copy of that song to replace it). Once I'm happy with the state of the library, I use automedia to add on par2 files to avoid this going forward. Verification is for libraries that have no existing hashes or parity, essentially.
- UncleEntity 5y ago
  I think par2 just uses a list of hashes to determine if it needs to do some error correction. Though I do see the value of checking the files when first ‘checked in’ to this recovery scheme so you start with a known good copy. The idea is solid and since it has transcoding build in it makes perfect sense to just use ffmpeg. —edit— The author beat me to the answer…
fnordpiglet 5y ago
In addition to the other comments I don’t think a merkle tree would help in this situation. Bitrot is a random event and wouldn’t trigger anything that would take advantage of the tree data structure.

arendtio 5y ago

Just to give you an early warning: Someone else seems to have a project with a similar name. At least the Arch Linux User Repositories (AUR) have this package and it seems to be a different project: https://aur.archlinux.org/packages/automedia-git https://aur.archlinux.org/packages/automedia-git

tomc1985 5y ago

Curious as to how frequently bitrot shows up? Has anyone modeled this in terms of bits flipped per gigabyte per year or something like this? I have about half a terabyte of media, including some original music I made a long time ago and lost the project files for. Backups are great but it will just back of the corrupted versions

joshstrange 5y ago
I don't have hard and fast numbers but with ~120TB of media I can tell you I'm pretty sure I've had it happen or I've suffered from raid issues, I have no way to tell which. I've seen multiple instances when media that I know played flawlessly suddenly has stopped working. Most this occurred when I was using UnRaid (I still am for a few things) and could have been 100% my fault (most likely reason). I now just use a Synology and it has a data integrity feature that I have enabled. Time will tell if I suffer from "bitrot" again. Either way, bitrot or raid issues, a tool like this is interesting to me. I'd love something that watched my collection, stored the hashes, and altered me when they had changed (I know that's not what this tool does). I'm not sure I'd be willing to give up the space for something like par2 files but even a heads up other than "Oh, this isn't playing" would be nice.
- mmastrac 5y ago
  I tried to make the tool easily pluggable so I wager someone could add hash support pretty easily. I might take a shot at it if I find some time as others seem to be interested in monitoring (but not paying the storage penalty) for bitrot.
jmillikin 5y ago
I'm currently migrating my media library from git-annex to a self-written replacement, and have encountered maybe ~300 instances of bitrot so far (out of ~5 TB). Detected through SHA256 checksum mismatch. Usually shows up as a run of zeros in an otherwise healthy file. So far all of the errors have been in files I could re-rip or recover from public sources, but I'm dreading finding an error in a file that's hard to replace. I'm cataloging those separately as I go along, and planning to do a full mirror in cloud storage.