7 ms·

I dream of having a BTFS that will fix my "damaged" media files. E.g. ones I media shift, if my disk was scratched and portions are missing, or if the codec opt

I dream of having a BTFS that will fix my "damaged" media files. E.g. ones I media shift, if my disk was scratched and portions are missing, or if the codec options I picked suck, it could download the "damaged" portions of my media and fix it seamlessly.

gosub100 2y ago
another use of this is to share media after I've imported it into my library. if I voluntarily scan hashes of all my media, if a smart torrent client could offer those files only (so a partial torrent because I always remove the superfluous files) it would help seed a lot of rare media files.
- Stephen304 2y ago
  This happens to be one of the pipe dream roadmap milestones for bitmagnet: https://bitmagnet.io/#pipe-dream-features https://bitmagnet.io/#pipe-dream-features I used to use magnetico and wanted to make something that would use crawled info hashes to fetch the metadata and retrieve the file listing, then search a folder for any matching files. You'd probably want to pre-hash everything in the folder and cache the hashes. I hope bitmagnet gets that ability, it would be super cool
- jonhohle 2y ago
  I’ve done a lot of archival of CD-ROM based games, and it’s not clear to me this is possible without a lot of coordination and consistency (there are like 7 programs that use AccurateRip, )and those only deal with audio). I have found zero instances where a bin/cue I’ve downloaded online perfectly matches (hashes) to the same disc I’ve ripped locally. I’ve had some instances where different pressings if the same content hash differently. I’ve written tools to inspect content (say in an ISO file system), and those will hash to the same value (so different sector data but the same resulting file system). Audio converted to CDDA (16-bit PCM) will hash as well. If audio is transcoded into anything else, there’s no way it would hash the same. At my last job I did something similar for build artifacts. You need the same compiler, same version, same settings, the ability to look inside the final artifact and avoid all the variable information (e.g. time). That requires a bit of domain specific information to get right.
  - gosub100 2y ago
    Sorry I think you misunderstood me. I mean when I download a torrent called "Rachmaninov Complete Discography" I copy the files to the Music/Classical folder on my NAS. I can no longer seed the torrent unless I leave the original in the shared folder. But if I voluntarily let a crawler index and share my Music folder, it could see the hash of track1.flac and know that it associates with a particular file in the original torrent, thus allowing others to download it.
pigpang 2y ago
How you will calculate hash of file, when it broken, to lookup for?
- 01HNNWZ0MV43FF 2y ago
  You could do a rolling hash and say that a chunk with a given hash should appear between two other chunks of certain hashes
  - arijun 2y ago
    That seems like a recipe for nefarious code insertion.
    - 01HNNWZ0MV43FF 2y ago
      oh shit yeah it does lol
- everfree 2y ago
  Just hash it before it's broken.
  - jonhohle 2y ago
    Maybe this is a joke that’s over my head, but the OP wants a system where damaged media can be repaired. They have the damaged media so there’s no way to make a hash of the content they want.
    - OnlyMortal 2y ago
      How far would error correction go?
- selcuka 2y ago
  Just use the sector number(s) of the damaged parts.
- 2y ago
kelchm 2y ago
Not the same as what you are talking about, but your comment reminded me of AccurateRip [1] which I used to make extensive use of back when I was ripping hundreds of CDs every year. 1: http://www.accuraterip.com/ http://www.accuraterip.com/
- neolithicum 2y ago
  Do you have any tricks you can share on how to rip a large library of CDs? It would be nice to semi-automate the ripping process but I haven't found any tools to help with that. Also the MusicBrainz audio tagging library (the only open one I am aware of?) almost never has good tags for my CDs that don't have to be edited afterwards.
  - mavhc 2y ago
    https://github.com/whipper-team/whipper https://github.com/whipper-team/whipper https://github.com/thomas-mc-work/most-possible-unattended-rip https://github.com/thomas-mc-work/most-possible-unattended-r... Finding a good CD drive to rip them is the first step. https://flemmingss.com/importing-data-from-discogs-and-others-to-musicbrainz/ https://flemmingss.com/importing-data-from-discogs-and-other... IME Discogs had the track data most often. And obviously rip to flac
    - neolithicum 2y ago
      Great suggestions, I'll have to try these out. Thank you!
  - bayindirh 2y ago
    I was ripping my CD's with KDE's own KIO interface, which also does CDDB checks and embeds original information to ID3 tags. Passing through MusicBrainz Picard always gave me good tags, but I remember fine tuning it a bit. Now, I'll start another round with DBPowerAmp's ripper on macOS, then I'll see which tool brings the better metadata.
  - jonhohle 2y ago
    In the distant past iTunes was great at this (really). Insert a disc, its metadata is pulled in automatically, it’s ripped and tagged using whatever coded settings you want and when it’s done the disc is ejected. Watch a show do some other work and when the toast pops out a new one in. Ripping DVDs with HandBrake was almost as easy, but it wouldn’t eject the disc afterwards (though it could have supported running a script at the end, I don’t recall).
Fnoord 2y ago
Distribute parity files together with the real deal, like they do on Usenet? Usenet itself is pretty much this anyway. Not sure if the NNTP filesystem implementations work. Also, there's nzbfs [1] [1] https://github.com/danielfullmer/nzbfs https://github.com/danielfullmer/nzbfs
drlemonpepper 2y ago
storj does this
tarnith 2y ago
Why not run a filesystem that maintains this? (ZFS exists, storage is cheap)

gosub100 2y ago

another use of this is to share media after I've imported it into my library. if I voluntarily scan hashes of all my media, if a smart torrent client could offer those files only (so a partial torrent because I always remove the superfluous files) it would help seed a lot of rare media files.

Stephen304 2y ago
This happens to be one of the pipe dream roadmap milestones for bitmagnet: https://bitmagnet.io/#pipe-dream-features https://bitmagnet.io/#pipe-dream-features I used to use magnetico and wanted to make something that would use crawled info hashes to fetch the metadata and retrieve the file listing, then search a folder for any matching files. You'd probably want to pre-hash everything in the folder and cache the hashes. I hope bitmagnet gets that ability, it would be super cool
jonhohle 2y ago
I’ve done a lot of archival of CD-ROM based games, and it’s not clear to me this is possible without a lot of coordination and consistency (there are like 7 programs that use AccurateRip, )and those only deal with audio). I have found zero instances where a bin/cue I’ve downloaded online perfectly matches (hashes) to the same disc I’ve ripped locally. I’ve had some instances where different pressings if the same content hash differently. I’ve written tools to inspect content (say in an ISO file system), and those will hash to the same value (so different sector data but the same resulting file system). Audio converted to CDDA (16-bit PCM) will hash as well. If audio is transcoded into anything else, there’s no way it would hash the same. At my last job I did something similar for build artifacts. You need the same compiler, same version, same settings, the ability to look inside the final artifact and avoid all the variable information (e.g. time). That requires a bit of domain specific information to get right.
- gosub100 2y ago
  Sorry I think you misunderstood me. I mean when I download a torrent called "Rachmaninov Complete Discography" I copy the files to the Music/Classical folder on my NAS. I can no longer seed the torrent unless I leave the original in the shared folder. But if I voluntarily let a crawler index and share my Music folder, it could see the hash of track1.flac and know that it associates with a particular file in the original torrent, thus allowing others to download it.

pigpang 2y ago

How you will calculate hash of file, when it broken, to lookup for?

01HNNWZ0MV43FF 2y ago
You could do a rolling hash and say that a chunk with a given hash should appear between two other chunks of certain hashes
- arijun 2y ago
  That seems like a recipe for nefarious code insertion.
  - 01HNNWZ0MV43FF 2y ago
    oh shit yeah it does lol
everfree 2y ago
Just hash it before it's broken.
- jonhohle 2y ago
  Maybe this is a joke that’s over my head, but the OP wants a system where damaged media can be repaired. They have the damaged media so there’s no way to make a hash of the content they want.
  - OnlyMortal 2y ago
    How far would error correction go?
selcuka 2y ago
Just use the sector number(s) of the damaged parts.
2y ago

kelchm 2y ago

Not the same as what you are talking about, but your comment reminded me of AccurateRip [1] which I used to make extensive use of back when I was ripping hundreds of CDs every year. 1: http://www.accuraterip.com/ http://www.accuraterip.com/

neolithicum 2y ago
Do you have any tricks you can share on how to rip a large library of CDs? It would be nice to semi-automate the ripping process but I haven't found any tools to help with that. Also the MusicBrainz audio tagging library (the only open one I am aware of?) almost never has good tags for my CDs that don't have to be edited afterwards.
- mavhc 2y ago
  https://github.com/whipper-team/whipper https://github.com/whipper-team/whipper https://github.com/thomas-mc-work/most-possible-unattended-rip https://github.com/thomas-mc-work/most-possible-unattended-r... Finding a good CD drive to rip them is the first step. https://flemmingss.com/importing-data-from-discogs-and-others-to-musicbrainz/ https://flemmingss.com/importing-data-from-discogs-and-other... IME Discogs had the track data most often. And obviously rip to flac
  - neolithicum 2y ago
    Great suggestions, I'll have to try these out. Thank you!
- bayindirh 2y ago
  I was ripping my CD's with KDE's own KIO interface, which also does CDDB checks and embeds original information to ID3 tags. Passing through MusicBrainz Picard always gave me good tags, but I remember fine tuning it a bit. Now, I'll start another round with DBPowerAmp's ripper on macOS, then I'll see which tool brings the better metadata.
- jonhohle 2y ago
  In the distant past iTunes was great at this (really). Insert a disc, its metadata is pulled in automatically, it’s ripped and tagged using whatever coded settings you want and when it’s done the disc is ejected. Watch a show do some other work and when the toast pops out a new one in. Ripping DVDs with HandBrake was almost as easy, but it wouldn’t eject the disc afterwards (though it could have supported running a script at the end, I don’t recall).

Fnoord 2y ago

Distribute parity files together with the real deal, like they do on Usenet? Usenet itself is pretty much this anyway. Not sure if the NNTP filesystem implementations work. Also, there's nzbfs [1] [1] https://github.com/danielfullmer/nzbfs https://github.com/danielfullmer/nzbfs

drlemonpepper 2y ago

storj does this

tarnith 2y ago

Why not run a filesystem that maintains this? (ZFS exists, storage is cheap)