4 ms·

I've been dealing with a lot of CD copying problems lately. I'm working on a setup to automatically detect and correct errors reading audio CDs which I've found

I've been dealing with a lot of CD copying problems lately. I'm working on a setup to automatically detect and correct errors reading audio CDs which I've found don't use checksums or other mechanisms to prevent small defects from mangling the bitstream. I'd previously used cdparanoia, but found it not paranoid enough.(for some reason it always outputs aiff regardless of options as well) I'm looking at defeating the drive cache by actually reading the disc in a second and 3rd drive and doing a NASA style "all 3 must agree". It's annoying, but I've got 800,000 CDs to rip right now, and I have to make sure each one is perfect.(Without proprietary solutions) I also need to be able to positively reduplicate heavily scratched discs. It's a tough problem.

unnouinceput 6y ago
800k? are you insane? even if everything else is automated and you have 800K devices at your fingertip ready to go, the simple amount of time to lift CD, open player tray / insert CD / close tray is in decades!. Let's say you do this 8 hours per day for 5 days. The above series of movements can take around 30 seconds. That's 400k minutes which results in 400k / 40 hours == 10k weeks. At 50 weeks per year (only 2 weeks vacation each year) that still results in 200 years!!. Even if you do one CD every second this result in 200 years / 30 ==~7 years.
- pontifier 6y ago
  I very well might be. I took over Murfie when they shut down, and am in the process of re-building the company. People don't understand the scale, and why I might want to make sure I do it right the first time rather than get halfway through and have to start over because I missed something.
  - gruez 6y ago
    I'm assuming you're the guy in this article? https://www.theverge.com/2020/2/5/21121594/crossies-murfie-madison-wisconsin-arkansas-1-million-abandoned-cds https://www.theverge.com/2020/2/5/21121594/crossies-murfie-m...
    - pontifier 6y ago
      Yeah, that's me :)
  - owenmarshall 6y ago
    Are you really looking at 800k CDs to rip, or are you planning on removing duplicates and finding a good specimen?
    - pontifier 6y ago
      I will need to verify each disc. I may not have to rip all of them, but I will probably do some sort of rapid de-duplication check. It will probably be much more involved than the normal toc disc id though. I might compare bits from random areas of each disc to verify it is the correct pressing. I'm still not 100% sure how I'll do it, but I don't want to store more than I have to. I'd like to be able to take the pieces of a broken disc and verify somehow that it's a legit copy. Then I'd treat the pieces like every other copy of that disc and give the owner access.
  - unnouinceput 6y ago
    Ah, I made a terrible mistake on my Math above. It's 400k minutes which is actually only 6666.66 hours (400k / 60). So in the end you'll have only 200 years / 60 = 3.33 years. I suppose you're going to pay people to do it, so lemme help you a little. 6667 hours at let's say $15/hour that's roughly one million dollars for this task. I hope you have large pockets ;).
    - pontifier 6y ago
      Actually your math is way off. I don't think it will take more than about 10 seconds per disc to process them if I get things set up right and everything is well organized and streamlined. https://twitter.com/pontifier/status/1236002652468674562 https://twitter.com/pontifier/status/1236002652468674562 That's 6 per minute. 360 per hour. 800,000 should take about 2,222 hours. I've got a bunch of people that will work for about $10/hour, and that's only $22,220 worth of labor over 277 days for 1 person.
TacticalCoder 6y ago
Don't know if it can help but... If it's Audio CDs the problem is kinda solved in that you can make sure you extract a bit-perfect copy by verifying the checksum with the checksums of other people who extracted the same CD. Even though the Audio CD format tried to make it "hard" to rip in a bit-perfect way, it's done precisely like that since years and years. There's a community of people ripping Audio CD to flac file and making sure all their rips are bit-perfect. I've ripped my own collection and used, as far as I remember, "whipper" on Fedora Linux (for whatever reason I couldn't make that software work on Debian back then). After ripping a track/CD the software would automatically verify the checksum with an online DB of rips. You say you can't use proprietary solution: as far as I know there are several free rippers on Linux and the online DB of checksum isn't proprietary either (?). In case you rip with an error, it's near impossible that someone else who ripped the CD would have read exactly the same error and end up with the same checksum. Now: for all the audio CDs I tried there was at least one other person who had already ripped it but it's not always the case. For example I've read about some collection of classical music CDs coming in pack of 300 CDs (!) and nobody had bothered, at home, to rip them / to share the checksum. But it can already help for all the "common" CDs you own.
- pontifier 6y ago
  The exact audio copy checksum database is not an option for me because of it's closed nature and the provenance of the data in it.
  - cellularmitosis 6y ago
    You are ripping 800,000 CDs and you are willfully not taking advantage of the best solution available?
    - pontifier 6y ago
      It's not the best solution if I have to use a gui, and the software they have available to do it. I have scripts that will streamline things to a high degree. The EAC database doesn't seem to be open, and I don't feel like paying for it if I don't have to. I've got enough copies of each disc that I don't need their data, and I want a better check than they do.
garaetjjte 6y ago
>audio CDs which I've found don't use checksums or other mechanisms to prevent small defects from mangling the bitstream Redbook audio CDs does use CIRC error correction, by storing 8 bytes of parity data inside each 33 byte F3 frame. It is not enough to correct all errors though, and Yellowbook data CDs stores extra correction codes on top of it. (276 bytes for each 2352 byte sector). (and there's also issue below F3 frame level about EFM modulation, whether merging codes are generated properly to keep DSV low enough) See: https://byuu.net/compact-discs/structure/ https://byuu.net/compact-discs/structure/ https://john-millikin.com/%F0%9F%A4%94/why-i-ripped-the-same-cd-300-times https://john-millikin.com/%F0%9F%A4%94/why-i-ripped-the-same... https://john-millikin.com/%F0%9F%A4%94/error-beneath-the-wavs https://john-millikin.com/%F0%9F%A4%94/error-beneath-the-wav...
rasz 6y ago
>800,000 CDs This is going to be quite a task, even with a bunch of the big boy 1000 disk auto loaders. Im not familiar with any solutions capable of opening CD jewel cases. You might want to contact few hackerspaces to inquire about building specialized machines just for the task of loading disks into spindles. btw why not proprietary? like http://wiki.hydrogenaud.io/index.php?title=Exact_Audio_Copy http://wiki.hydrogenaud.io/index.php?title=Exact_Audio_Copy

unnouinceput 6y ago

800k? are you insane? even if everything else is automated and you have 800K devices at your fingertip ready to go, the simple amount of time to lift CD, open player tray / insert CD / close tray is in decades!. Let's say you do this 8 hours per day for 5 days. The above series of movements can take around 30 seconds. That's 400k minutes which results in 400k / 40 hours == 10k weeks. At 50 weeks per year (only 2 weeks vacation each year) that still results in 200 years!!. Even if you do one CD every second this result in 200 years / 30 ==~7 years.

pontifier 6y ago
I very well might be. I took over Murfie when they shut down, and am in the process of re-building the company. People don't understand the scale, and why I might want to make sure I do it right the first time rather than get halfway through and have to start over because I missed something.
- gruez 6y ago
  I'm assuming you're the guy in this article? https://www.theverge.com/2020/2/5/21121594/crossies-murfie-madison-wisconsin-arkansas-1-million-abandoned-cds https://www.theverge.com/2020/2/5/21121594/crossies-murfie-m...
  - pontifier 6y ago
    Yeah, that's me :)
- owenmarshall 6y ago
  Are you really looking at 800k CDs to rip, or are you planning on removing duplicates and finding a good specimen?
  - pontifier 6y ago
    I will need to verify each disc. I may not have to rip all of them, but I will probably do some sort of rapid de-duplication check. It will probably be much more involved than the normal toc disc id though. I might compare bits from random areas of each disc to verify it is the correct pressing. I'm still not 100% sure how I'll do it, but I don't want to store more than I have to. I'd like to be able to take the pieces of a broken disc and verify somehow that it's a legit copy. Then I'd treat the pieces like every other copy of that disc and give the owner access.
- unnouinceput 6y ago
  Ah, I made a terrible mistake on my Math above. It's 400k minutes which is actually only 6666.66 hours (400k / 60). So in the end you'll have only 200 years / 60 = 3.33 years. I suppose you're going to pay people to do it, so lemme help you a little. 6667 hours at let's say $15/hour that's roughly one million dollars for this task. I hope you have large pockets ;).
  - pontifier 6y ago
    Actually your math is way off. I don't think it will take more than about 10 seconds per disc to process them if I get things set up right and everything is well organized and streamlined. https://twitter.com/pontifier/status/1236002652468674562 https://twitter.com/pontifier/status/1236002652468674562 That's 6 per minute. 360 per hour. 800,000 should take about 2,222 hours. I've got a bunch of people that will work for about $10/hour, and that's only $22,220 worth of labor over 277 days for 1 person.

TacticalCoder 6y ago

Don't know if it can help but... If it's Audio CDs the problem is kinda solved in that you can make sure you extract a bit-perfect copy by verifying the checksum with the checksums of other people who extracted the same CD. Even though the Audio CD format tried to make it "hard" to rip in a bit-perfect way, it's done precisely like that since years and years. There's a community of people ripping Audio CD to flac file and making sure all their rips are bit-perfect. I've ripped my own collection and used, as far as I remember, "whipper" on Fedora Linux (for whatever reason I couldn't make that software work on Debian back then). After ripping a track/CD the software would automatically verify the checksum with an online DB of rips. You say you can't use proprietary solution: as far as I know there are several free rippers on Linux and the online DB of checksum isn't proprietary either (?). In case you rip with an error, it's near impossible that someone else who ripped the CD would have read exactly the same error and end up with the same checksum. Now: for all the audio CDs I tried there was at least one other person who had already ripped it but it's not always the case. For example I've read about some collection of classical music CDs coming in pack of 300 CDs (!) and nobody had bothered, at home, to rip them / to share the checksum. But it can already help for all the "common" CDs you own.

pontifier 6y ago
The exact audio copy checksum database is not an option for me because of it's closed nature and the provenance of the data in it.
- cellularmitosis 6y ago
  You are ripping 800,000 CDs and you are willfully not taking advantage of the best solution available?
  - pontifier 6y ago
    It's not the best solution if I have to use a gui, and the software they have available to do it. I have scripts that will streamline things to a high degree. The EAC database doesn't seem to be open, and I don't feel like paying for it if I don't have to. I've got enough copies of each disc that I don't need their data, and I want a better check than they do.

garaetjjte 6y ago

>audio CDs which I've found don't use checksums or other mechanisms to prevent small defects from mangling the bitstream Redbook audio CDs does use CIRC error correction, by storing 8 bytes of parity data inside each 33 byte F3 frame. It is not enough to correct all errors though, and Yellowbook data CDs stores extra correction codes on top of it. (276 bytes for each 2352 byte sector). (and there's also issue below F3 frame level about EFM modulation, whether merging codes are generated properly to keep DSV low enough) See: https://byuu.net/compact-discs/structure/ https://byuu.net/compact-discs/structure/ https://john-millikin.com/%F0%9F%A4%94/why-i-ripped-the-same-cd-300-times https://john-millikin.com/%F0%9F%A4%94/why-i-ripped-the-same... https://john-millikin.com/%F0%9F%A4%94/error-beneath-the-wavs https://john-millikin.com/%F0%9F%A4%94/error-beneath-the-wav...

rasz 6y ago

>800,000 CDs This is going to be quite a task, even with a bunch of the big boy 1000 disk auto loaders. Im not familiar with any solutions capable of opening CD jewel cases. You might want to contact few hackerspaces to inquire about building specialized machines just for the task of loading disks into spindles. btw why not proprietary? like http://wiki.hydrogenaud.io/index.php?title=Exact_Audio_Copy http://wiki.hydrogenaud.io/index.php?title=Exact_Audio_Copy