7 ms·

Compact Disc Structure

237 points by arantius 7y ago

nullc 7y ago
I've long been sad that no one has yet built a state of the art decoder for the recovery of error riddled CDDA discs. A lot of old CDs are not holding up as well as advertised, resulting in an actual loss of archived information. The RS decoders used in typical CD players doesn't come anywhere near the theoretical performance possible from the format, they're only used as erasure codes. In particular, list decoding with a prior on the audio continuity (using techniques like the Postfish declipper) should do really well because of the the enormous and channel independent interleave used by CDDA... but I've never seen it detecting. 30 years ago that sort of thing would have been computationally infeasible, but it should be possible now.
- daveslash 7y ago
  >> A lot of old CDs are not holding up as well as advertised Are you referring to pressed or burned CDs? Or Both?
  - hnlmorg 7y ago
    Sadly even pressed CDs have been known to rot[0][1]. Albeit it's not I tend to hear about with audio CDs because (presumably because there are usually other ways to acquire copies) however it is something that worries a lot of retro gamers like myself. It means that many of the Mega CD, Saturn, etc games (for example) have a potential life expectency. For me though, scratched discs affect me more than disc rot. It can be a lottery sometimes when buying retro games because you can't always be sure just how well looked after the disc was (even when you have pictures and the item is marked as tested/working). [0] https://en.wikipedia.org/wiki/Disc_rot https://en.wikipedia.org/wiki/Disc_rot [1] https://blog.discogs.com/en/say-no-to-disc-rot-how-to-look-after-cds/ https://blog.discogs.com/en/say-no-to-disc-rot-how-to-look-a...
    - codetrotter 7y ago
      > even when you have pictures and the item is marked as tested/working I bought a PlayStation 2 second hand a while ago to relive some memories and to play some of the games that I hadn’t yet played. One of the games played perfectly well up until about 60% through the story and then there is a bad scratch on the disc at which point the game freezes. I think most sellers of used games will only pop the disc in and see that it starts, if they even bother to test the game at all. That is reasonable of course — we can’t expect anyone to play through a whole game another time over just to see that the whole thing works as it should before they sell it for a couple of dollars. But yeah it’s a risk and it does suck a little bit.
      - Aengeuad 7y ago
        Had a similar problem when buying a used copy of Metal Gear Solid 1 on the PS1, the game and both discs worked fine but if you died on the second disc while in story mode the game would lock up. Added a bit of extra difficulty at any rate!
      - behringer 7y ago
        If you're buying a used game you can be sure it isn't tested and is sold as is. Even if they say it's tested or guaranteed, that only means they're willing to give you a refund if anything is wrong.
      - rasz 7y ago
        CD data is stored on the "paint" side of the disk https://en.wikipedia.org/wiki/Blu-ray#/media/File:Comparison_CD_DVD_HDDVD_BD.svg https://en.wikipedia.org/wiki/Blu-ray#/media/File:Comparison... , so if the scratch is on the shiny side you can buff it out.
    - byuu 7y ago
      I feel we're already reaching a point where game preservation for older systems like Mega CD and PC Engine CD really need to have multiple rips that then weigh all the rips to try and correct for individual errors in each rip. In fact I'm pretty sure groups like Redump are probably doing just that. Fun related article: https://john-millikin.com/%F0%9F%A4%94/why-i-ripped-the-same-cd-300-times https://john-millikin.com/%F0%9F%A4%94/why-i-ripped-the-same...
- byuu 7y ago
  > The RS decoders used in typical CD players doesn't come anywhere near the theoretical performance possible from the format, they're only used as erasure codes. Reed-Solomon is fascinating, especially the way both RSPC and CIRC have both P and Q parity. Depending on the order and how many passes you do of each, you can recover more and more data. If you can guess where the error is, you can correct up to twice as many errors. And as a last resort for RSPC, you can potentially brute force corrections via the checksum values. Reed-Solomon error correction ends up being more of an art than a science, and it's why different CD-ROM drives have different success rates at reading damaged discs. Explaining Reed-Solomon would be a fun article, but probably too intense and niche a subject. Tons of linear algebra over galois fields, and intimidating if not overly complex algorithms like Berlekamp-Massey, Chien search, and Forney's formula (some of which can be brute forced these days.) RSPC makes it even more complex with two overlapping channels of parity. CIRC (cross-interleave) makes that even more complex still with differing delay slots per byte of error correcting data. See pages 45 and 46: http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-130.pdf http://www.ecma-international.org/publications/files/ECMA-ST...
  - tinus_hn 7y ago
    The same error correction is used in qr codes. To correct errors you have to guess which bits are unsure.
    - nullc 7y ago
      You don't have to guess, in that if there are below a threshold number of errors there is a unique decode which will always be right. And a decoder that uses Berlekamp-Massey can find that unique decode with a lot less work than trying every possibility. :)
- Erlich_Bachman 7y ago
  Small clarification: "state of the art" just means "the best currently available", not "a perfect and effective solution that handles all the problems and has all the possbile functionality". So if no one has ever built a decoder of a certain quality level, that specific level can by definition not be "state of the art".
deleted 7y ago
[deleted]
sohkamyung 7y ago
Having helped to design some early CD-ROM drives many years ago (initially using Philips CDROM components), the information here brings back memories. :-) On a hardware level, the first thing we do is do a eye pattern check to see if all the modulated signals are coming in as expected [1]. If not, a hardware check on components is performed. [1] http://www.repairfaq.org/REPAIR/F_cdfaqd.html#CDFAQD_004 http://www.repairfaq.org/REPAIR/F_cdfaqd.html#CDFAQD_004
byuu 7y ago
Emulator development has cross-sections across many diverse fields of computer science, and I'm trying to share my experiences with all of them. If anyone here has suggestions for future articles for my site, I'm all ears ^-^ While folks are thinking about CDs: I've not been able to get an F2/F3-frame CD image. If anyone has such a thing (not copyrighted, just a CD containing a text file or public domain music piece is fine), I'd love to get it so I can implement a CIRC encoder/decoder! My GitHub repository (linked in the article) has an RSPC Reed-Solomon encoder/decoder, a disc scrambler/descrambler, and Q-subchannel encoder/decoder. I'm trying to build up a comprehensive ISC-licensed C++ library for working with every frame level of CD data. For folks interested in the lead-in / lead-out data, to my knowledge only very few drives allow reading this raw data, namely older Plextor drives with the 0xd8 (READ_CDDA) command. If there are any hardware engineers who want to save an entire generation of CD history in a more exacting archival format, we could desperately use either hacked drive firmware for a modern CD drive, or custom CD reading hardware to preserve this data. Time is really of the essence. The vanguard of CD preservation right now, to the best of my knowledge, is Claunia's DiscImageChef project: http://discimagechef.claunia.com/ http://discimagechef.claunia.com/ I'm also working with the Game Preservation Society in Japan ( https://www.gamepres.org/ https://www.gamepres.org/ ), which has archives of thousands of rare and obscure Japanese CD media, so by all means if anyone reading this can help, please let me know.
- rossy 7y ago
  > While folks are thinking about CDs: I've not been able to get an F2/F3-frame CD image. If anyone has such a thing (not copyrighted, just a CD containing a text file or public domain music piece is fine), I'd love to get it so I can implement a CIRC encoder/decoder! I don't know if you've already seen this, but I've heard of someone getting an image like this. They used it to discover what the AccurateRip drive offset values mean in terms of the physical location of the subchannel data and interleaved main channel data on the disc. There's a vague description of their process in there as well. https://club.myce.com/t/offsets-handling-syncing-of-audio-data-vs-q-channel/100931 https://club.myce.com/t/offsets-handling-syncing-of-audio-da... > Using my neat hardware setup I now have captured a precious 0.1 seconds worth of 588-bit channel data frames from a pristine audio CD, each frame containing, among other things, a subchannel byte and 24 bytes of audio data. > I can reproduce the CIRC, get the de-interleaved audio data, and also do the Reed-Solomon C1/C2 stuff in software from my analog laser readouts - showing an error-free read, identical to what I get with my Plextor using somewhat more conventional methods.
pedrow 7y ago
Fascinating article , thanks. > Some bands got clever ... by rewinding, you would reveal a hidden "track 0" of audio. Anyone know any discs that have this?
- rossy 7y ago
  There's a good list here: https://en.wikipedia.org/wiki/List_of_albums_with_tracks_hidden_in_the_pregap https://en.wikipedia.org/wiki/List_of_albums_with_tracks_hid... I was using Arcade Fire's Reflektor as a test disc for this a little while back.
jovica 7y ago
asd
1wd 7y ago
> older Plextor drives, which CD-ROM preservationists have the ability to acquire, although they are quite pricey How could such a drive be identified? Is there an exact model? How pricey are they?
- noonespecial 7y ago
  >How pricey are they? Hopefully not very. I used to work at a CD duplication place and we had 100's of them in giant robotic duplicator towers. They were the drive of choice for exactly these sorts of capabilities. When we closed down I pitched them all into the dumpster because at the time I couldn't even get $5 each for them on ebay. It would hurt to discover that they are now a rare and expensive find!
- einr 7y ago
  From https://github.com/saramibreak/DiscImageCreator https://github.com/saramibreak/DiscImageCreator: PX-760, PX-755, PX-716, PX-714, PX-712, PX-708, PX-704, Premium2, Premium, PX-W5224, PX-4824, PX-4012 I don't believe this is an exhaustive list, Plextor had a lot of good models back in the day. You'll want to look at older, true Plextor models, all SCSI or ATAPI. Once SATA rolled around they'd started rebadging drives from other manufacturers that do not have the same capabilities (or quality) Depending on the model you're looking at maybe $50-$150 on eBay, which is pricy for a used old CD-ROM drive but not a lot of money for someone who really needs one. The Premium/Premium2 models in particular are sought after by audio professionals and (used to) get quite pricy.
  - aidenn0 7y ago
    $50 isn't that pricey. Similar to the cost of a new BD-ROM drive. I was imagining a lot more from TFA. What's a good way to hook up an ATAPI drive to a modern computer? I'm curious if mine still works.
    - markrages 7y ago
      It's just an IDE interface. You can buy USB or PCI-express adapters for $20 or so.
userbinator 7y ago
For those really curious, the standard explaining the detailed structure of data CDs is free: https://www.ecma-international.org/publications/standards/Ecma-130.htm https://www.ecma-international.org/publications/standards/Ec... DVD-ROM is here too, for comparison: https://www.ecma-international.org/publications/standards/Ecma-267.htm https://www.ecma-international.org/publications/standards/Ec... And indeed there's really not much point in doing so. Any disc copy protection scheme trying to mess with CIRC codes, or worse, EFM codes, would have a really hard time having any drives read the resulting discs. Some copy protections did play around with EFM, leading to "weak sectors". Amusingly enough, when I tried to search for info on it for this post, I found an article previously linked from HN, and which quoted me... https://john-millikin.com/%F0%9F%A4%94/error-beneath-the-wavs https://john-millikin.com/%F0%9F%A4%94/error-beneath-the-wav...
garaetjjte 7y ago
I don't quite get why scrambling XOR is necessary. Isn't avoiding consecutive bits already achieved by EFM?
- tenebrisalietum 7y ago
  I think because of you can still get long sequences of 1's or 0's depending on which bytes are next to each other. http://www.reverse-engineering.info/CD/EFM_Table.html http://www.reverse-engineering.info/CD/EFM_Table.html
  - byuu 7y ago
    Correct. And since the scrambling XOR pattern is known, it's possible to burn a disc that ends up being all zeroes or ones post-scrambling anyway. I'm told such sectors tend to be unreadable, but I haven't tried it.
  - garaetjjte 7y ago
    But there are also 3 "merge bits" between each EFM code, so this shouldn't matter? edit: The problem solved by scrambler is that number of pits and lands must be equal on average, or else drive couldn't distinguish pit and lands as dark/light decision signal level wouldn't be longer average of output signal. There is excellent description here: https://archive.org/download/CDCrackingUncoveredProtectionAgainstUnsanctionedCDCopyingKrisKaspersky/CD%20Cracking%20Uncovered%20-%20Protection%20Against%20Unsanctioned%20CD%20Copying%20-%20Kris%20Kaspersky.pdf https://archive.org/download/CDCrackingUncoveredProtectionAg..., page 33 in PDF under "Sync Groups, Merging Bits, and DSV" header)

237 points by arantius 7y ago

I've long been sad that no one has yet built a state of the art decoder for the recovery of error riddled CDDA discs. A lot of old CDs are not holding up as well as advertised, resulting in an actual loss of archived information. The RS decoders used in typical CD players doesn't come anywhere near the theoretical performance possible from the format, they're only used as erasure codes. In particular, list decoding with a prior on the audio continuity (using techniques like the Postfish declipper) should do really well because of the the enormous and channel independent interleave used by CDDA... but I've never seen it detecting. 30 years ago that sort of thing would have been computationally infeasible, but it should be possible now.

daveslash 7y ago
>> A lot of old CDs are not holding up as well as advertised Are you referring to pressed or burned CDs? Or Both?
- hnlmorg 7y ago
  Sadly even pressed CDs have been known to rot[0][1]. Albeit it's not I tend to hear about with audio CDs because (presumably because there are usually other ways to acquire copies) however it is something that worries a lot of retro gamers like myself. It means that many of the Mega CD, Saturn, etc games (for example) have a potential life expectency. For me though, scratched discs affect me more than disc rot. It can be a lottery sometimes when buying retro games because you can't always be sure just how well looked after the disc was (even when you have pictures and the item is marked as tested/working). [0] https://en.wikipedia.org/wiki/Disc_rot https://en.wikipedia.org/wiki/Disc_rot [1] https://blog.discogs.com/en/say-no-to-disc-rot-how-to-look-after-cds/ https://blog.discogs.com/en/say-no-to-disc-rot-how-to-look-a...
  - codetrotter 7y ago
    > even when you have pictures and the item is marked as tested/working I bought a PlayStation 2 second hand a while ago to relive some memories and to play some of the games that I hadn’t yet played. One of the games played perfectly well up until about 60% through the story and then there is a bad scratch on the disc at which point the game freezes. I think most sellers of used games will only pop the disc in and see that it starts, if they even bother to test the game at all. That is reasonable of course — we can’t expect anyone to play through a whole game another time over just to see that the whole thing works as it should before they sell it for a couple of dollars. But yeah it’s a risk and it does suck a little bit.
    - Aengeuad 7y ago
      Had a similar problem when buying a used copy of Metal Gear Solid 1 on the PS1, the game and both discs worked fine but if you died on the second disc while in story mode the game would lock up. Added a bit of extra difficulty at any rate!
    - behringer 7y ago
      If you're buying a used game you can be sure it isn't tested and is sold as is. Even if they say it's tested or guaranteed, that only means they're willing to give you a refund if anything is wrong.
    - rasz 7y ago
      CD data is stored on the "paint" side of the disk https://en.wikipedia.org/wiki/Blu-ray#/media/File:Comparison_CD_DVD_HDDVD_BD.svg https://en.wikipedia.org/wiki/Blu-ray#/media/File:Comparison... , so if the scratch is on the shiny side you can buff it out.
  - byuu 7y ago
    I feel we're already reaching a point where game preservation for older systems like Mega CD and PC Engine CD really need to have multiple rips that then weigh all the rips to try and correct for individual errors in each rip. In fact I'm pretty sure groups like Redump are probably doing just that. Fun related article: https://john-millikin.com/%F0%9F%A4%94/why-i-ripped-the-same-cd-300-times https://john-millikin.com/%F0%9F%A4%94/why-i-ripped-the-same...
byuu 7y ago
> The RS decoders used in typical CD players doesn't come anywhere near the theoretical performance possible from the format, they're only used as erasure codes. Reed-Solomon is fascinating, especially the way both RSPC and CIRC have both P and Q parity. Depending on the order and how many passes you do of each, you can recover more and more data. If you can guess where the error is, you can correct up to twice as many errors. And as a last resort for RSPC, you can potentially brute force corrections via the checksum values. Reed-Solomon error correction ends up being more of an art than a science, and it's why different CD-ROM drives have different success rates at reading damaged discs. Explaining Reed-Solomon would be a fun article, but probably too intense and niche a subject. Tons of linear algebra over galois fields, and intimidating if not overly complex algorithms like Berlekamp-Massey, Chien search, and Forney's formula (some of which can be brute forced these days.) RSPC makes it even more complex with two overlapping channels of parity. CIRC (cross-interleave) makes that even more complex still with differing delay slots per byte of error correcting data. See pages 45 and 46: http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-130.pdf http://www.ecma-international.org/publications/files/ECMA-ST...
- tinus_hn 7y ago
  The same error correction is used in qr codes. To correct errors you have to guess which bits are unsure.
  - nullc 7y ago
    You don't have to guess, in that if there are below a threshold number of errors there is a unique decode which will always be right. And a decoder that uses Berlekamp-Massey can find that unique decode with a lot less work than trying every possibility. :)
Erlich_Bachman 7y ago
Small clarification: "state of the art" just means "the best currently available", not "a perfect and effective solution that handles all the problems and has all the possbile functionality". So if no one has ever built a decoder of a certain quality level, that specific level can by definition not be "state of the art".

deleted 7y ago

[deleted]

sohkamyung 7y ago

Having helped to design some early CD-ROM drives many years ago (initially using Philips CDROM components), the information here brings back memories. :-) On a hardware level, the first thing we do is do a eye pattern check to see if all the modulated signals are coming in as expected [1]. If not, a hardware check on components is performed. [1] http://www.repairfaq.org/REPAIR/F_cdfaqd.html#CDFAQD_004 http://www.repairfaq.org/REPAIR/F_cdfaqd.html#CDFAQD_004

byuu 7y ago

Emulator development has cross-sections across many diverse fields of computer science, and I'm trying to share my experiences with all of them. If anyone here has suggestions for future articles for my site, I'm all ears ^-^ While folks are thinking about CDs: I've not been able to get an F2/F3-frame CD image. If anyone has such a thing (not copyrighted, just a CD containing a text file or public domain music piece is fine), I'd love to get it so I can implement a CIRC encoder/decoder! My GitHub repository (linked in the article) has an RSPC Reed-Solomon encoder/decoder, a disc scrambler/descrambler, and Q-subchannel encoder/decoder. I'm trying to build up a comprehensive ISC-licensed C++ library for working with every frame level of CD data. For folks interested in the lead-in / lead-out data, to my knowledge only very few drives allow reading this raw data, namely older Plextor drives with the 0xd8 (READ_CDDA) command. If there are any hardware engineers who want to save an entire generation of CD history in a more exacting archival format, we could desperately use either hacked drive firmware for a modern CD drive, or custom CD reading hardware to preserve this data. Time is really of the essence. The vanguard of CD preservation right now, to the best of my knowledge, is Claunia's DiscImageChef project: http://discimagechef.claunia.com/ http://discimagechef.claunia.com/ I'm also working with the Game Preservation Society in Japan ( https://www.gamepres.org/ https://www.gamepres.org/ ), which has archives of thousands of rare and obscure Japanese CD media, so by all means if anyone reading this can help, please let me know.

rossy 7y ago
> While folks are thinking about CDs: I've not been able to get an F2/F3-frame CD image. If anyone has such a thing (not copyrighted, just a CD containing a text file or public domain music piece is fine), I'd love to get it so I can implement a CIRC encoder/decoder! I don't know if you've already seen this, but I've heard of someone getting an image like this. They used it to discover what the AccurateRip drive offset values mean in terms of the physical location of the subchannel data and interleaved main channel data on the disc. There's a vague description of their process in there as well. https://club.myce.com/t/offsets-handling-syncing-of-audio-data-vs-q-channel/100931 https://club.myce.com/t/offsets-handling-syncing-of-audio-da... > Using my neat hardware setup I now have captured a precious 0.1 seconds worth of 588-bit channel data frames from a pristine audio CD, each frame containing, among other things, a subchannel byte and 24 bytes of audio data. > I can reproduce the CIRC, get the de-interleaved audio data, and also do the Reed-Solomon C1/C2 stuff in software from my analog laser readouts - showing an error-free read, identical to what I get with my Plextor using somewhat more conventional methods.

pedrow 7y ago

Fascinating article , thanks. > Some bands got clever ... by rewinding, you would reveal a hidden "track 0" of audio. Anyone know any discs that have this?

rossy 7y ago
There's a good list here: https://en.wikipedia.org/wiki/List_of_albums_with_tracks_hidden_in_the_pregap https://en.wikipedia.org/wiki/List_of_albums_with_tracks_hid... I was using Arcade Fire's Reflektor as a test disc for this a little while back.

jovica 7y ago

asd

1wd 7y ago

> older Plextor drives, which CD-ROM preservationists have the ability to acquire, although they are quite pricey How could such a drive be identified? Is there an exact model? How pricey are they?

noonespecial 7y ago
>How pricey are they? Hopefully not very. I used to work at a CD duplication place and we had 100's of them in giant robotic duplicator towers. They were the drive of choice for exactly these sorts of capabilities. When we closed down I pitched them all into the dumpster because at the time I couldn't even get $5 each for them on ebay. It would hurt to discover that they are now a rare and expensive find!
einr 7y ago
From https://github.com/saramibreak/DiscImageCreator https://github.com/saramibreak/DiscImageCreator: PX-760, PX-755, PX-716, PX-714, PX-712, PX-708, PX-704, Premium2, Premium, PX-W5224, PX-4824, PX-4012 I don't believe this is an exhaustive list, Plextor had a lot of good models back in the day. You'll want to look at older, true Plextor models, all SCSI or ATAPI. Once SATA rolled around they'd started rebadging drives from other manufacturers that do not have the same capabilities (or quality) Depending on the model you're looking at maybe $50-$150 on eBay, which is pricy for a used old CD-ROM drive but not a lot of money for someone who really needs one. The Premium/Premium2 models in particular are sought after by audio professionals and (used to) get quite pricy.
- aidenn0 7y ago
  $50 isn't that pricey. Similar to the cost of a new BD-ROM drive. I was imagining a lot more from TFA. What's a good way to hook up an ATAPI drive to a modern computer? I'm curious if mine still works.
  - markrages 7y ago
    It's just an IDE interface. You can buy USB or PCI-express adapters for $20 or so.

userbinator 7y ago

For those really curious, the standard explaining the detailed structure of data CDs is free: https://www.ecma-international.org/publications/standards/Ecma-130.htm https://www.ecma-international.org/publications/standards/Ec... DVD-ROM is here too, for comparison: https://www.ecma-international.org/publications/standards/Ecma-267.htm https://www.ecma-international.org/publications/standards/Ec... And indeed there's really not much point in doing so. Any disc copy protection scheme trying to mess with CIRC codes, or worse, EFM codes, would have a really hard time having any drives read the resulting discs. Some copy protections did play around with EFM, leading to "weak sectors". Amusingly enough, when I tried to search for info on it for this post, I found an article previously linked from HN, and which quoted me... https://john-millikin.com/%F0%9F%A4%94/error-beneath-the-wavs https://john-millikin.com/%F0%9F%A4%94/error-beneath-the-wav...

garaetjjte 7y ago

I don't quite get why scrambling XOR is necessary. Isn't avoiding consecutive bits already achieved by EFM?

tenebrisalietum 7y ago
I think because of you can still get long sequences of 1's or 0's depending on which bytes are next to each other. http://www.reverse-engineering.info/CD/EFM_Table.html http://www.reverse-engineering.info/CD/EFM_Table.html
- byuu 7y ago
  Correct. And since the scrambling XOR pattern is known, it's possible to burn a disc that ends up being all zeroes or ones post-scrambling anyway. I'm told such sectors tend to be unreadable, but I haven't tried it.
- garaetjjte 7y ago
  But there are also 3 "merge bits" between each EFM code, so this shouldn't matter? edit: The problem solved by scrambler is that number of pits and lands must be equal on average, or else drive couldn't distinguish pit and lands as dark/light decision signal level wouldn't be longer average of output signal. There is excellent description here: https://archive.org/download/CDCrackingUncoveredProtectionAgainstUnsanctionedCDCopyingKrisKaspersky/CD%20Cracking%20Uncovered%20-%20Protection%20Against%20Unsanctioned%20CD%20Copying%20-%20Kris%20Kaspersky.pdf https://archive.org/download/CDCrackingUncoveredProtectionAg..., page 33 in PDF under "Sync Groups, Merging Bits, and DSV" header)