9 ms·

Disc rot is a challenge for both archivists and collectors (2017)

137 points by anonymfus 7y ago

bigdubs 7y ago
The loss of what.cd wasn't difficult because it made piracy hard, it was because the users had created one of the best fully verified lossless cd audio archives on the internet. Much of this content has popped up (or even been extended) on replacement trackers, but it's only a matter of time before the heroic operators of these trackers get forced to shut down too.
- Iv 7y ago
  I remember the early days of piracy, Napster, Kaza, eDonkey. People were simply sharing all their files. There was pride in having a hard to obtain item, such as TV shows from the 60s, movies not released in DVDs, etc... P2P could have made archivists jobs so much easier. This is a tech that should have been the next iteration of internet that we killed for no good reason. Not even greed, just a refusal to evolve a profitable but obsolete system.
  - Aeolun 7y ago
    For me it had more to do with the bootleg sharing programs and the inability to download anything without 1500 different versions of malware also making it onto your computer. For some reason the torrent community doesn’t have quite the same problem.
  - Mediterraneo10 7y ago
    While early-millennium filesharing was useful for finding obscure content, the stuff available wasn’t up to the standards of professional archivists. In the library world, there are strict standards for how information is to be ripped, encoded, and tagged, but the content on filesharing networks seldom lived up to solid archival standards. It wasn’t until Oink and What.cd that we got a rigorous archive of audio material.
  - vbezhenar 7y ago
    How is P2P killed? I would say P2P matured. There's open source stable torrent software. Just use it, it works.
    - oneeyedpigeon 7y ago
      I think two things happened which effectively destroyed p2p - for the music-sharing case, for me specifically, at least. Streaming and greater official release of bootlegs, demos, etc. For me, there is so much available through these channels, that I don’t even consider p2p as an option.
    - bonoboTP 7y ago
      Torrent is still very centralized. The average user just downloads and seeds torrents made by a few torrent curators on each site. Becoming an uploader on a torrent site is a big privilege/responsibility that few people can take on. Other older platforms like Napster or DC++ (which actually still exists) let people share their own collection at the click of a button. This gave access to a really really long tail of rarities that perhaps only a handful of people were interested in, while torrent sites are mostly mainstream content.
      - dewey 7y ago
        > Becoming an uploader on a torrent site What do you mean by that? You can upload your torrents to any of the big public torrent sites and if it's a private site just being a member allows you to upload already. Public trackers are also all using DHT which makes this not centralized.
        bonoboTP 7y ago
        On private sites I know (in Hungary), being an uploader is a privilege one must earn, by being around for some time, earning the trust of the admins, having good uplink, learning all the rules and formats so all torrents are consistent etc. I guess perhaps about 0.1% of members upload stuff to these trackers. The situation may be different elsewhere. Context: Torrent is hugely popular in Hungary, basically everyone is a member who uses the internet under ~40 years and is even remotely technical, which means more than a million people (with the single most popular site having over 700k members, with account sharing explicitly allowed). Perhaps not so much for the newest teenager generation (I don't know). Until very recently (and still now) streaming services have been limited contentwise and people just cannot afford to buy all the movies, songs, games etc. with the same ease as in Western Europe / North America.
  - javajosh 7y ago
    I used to agree totally, but now I have a more nuanced view. While artificial scarcity offends my hacker sensibility, I also have to admit that this creates a society I like, with talented wealthy celebrities, and a large population of people who make their living in and around the industry. I'm not sure I would like a society where creative acts only happens in home studios and the work is given away for free, and all artists have to have day jobs.
    - Iv 7y ago
      I never said the new model was "give away everything for free" I just said that "Consider every copy needs to be paid even though making a copy is basically free of cost and that this forces us to put a terrible censorship mechanism into place" was not the right approach. Now that there are crowdfunding campaigns and tons of tools for microtransaction, that is even less excusable than when you had the effort to imagine the apparition of these obvious solutions.
      - hakfoo 7y ago
        I feel like we're screwed until we get over the "ownership" model. It means we spend more effort on metadata (What are we allowed to do with it? Who owns it?) than the actual data. It has stood directly in the way of new creative types (sampling in music, for example) and locked away content where the metadata is faulty/lost (orphan works problem) It falls apart badly at scale (if you wanted to get distribution rights for everything on an OSX/Windows installation image, imagine the number of third parties alone, plus the original developers, you'd need to sign off) and then it only really incentivizes work that's commercially appealing, not necessarily the technical best. I dream of a day where people who want to create will be able to get a stipend-- it might cover a modest lifestyle, but ensures they can live the dream of being an artist/musician/programmer without having to clutch tight over every scrap of their output to keep their income streams alive. If we did it through a tax-funded endowment rather than market forces, we could probably finance a lot more creativity that way (think how many touring singers we could finance at 50k per year for the price of one Taylor Swift, or how many programmers we could hire to work om their passion projects in database theory for what is currently being spent on Oracle licenses.)
- xouse 7y ago
  People have learned from what.cd. Lots of people have the same sense of inevitable doom you do so there's more people who have put thought and effort into being capable of rebuilding quickly if that happens again. Just varroa musica alone has created a giant decentralized metadata archive(and that's not even its main purpose just a neat side effect) that would make the restarting process much easier. The downside is that the post what community is so much smaller. Spotify et al really put a huge damper on the music tracker community, and that makes it less resilient.
  - Lammy 7y ago
    > The downside is that the post what community is so much smaller. People were saying the same thing in 2007 when OiNK shut down and What was starting up. The community is not smaller; it's more fragmented, and that's a good thing. Centralization is no better in our societal constructs than it is in our technological ones. I'll trade some immediacy and convenience for longevity any day.
    - xouse 7y ago
      I disagree, the community is both more fragmented and smaller, what had 144k users on its last day vs the 35k of its spiritual successor today. The few music trackers that have sprung up post what.cd after said spiritual successor are all pretty small. The niche genre trackers all have about the same user numbers since then so they cancel out. Then factor in that waffles has been down for some time now and the difference is pretty big. I can only estimate because exact numbers are hard to find, but I'd say peak what era music tracker community total users was in the 350k region and we're probably somewhere around 100k now.
  - WilTimSon 7y ago
    Does Spotify really cut that deep into the demographic of hardcore music tracker users? I'd presume those people are chasing rare and lost releases in high quality and Spotify and other streaming services aren't exactly touting rarities. Preserving underground music will always be a job for the fans, not streaming services.
- shortandsweet 7y ago
  What.cd most certainly wasn't the best. They were really stuck up and got what they deserved. I was amazed at the process just to get access to what they didn't even own. Waffles and oink were so much better.
  - serf 7y ago
    > I was amazed at the process just to get access to what they didn't even own. it was that process that helped to curate one of the highest quality audio libraries to have ever been. it's not all that important for most folks to listen to and compare 20 different releases of Bowie's 'Rise and Fall of Ziggy Stardust and the Spiders from Mars', but to those of us that enjoy music cataloguing as a hobby , the loss of what.cd was pretty terrible.
    - deleted 7y ago
      [deleted]
  - dewey 7y ago
    By what metric where Waffles and oink better? What.cd was run incredibly professionally, as you can see by the fact that it was active for almost 9 years and nothing happened to any of the people involved.
    - MivLives 7y ago
      They're complaining about the invite system. Either you had to know someone or you had to get lucky that they'd be doing irc interviews that day.
      - dewey 7y ago
        Then their criticism doesn't make much sense as both Waffles and Oink had an invite system. At least What.cd was more approachable as there was an interview system which made it possible for everyone to join as long as they learned some basics about audio formats / transcoding / site rules and had an hour to spend on an interview.
- dewey 7y ago
  Luckily the internet archive was also working on archiving the content on what.cd even though the results are not public (yet?). https://archive.org/details/whatcdcrawl https://archive.org/details/whatcdcrawl
  - ripdog 7y ago
    Getting the metadata is great, but the content won't come back.
- pontifier 7y ago
  I'm working on something that should be the best place to find any media... unfortunately the VidAngel vs Disney lawsuit has made things a little tougher. Fair use took a big blow there.
sowbug 7y ago
Has anyone looked into whether M Disc is less susceptible to the problems mentioned in TFA?
- masklinn 7y ago
  They claim to, but it's hard to determine that, and whether other archival-targeted discs (or which there are several e.g. Northern Star DataTresorDisc, MPO Gold, Verbatim Archival) are also resistant to disc rot, because it's unclear whether accelerated aging triggers disc rot in general or only some very specific instances thereof. (apparently defunct) syylex's GMD would most likely be structurally immune to disc rot, but you had / have to really really want to make your data rotproof: single-disc (4.7GB) order was 160€ (>$200 at the time). Not including VAT/GST.
pmoriarty 7y ago
For DVDs and CDs, error correction can be added using dvdisaster: http://dvdisaster.net/ http://dvdisaster.net/
- tzs 7y ago
  That link does not work, because there is no "A" record in DNS for dvdisaster.net. Here's the latest snapshot in archive.org [1]. It also seems to be available on Sourceforge [2], but that seems to be about 5 years behind what was on the now non-existent dvdisaster.net website. [1] https://web.archive.org/web/20180428070843/http://dvdisaster.net/en/index.html https://web.archive.org/web/20180428070843/http://dvdisaster... [2] https://sourceforge.net/projects/dvdisaster/ https://sourceforge.net/projects/dvdisaster/
userbinator 7y ago
Theoretically, optical media should be easier to recover than magnetic, because the technology to view their surfaces in detail is readily available --- a light microscope is enough to humanly view the actual recorded data on a CD and even a DVD: https://3.14.by/en/read/cd-dvd-microscope https://3.14.by/en/read/cd-dvd-microscope https://www.youtube.com/watch?v=Y8_HCLPdpkI https://www.youtube.com/watch?v=Y8_HCLPdpkI (CD-R) It would be interesting to see what "disc rot" looks like at that level of detail; and if better reading hardware may be able to read it. In particular, pressed discs actually store data in the varying height of the polycarbonate disc, so as long as those pits and lands haven't been damaged, even if the metallisation is gone, the data remains intact; and if there was a way to strip and re-coat that layer, pressed discs could be recovered. For CD-Rs, perhaps the dye has faded to the point where the typical drive (which can't really spin the disc slower than 1x) won't be able to discern the pits from the lands, but if that data can still be seen somehow, it's recoverable.
- dehrmann 7y ago
  TIL entirely numeric subdomains are legal.
- jacobush 7y ago
  Oh, so my rotted Laserdiscs can be recovered? :)
- BarsMonster 7y ago
  I am the author of these photos... If you have a sample of rotten disk - you can send it to me and I'll take a look. Contacts are on the site. Now my hardware is better, CD and DVD are easy to observe without physical damage. I also can see data on BD disk, but it is on the border of what is possible. I also tried to decode data from the image, but it will require more collaboration.
james-skemp 7y ago
I've got a pretty significant collection of PlayStation 1 and 2 games, and every time I see an article on disc rot I look to see when I need to be worried. Has anyone seen a list of when disc-based systems might start showing signs? Something like game consoles and capacitors.
- propter_hoc 7y ago
  I had a lot of problems with Dreamcast games in particular. I think the black-surfaced PS1 games seem to be a bit better made.
- stevenwoo 7y ago
  Not exactly what you are asking but I recently sold off my physical CD collection of about 500, after being in my closet for the past 20 years so no temperature/climate extremes, a few of them had signficant value on EBay, but when I tried to use Exact Audio Copy to verify they still could be read, about half no longer matched the online checksums or the original CD rip I had done before putting them in storage. There were three from Denon Japan that had physically deteriorated inside the disk, I could see there was a problem from a visual inspection.
  - rixrax 7y ago
    Aha! This must be why them audiophiles prefer to listen to their music using $$$ cd transport from original media! You can hear the difference! On a serious note, I’m in a similar boat, we’re in a middle of the move and it appears that my cd collection is in same predicament. So I’m debating whether to throw them to trash, or keep around (mostly for cover artwork, and to keep my Exact Audio Copy -> flac transfers just a bit more legit (I care of such things)).
    - diamondlovesyou 7y ago
      I dislike CD. My equipment is capable of 192khz/32bit (some only 24bit though), so CD are pretty much trash at 41khz. Better than MP3, solely due to the info MP3 throws away, but not great. Tidal goes higher, to 96khz on master quality where available (the vast number of tracks are not master quality, sadly).
nicholast 7y ago
I think I've heard that exposure to high variance in temperature can contribute. For example by storing CD's in your car.
propter_hoc 7y ago
This is such a huge problem for retro gamers. Any games on optical media, even those that aren't particularly old are at risk. As the article mentions, it's doubly infuriating given how optical media was marketed as more durable than cartridges. Meanwhile, a bunch of my Dreamcast and PS2 games have disc rot, while all my carts are still 100% playable. Besides disc rot, optical drives are also at high risk of the laser dying - this has happened to a PS1, PS2 and Dreamcast of mine. For these reasons, the most serious retro gamers I know are modding their optical drive systems to replace the CD drive with a SD-card based optical drive emulator. Anyway, thank heavens for emulation and the homebrew fan community. Without them, so much of video game history would be lost.
- deleted 7y ago
  [deleted]
- Grazester 7y ago
  I have not come across a Dreamcast game with rot. I have some early games which were just pressed badly and the top player in completely pealing off but it was not rot. The Saturn Discs on the other hand I have come across more than a few with rot.
pmarreck 7y ago
What prevents, or tries to prevent, bit rot in standard CPU hard drives/flash drives? Is there some built-in forward error correction at some layer? If not, how could one add some?
- Tomte 7y ago
  Nothing. There's par2 (and its offsprings MultiPar and QuickPar), there's rsbep, there's pyFileFixity. All have their problems why you probably won't "shield" your files with them (I don't, although I worry).
  - planteen 7y ago
    Curious - what are your complaints on these programs? My main one is that they are slow to create archives and rsbep only does error decoding, not erasure. I'm working on an alternative tool that is much faster, but curious if you have any other feedback.
- wazoox 7y ago
  Most SSDs only guarantee your data for 90 days when off. Even high grade, SLC, professional SSDs. Hard drives fare better, but your best bet for decade-long conservation remains tape.
  - ByThyGrace 7y ago
    Is anyone interested in amateur digital archiving just destined to losing it all in the long run? You'd think there'd be a market for long life HDDs. Edit: Yes, you could just replace old hardware with new hardware, but it creates so much waste...
    - Multicomp 7y ago
      That's where hopefully Microsoft project silica will make it permanent for all practical purposes digital data storage available. It apparently uses a similar basic process as those mall kiosks that take a volumetric picture of your face and then etch it into a cube of glass? But in this case, they just write the data to the side of a small piece of glass over the course of days and it's basically permanent for all eternity unless something breaks it.
FpUser 7y ago
I have whole bunch of CDs and DVDs (media and data) many way older then 10 years. Some I burned myself and some were bought. Nearly all doing nice and dandy. More then that I have an old box of 3" floppies dated 92-95. I recently decided to check some old code, bought USB floppy drive and I was actually able to read them. Well not all of them but still ...
- ddingus 7y ago
  Thing is they work, until they don't. Nearly all may = massive decline soon as much as may mean you are done for another decade... If you value it, move it now.
  - FpUser 7y ago
    Who says that I did not. All that really matters was backed up years ago in a few places. But I still have the originals and was just curious. There is something about fishing out old CD and putting it into player. As for those floppies I was just curious. I could care less if I did not find that old piece of code I mentioned. I was just curious to look at it and see how my way of thinking did change.
    - ddingus 7y ago
      No worries. That is written for passersby as much as anything else.
Erlich_Bachman 7y ago
For archivers there is now writable M-Disc bluray discs which claim to have longevity of 1000 years. It might be an overexaggeration, but they do use a different technology to record the information, a non-organic layer which is chemically much less active and will inevitably decay slower than a regular CD/DVD. There have even been some accelerated aging tests that were able to somewhat confirm that. If you're into archiving you might want to look that up.
- copperx 7y ago
  After looking that up I've decided it's far cheaper to upload my data to S3 with maximum data center redundancy, fund my AWS account with a big enough balance to be able to pay for a decade of service, and give my keys to people I trust.
  - vbezhenar 7y ago
    You should not put all eggs into one basket. Amazon might ban your account any day and you'll end up with no data. Or your data might lost because of some bug. There's Google coldline, there's online.net C13, they have similar price. Use at least 2 different providers.
    - masklinn 7y ago
      Backblaze B2 as well, they advertise B2 at 0.5c/GB/mo, coldline is advertised at 0.4 (though the official docs quote 0.7c[0]), Glacier is 0.4 to 0.5 depending on the zone. Azure apparently has an "archive" tier for 0.1c, which you can lower even further with long-term high-volume reservation: if you reserve 1PB for 3 years it's 0.081c/GB/mo (using only local redundancy) in the cheapest DCs where it's available (some DCs either don't have that feature at all, or it's way more expensive e.g. 0.1636 in asian DCs where it's available). [0] https://cloud.google.com/storage/pricing#storage-pricing https://cloud.google.com/storage/pricing#storage-pricing
- masklinn 7y ago
  > For archivers there is now writable M-Disc bluray discs which claim to have longevity of 1000 years. ISO cert' states 1000 years mean lifetime (coin flip chance) but only 530 at 95% confidence. And that's at a constant 22C and 50% humidity. And that's solely for the original 4.7GB DVD, not blu-rays. > There have even been some accelerated aging tests that were able to somewhat confirm that. OTOH others (the French National Laboratory of Metrology and Testing) did not and found m-discs to be no better (and possibly worse) than other high-quality discs with an accelerated aging setup of 90C 85% humidity.
fzeroracer 7y ago
This is going to be one of the reasons why CD collecting (whether it's games, music or otherwise) will likely stall. Even games which have been kept in perfect condition can suffer from rot as a result of poor quality control. Cartridges are generally more hardened unless they contain some battery-based saving system and even then a dead battery can be replaced.
hadlock 7y ago
I have the original install disks for BoeingCalc, the precusor to the more famous VisiCalc (Excel) but they're unreadable using consumer grade tech. Probably the only public copy of the software that I'm aware of.
- kevingadd 7y ago
  If you contact the Internet Archive, they might be interested in the disks and might have the necessary hardware to copy them. Try Jason Scott, maybe.
  - drankula3 7y ago
    Yeah, a single iso can be recovered from multiple disks with different damaged sections using software like ddrescue[1]. [1] https://en.wikipedia.org/wiki/Ddrescue https://en.wikipedia.org/wiki/Ddrescue
- acqq 7y ago
  > BoeingCalc, the precusor to the more famous VisiCalc No way: https://en.wikipedia.org/wiki/VisiCalc https://en.wikipedia.org/wiki/VisiCalc VisiCalc: Initial release 1979 https://en.wikipedia.org/wiki/SuperCalc https://en.wikipedia.org/wiki/SuperCalc SuperCalc: Initial release 1980 https://en.wikipedia.org/wiki/Multiplan https://en.wikipedia.org/wiki/Multiplan Multiplan: Initial release 1982 https://en.wikipedia.org/wiki/Lotus_1-2-3 https://en.wikipedia.org/wiki/Lotus_1-2-3 Lotus 1 2 3: Initial release 1983 https://en.wikipedia.org/wiki/Boeing_Calc https://en.wikipedia.org/wiki/Boeing_Calc Boeing Calc: Initial release 1985 (6 years after VisiCalc). As it was released, it could read "standard" VisiCalc and Lotus 1-2-3 documents, according to Computerworld: https://books.google.at/books?id=97eagSKygYsC&pg=PA48&redir_esc=y https://books.google.at/books?id=97eagSKygYsC&pg=PA48&redir_... What's interesting about Boeing Calc is that it allowed storage of its documents "online". But it was surely only for specific local setup, so it was probably "on the local server."
  - hadlock 7y ago
    Ah, I think what I was remembering was that you could have multiple worksheets/"tabs" and run calcs between them, like modern excel. I don't think visicalc supported that level of complexity until later.
jiofih 7y ago
I considered all my old CDs dead (music and PC games - RIP Phantasmagoria, plus MP3 collections) over a decade ago. The majority of them developed white spots or detachment of the disc layer. This has been known since the early 2000s, I’m surprised anyone would collect them without being aware of the risks.
madengr 7y ago
Makes me wonder if there is some probability of error vs information density, irrespective of media. I assume eventually you’ll hit some quantum limit where so much of the data, is error correction that it’s not worthwhile. Maybe instead of going denser, it just needs to go larger.
ohiovr 7y ago
I was tasked with transfering a large collection of cd-r media to hard disk. About 10% was unreadable or corrupted. I have not had a shelved hd go bad yet. I had an amiga 3000 from 92 boot after 25 years of inactivity. Also booted up a classic 8088 that was stored in a basement for about 30 years. No problems with either. Looks like a shelved hd is a cheap and reliable archival option. On a long enough timeline I would expect all cd-r media to be unreadable.
rasz 7y ago
Afaik pressed disk rot is reversible, data is stored on the plastic part (as the name suggests its pressed in by the die), repair would involve sputtering new reflective surface. Plastic itself will probably hold tens of thousands of years. Writable disks are another matter altogether.

137 points by anonymfus 7y ago

The loss of what.cd wasn't difficult because it made piracy hard, it was because the users had created one of the best fully verified lossless cd audio archives on the internet. Much of this content has popped up (or even been extended) on replacement trackers, but it's only a matter of time before the heroic operators of these trackers get forced to shut down too.

Iv 7y ago
I remember the early days of piracy, Napster, Kaza, eDonkey. People were simply sharing all their files. There was pride in having a hard to obtain item, such as TV shows from the 60s, movies not released in DVDs, etc... P2P could have made archivists jobs so much easier. This is a tech that should have been the next iteration of internet that we killed for no good reason. Not even greed, just a refusal to evolve a profitable but obsolete system.
- Aeolun 7y ago
  For me it had more to do with the bootleg sharing programs and the inability to download anything without 1500 different versions of malware also making it onto your computer. For some reason the torrent community doesn’t have quite the same problem.
- Mediterraneo10 7y ago
  While early-millennium filesharing was useful for finding obscure content, the stuff available wasn’t up to the standards of professional archivists. In the library world, there are strict standards for how information is to be ripped, encoded, and tagged, but the content on filesharing networks seldom lived up to solid archival standards. It wasn’t until Oink and What.cd that we got a rigorous archive of audio material.
- vbezhenar 7y ago
  How is P2P killed? I would say P2P matured. There's open source stable torrent software. Just use it, it works.
  - oneeyedpigeon 7y ago
    I think two things happened which effectively destroyed p2p - for the music-sharing case, for me specifically, at least. Streaming and greater official release of bootlegs, demos, etc. For me, there is so much available through these channels, that I don’t even consider p2p as an option.
  - bonoboTP 7y ago
    Torrent is still very centralized. The average user just downloads and seeds torrents made by a few torrent curators on each site. Becoming an uploader on a torrent site is a big privilege/responsibility that few people can take on. Other older platforms like Napster or DC++ (which actually still exists) let people share their own collection at the click of a button. This gave access to a really really long tail of rarities that perhaps only a handful of people were interested in, while torrent sites are mostly mainstream content.
    - dewey 7y ago
      > Becoming an uploader on a torrent site What do you mean by that? You can upload your torrents to any of the big public torrent sites and if it's a private site just being a member allows you to upload already. Public trackers are also all using DHT which makes this not centralized.
      - bonoboTP 7y ago
        On private sites I know (in Hungary), being an uploader is a privilege one must earn, by being around for some time, earning the trust of the admins, having good uplink, learning all the rules and formats so all torrents are consistent etc. I guess perhaps about 0.1% of members upload stuff to these trackers. The situation may be different elsewhere. Context: Torrent is hugely popular in Hungary, basically everyone is a member who uses the internet under ~40 years and is even remotely technical, which means more than a million people (with the single most popular site having over 700k members, with account sharing explicitly allowed). Perhaps not so much for the newest teenager generation (I don't know). Until very recently (and still now) streaming services have been limited contentwise and people just cannot afford to buy all the movies, songs, games etc. with the same ease as in Western Europe / North America.
- javajosh 7y ago
  I used to agree totally, but now I have a more nuanced view. While artificial scarcity offends my hacker sensibility, I also have to admit that this creates a society I like, with talented wealthy celebrities, and a large population of people who make their living in and around the industry. I'm not sure I would like a society where creative acts only happens in home studios and the work is given away for free, and all artists have to have day jobs.
  - Iv 7y ago
    I never said the new model was "give away everything for free" I just said that "Consider every copy needs to be paid even though making a copy is basically free of cost and that this forces us to put a terrible censorship mechanism into place" was not the right approach. Now that there are crowdfunding campaigns and tons of tools for microtransaction, that is even less excusable than when you had the effort to imagine the apparition of these obvious solutions.
    - hakfoo 7y ago
      I feel like we're screwed until we get over the "ownership" model. It means we spend more effort on metadata (What are we allowed to do with it? Who owns it?) than the actual data. It has stood directly in the way of new creative types (sampling in music, for example) and locked away content where the metadata is faulty/lost (orphan works problem) It falls apart badly at scale (if you wanted to get distribution rights for everything on an OSX/Windows installation image, imagine the number of third parties alone, plus the original developers, you'd need to sign off) and then it only really incentivizes work that's commercially appealing, not necessarily the technical best. I dream of a day where people who want to create will be able to get a stipend-- it might cover a modest lifestyle, but ensures they can live the dream of being an artist/musician/programmer without having to clutch tight over every scrap of their output to keep their income streams alive. If we did it through a tax-funded endowment rather than market forces, we could probably finance a lot more creativity that way (think how many touring singers we could finance at 50k per year for the price of one Taylor Swift, or how many programmers we could hire to work om their passion projects in database theory for what is currently being spent on Oracle licenses.)
xouse 7y ago
People have learned from what.cd. Lots of people have the same sense of inevitable doom you do so there's more people who have put thought and effort into being capable of rebuilding quickly if that happens again. Just varroa musica alone has created a giant decentralized metadata archive(and that's not even its main purpose just a neat side effect) that would make the restarting process much easier. The downside is that the post what community is so much smaller. Spotify et al really put a huge damper on the music tracker community, and that makes it less resilient.
- Lammy 7y ago
  > The downside is that the post what community is so much smaller. People were saying the same thing in 2007 when OiNK shut down and What was starting up. The community is not smaller; it's more fragmented, and that's a good thing. Centralization is no better in our societal constructs than it is in our technological ones. I'll trade some immediacy and convenience for longevity any day.
  - xouse 7y ago
    I disagree, the community is both more fragmented and smaller, what had 144k users on its last day vs the 35k of its spiritual successor today. The few music trackers that have sprung up post what.cd after said spiritual successor are all pretty small. The niche genre trackers all have about the same user numbers since then so they cancel out. Then factor in that waffles has been down for some time now and the difference is pretty big. I can only estimate because exact numbers are hard to find, but I'd say peak what era music tracker community total users was in the 350k region and we're probably somewhere around 100k now.
- WilTimSon 7y ago
  Does Spotify really cut that deep into the demographic of hardcore music tracker users? I'd presume those people are chasing rare and lost releases in high quality and Spotify and other streaming services aren't exactly touting rarities. Preserving underground music will always be a job for the fans, not streaming services.
shortandsweet 7y ago
What.cd most certainly wasn't the best. They were really stuck up and got what they deserved. I was amazed at the process just to get access to what they didn't even own. Waffles and oink were so much better.
- serf 7y ago
  > I was amazed at the process just to get access to what they didn't even own. it was that process that helped to curate one of the highest quality audio libraries to have ever been. it's not all that important for most folks to listen to and compare 20 different releases of Bowie's 'Rise and Fall of Ziggy Stardust and the Spiders from Mars', but to those of us that enjoy music cataloguing as a hobby , the loss of what.cd was pretty terrible.
  - deleted 7y ago
    [deleted]
- dewey 7y ago
  By what metric where Waffles and oink better? What.cd was run incredibly professionally, as you can see by the fact that it was active for almost 9 years and nothing happened to any of the people involved.
  - MivLives 7y ago
    They're complaining about the invite system. Either you had to know someone or you had to get lucky that they'd be doing irc interviews that day.
    - dewey 7y ago
      Then their criticism doesn't make much sense as both Waffles and Oink had an invite system. At least What.cd was more approachable as there was an interview system which made it possible for everyone to join as long as they learned some basics about audio formats / transcoding / site rules and had an hour to spend on an interview.
dewey 7y ago
Luckily the internet archive was also working on archiving the content on what.cd even though the results are not public (yet?). https://archive.org/details/whatcdcrawl https://archive.org/details/whatcdcrawl
- ripdog 7y ago
  Getting the metadata is great, but the content won't come back.
pontifier 7y ago
I'm working on something that should be the best place to find any media... unfortunately the VidAngel vs Disney lawsuit has made things a little tougher. Fair use took a big blow there.

sowbug 7y ago

Has anyone looked into whether M Disc is less susceptible to the problems mentioned in TFA?

masklinn 7y ago
They claim to, but it's hard to determine that, and whether other archival-targeted discs (or which there are several e.g. Northern Star DataTresorDisc, MPO Gold, Verbatim Archival) are also resistant to disc rot, because it's unclear whether accelerated aging triggers disc rot in general or only some very specific instances thereof. (apparently defunct) syylex's GMD would most likely be structurally immune to disc rot, but you had / have to really really want to make your data rotproof: single-disc (4.7GB) order was 160€ (>$200 at the time). Not including VAT/GST.

pmoriarty 7y ago

For DVDs and CDs, error correction can be added using dvdisaster: http://dvdisaster.net/ http://dvdisaster.net/

tzs 7y ago
That link does not work, because there is no "A" record in DNS for dvdisaster.net. Here's the latest snapshot in archive.org [1]. It also seems to be available on Sourceforge [2], but that seems to be about 5 years behind what was on the now non-existent dvdisaster.net website. [1] https://web.archive.org/web/20180428070843/http://dvdisaster.net/en/index.html https://web.archive.org/web/20180428070843/http://dvdisaster... [2] https://sourceforge.net/projects/dvdisaster/ https://sourceforge.net/projects/dvdisaster/

userbinator 7y ago

Theoretically, optical media should be easier to recover than magnetic, because the technology to view their surfaces in detail is readily available --- a light microscope is enough to humanly view the actual recorded data on a CD and even a DVD: https://3.14.by/en/read/cd-dvd-microscope https://3.14.by/en/read/cd-dvd-microscope https://www.youtube.com/watch?v=Y8_HCLPdpkI https://www.youtube.com/watch?v=Y8_HCLPdpkI (CD-R) It would be interesting to see what "disc rot" looks like at that level of detail; and if better reading hardware may be able to read it. In particular, pressed discs actually store data in the varying height of the polycarbonate disc, so as long as those pits and lands haven't been damaged, even if the metallisation is gone, the data remains intact; and if there was a way to strip and re-coat that layer, pressed discs could be recovered. For CD-Rs, perhaps the dye has faded to the point where the typical drive (which can't really spin the disc slower than 1x) won't be able to discern the pits from the lands, but if that data can still be seen somehow, it's recoverable.

dehrmann 7y ago
TIL entirely numeric subdomains are legal.
jacobush 7y ago
Oh, so my rotted Laserdiscs can be recovered? :)
BarsMonster 7y ago
I am the author of these photos... If you have a sample of rotten disk - you can send it to me and I'll take a look. Contacts are on the site. Now my hardware is better, CD and DVD are easy to observe without physical damage. I also can see data on BD disk, but it is on the border of what is possible. I also tried to decode data from the image, but it will require more collaboration.

james-skemp 7y ago

I've got a pretty significant collection of PlayStation 1 and 2 games, and every time I see an article on disc rot I look to see when I need to be worried. Has anyone seen a list of when disc-based systems might start showing signs? Something like game consoles and capacitors.

propter_hoc 7y ago
I had a lot of problems with Dreamcast games in particular. I think the black-surfaced PS1 games seem to be a bit better made.
stevenwoo 7y ago
Not exactly what you are asking but I recently sold off my physical CD collection of about 500, after being in my closet for the past 20 years so no temperature/climate extremes, a few of them had signficant value on EBay, but when I tried to use Exact Audio Copy to verify they still could be read, about half no longer matched the online checksums or the original CD rip I had done before putting them in storage. There were three from Denon Japan that had physically deteriorated inside the disk, I could see there was a problem from a visual inspection.
- rixrax 7y ago
  Aha! This must be why them audiophiles prefer to listen to their music using $$$ cd transport from original media! You can hear the difference! On a serious note, I’m in a similar boat, we’re in a middle of the move and it appears that my cd collection is in same predicament. So I’m debating whether to throw them to trash, or keep around (mostly for cover artwork, and to keep my Exact Audio Copy -> flac transfers just a bit more legit (I care of such things)).
  - diamondlovesyou 7y ago
    I dislike CD. My equipment is capable of 192khz/32bit (some only 24bit though), so CD are pretty much trash at 41khz. Better than MP3, solely due to the info MP3 throws away, but not great. Tidal goes higher, to 96khz on master quality where available (the vast number of tracks are not master quality, sadly).

nicholast 7y ago

I think I've heard that exposure to high variance in temperature can contribute. For example by storing CD's in your car.

propter_hoc 7y ago

This is such a huge problem for retro gamers. Any games on optical media, even those that aren't particularly old are at risk. As the article mentions, it's doubly infuriating given how optical media was marketed as more durable than cartridges. Meanwhile, a bunch of my Dreamcast and PS2 games have disc rot, while all my carts are still 100% playable. Besides disc rot, optical drives are also at high risk of the laser dying - this has happened to a PS1, PS2 and Dreamcast of mine. For these reasons, the most serious retro gamers I know are modding their optical drive systems to replace the CD drive with a SD-card based optical drive emulator. Anyway, thank heavens for emulation and the homebrew fan community. Without them, so much of video game history would be lost.

deleted 7y ago
[deleted]
Grazester 7y ago
I have not come across a Dreamcast game with rot. I have some early games which were just pressed badly and the top player in completely pealing off but it was not rot. The Saturn Discs on the other hand I have come across more than a few with rot.

pmarreck 7y ago

What prevents, or tries to prevent, bit rot in standard CPU hard drives/flash drives? Is there some built-in forward error correction at some layer? If not, how could one add some?

Tomte 7y ago
Nothing. There's par2 (and its offsprings MultiPar and QuickPar), there's rsbep, there's pyFileFixity. All have their problems why you probably won't "shield" your files with them (I don't, although I worry).
- planteen 7y ago
  Curious - what are your complaints on these programs? My main one is that they are slow to create archives and rsbep only does error decoding, not erasure. I'm working on an alternative tool that is much faster, but curious if you have any other feedback.
wazoox 7y ago
Most SSDs only guarantee your data for 90 days when off. Even high grade, SLC, professional SSDs. Hard drives fare better, but your best bet for decade-long conservation remains tape.
- ByThyGrace 7y ago
  Is anyone interested in amateur digital archiving just destined to losing it all in the long run? You'd think there'd be a market for long life HDDs. Edit: Yes, you could just replace old hardware with new hardware, but it creates so much waste...
  - Multicomp 7y ago
    That's where hopefully Microsoft project silica will make it permanent for all practical purposes digital data storage available. It apparently uses a similar basic process as those mall kiosks that take a volumetric picture of your face and then etch it into a cube of glass? But in this case, they just write the data to the side of a small piece of glass over the course of days and it's basically permanent for all eternity unless something breaks it.

FpUser 7y ago

I have whole bunch of CDs and DVDs (media and data) many way older then 10 years. Some I burned myself and some were bought. Nearly all doing nice and dandy. More then that I have an old box of 3" floppies dated 92-95. I recently decided to check some old code, bought USB floppy drive and I was actually able to read them. Well not all of them but still ...

ddingus 7y ago
Thing is they work, until they don't. Nearly all may = massive decline soon as much as may mean you are done for another decade... If you value it, move it now.
- FpUser 7y ago
  Who says that I did not. All that really matters was backed up years ago in a few places. But I still have the originals and was just curious. There is something about fishing out old CD and putting it into player. As for those floppies I was just curious. I could care less if I did not find that old piece of code I mentioned. I was just curious to look at it and see how my way of thinking did change.
  - ddingus 7y ago
    No worries. That is written for passersby as much as anything else.

Erlich_Bachman 7y ago

For archivers there is now writable M-Disc bluray discs which claim to have longevity of 1000 years. It might be an overexaggeration, but they do use a different technology to record the information, a non-organic layer which is chemically much less active and will inevitably decay slower than a regular CD/DVD. There have even been some accelerated aging tests that were able to somewhat confirm that. If you're into archiving you might want to look that up.

copperx 7y ago
After looking that up I've decided it's far cheaper to upload my data to S3 with maximum data center redundancy, fund my AWS account with a big enough balance to be able to pay for a decade of service, and give my keys to people I trust.
- vbezhenar 7y ago
  You should not put all eggs into one basket. Amazon might ban your account any day and you'll end up with no data. Or your data might lost because of some bug. There's Google coldline, there's online.net C13, they have similar price. Use at least 2 different providers.
  - masklinn 7y ago
    Backblaze B2 as well, they advertise B2 at 0.5c/GB/mo, coldline is advertised at 0.4 (though the official docs quote 0.7c[0]), Glacier is 0.4 to 0.5 depending on the zone. Azure apparently has an "archive" tier for 0.1c, which you can lower even further with long-term high-volume reservation: if you reserve 1PB for 3 years it's 0.081c/GB/mo (using only local redundancy) in the cheapest DCs where it's available (some DCs either don't have that feature at all, or it's way more expensive e.g. 0.1636 in asian DCs where it's available). [0] https://cloud.google.com/storage/pricing#storage-pricing https://cloud.google.com/storage/pricing#storage-pricing
masklinn 7y ago
> For archivers there is now writable M-Disc bluray discs which claim to have longevity of 1000 years. ISO cert' states 1000 years mean lifetime (coin flip chance) but only 530 at 95% confidence. And that's at a constant 22C and 50% humidity. And that's solely for the original 4.7GB DVD, not blu-rays. > There have even been some accelerated aging tests that were able to somewhat confirm that. OTOH others (the French National Laboratory of Metrology and Testing) did not and found m-discs to be no better (and possibly worse) than other high-quality discs with an accelerated aging setup of 90C 85% humidity.

fzeroracer 7y ago

This is going to be one of the reasons why CD collecting (whether it's games, music or otherwise) will likely stall. Even games which have been kept in perfect condition can suffer from rot as a result of poor quality control. Cartridges are generally more hardened unless they contain some battery-based saving system and even then a dead battery can be replaced.

hadlock 7y ago

I have the original install disks for BoeingCalc, the precusor to the more famous VisiCalc (Excel) but they're unreadable using consumer grade tech. Probably the only public copy of the software that I'm aware of.

kevingadd 7y ago
If you contact the Internet Archive, they might be interested in the disks and might have the necessary hardware to copy them. Try Jason Scott, maybe.
- drankula3 7y ago
  Yeah, a single iso can be recovered from multiple disks with different damaged sections using software like ddrescue[1]. [1] https://en.wikipedia.org/wiki/Ddrescue https://en.wikipedia.org/wiki/Ddrescue
acqq 7y ago
> BoeingCalc, the precusor to the more famous VisiCalc No way: https://en.wikipedia.org/wiki/VisiCalc https://en.wikipedia.org/wiki/VisiCalc VisiCalc: Initial release 1979 https://en.wikipedia.org/wiki/SuperCalc https://en.wikipedia.org/wiki/SuperCalc SuperCalc: Initial release 1980 https://en.wikipedia.org/wiki/Multiplan https://en.wikipedia.org/wiki/Multiplan Multiplan: Initial release 1982 https://en.wikipedia.org/wiki/Lotus_1-2-3 https://en.wikipedia.org/wiki/Lotus_1-2-3 Lotus 1 2 3: Initial release 1983 https://en.wikipedia.org/wiki/Boeing_Calc https://en.wikipedia.org/wiki/Boeing_Calc Boeing Calc: Initial release 1985 (6 years after VisiCalc). As it was released, it could read "standard" VisiCalc and Lotus 1-2-3 documents, according to Computerworld: https://books.google.at/books?id=97eagSKygYsC&pg=PA48&redir_esc=y https://books.google.at/books?id=97eagSKygYsC&pg=PA48&redir_... What's interesting about Boeing Calc is that it allowed storage of its documents "online". But it was surely only for specific local setup, so it was probably "on the local server."
- hadlock 7y ago
  Ah, I think what I was remembering was that you could have multiple worksheets/"tabs" and run calcs between them, like modern excel. I don't think visicalc supported that level of complexity until later.

jiofih 7y ago

I considered all my old CDs dead (music and PC games - RIP Phantasmagoria, plus MP3 collections) over a decade ago. The majority of them developed white spots or detachment of the disc layer. This has been known since the early 2000s, I’m surprised anyone would collect them without being aware of the risks.

madengr 7y ago

Makes me wonder if there is some probability of error vs information density, irrespective of media. I assume eventually you’ll hit some quantum limit where so much of the data, is error correction that it’s not worthwhile. Maybe instead of going denser, it just needs to go larger.

ohiovr 7y ago

I was tasked with transfering a large collection of cd-r media to hard disk. About 10% was unreadable or corrupted. I have not had a shelved hd go bad yet. I had an amiga 3000 from 92 boot after 25 years of inactivity. Also booted up a classic 8088 that was stored in a basement for about 30 years. No problems with either. Looks like a shelved hd is a cheap and reliable archival option. On a long enough timeline I would expect all cd-r media to be unreadable.

rasz 7y ago

Afaik pressed disk rot is reversible, data is stored on the plastic part (as the name suggests its pressed in by the die), repair would involve sputtering new reflective surface. Plastic itself will probably hold tens of thousands of years. Writable disks are another matter altogether.