5 ms·
I don't know the numbers, but the probability of getting 26 corrupted at-rest files through natural causes sounds pretty much like winning the lottery twice on
by hosay123 12y ago
I don't know the numbers, but the probability of getting 26 corrupted at-rest files through natural causes sounds pretty much like winning the lottery twice on the same day you were struck by lightning twice
Checksums wouldn't have fixed this, they'd only alert the user to the fact the damage had already been done, which is exactly what the decompressor did in its own special way.
As another comment points out, error correcting codes are the way to handle this, and its already done in hardware, and probably too expensive to do in software in the general case
- clord 12y agoYou're right, that reported failure rate is way too low, I'd be glad to only have 26 corrupt files. I have one disk with about 300gb of CR2 files (HFS+). Not long ago I did a similar procedure as in the article (except that I also had the files on a ZFS mirror (I was cleaning up the HFS disk and checking for duplicates before formatting) and found 250 files with bit errors compared to the reference. Fortunately only about 1 in 3 of the bit errors resulted in corrupt files. Needless to say I no longer keep anything of consequence on non-checksummed filesystems now.
- Buge 12y agoIt annoys me when someone says "you're right" and then completely disagrees with the first person.
- deleted 12y ago[deleted]
- rtpg 12y agoI think you're discounting the utility of simple checks. Imagine you have weekly backups , but only for the past 26 weeks because of disk space. A file gets corrupted, but you only view it a year afterwards. You effectively have no way of recovering it. Knowing something is wrong can be useful (though being able to fix it is even more useful).
- Sami_Lehtinen 12y agoThat's exactly why I use par2 with all important data when backing it up. Bit corruption? So what?
- pling 12y agoWell I've been using NTFS exclusively since 1996 (NT4) and accumulated 870Gb of data. I've never had a single byte of corruption. Not one, not ever. The data is checksummed about once a month. 26 is a very high number for a deterministic system compared to zero. My recentish MBP purchase (2011) resulted in a single corrupt file copying 1/3 of that volume onto the machine over the LAN. this was 10.7 at the time. That scared me a little. The Linux kit I operate has had no discernable data corruption either in the last 12 years. I know this is an anecdote but over time that's a huge cumulative error.
- mantrax5 12y agoCan we please stop using lottery/lightning/meteor analogies? A modern computer does more operations in a single second, than a small country can buy lottery tickets in a lifetime. Many wildly unlikely things become quite likely with computers.
- TheCondor 12y agoYeah, it's a media problem. There are some missing and interesting pieces of information, was this a laptop? Does it get transferred while off or asleep? What did disk utility tell you about the drive?
- baldfat 12y agoHFS+ has had known file system issues and was suppose to be replaced by ZFS, but that crashed and burned. Apple has been working on NFS+ and adding features. HFS+ and NTFS are long in the tooth and really should have been replaced a long time ago.
- kevinchen 12y agoI can only give you anecdotal evidence, but hopefully it convinces you that hfs+ is the problem. I've been using Macs for 5 years now. I've stored time machine backups spanning two drives and OS releases from snow leopard to mavericks. (Each drive was used for about 2.5 years, and the OSes were installed soon after they became available.) Every 9-15 months, I ask Disk Utility to repair the Time Machine volume. The first time, I lost the entire TM directory and had to format the drive and start over. The second time, it crashed Disk Utility because there were so many errors. The third time, I lost the older ~70% of my backups. The most recent time, the operating system refused to even recognize the disk as an HFS+ volume! Each time, I had the opportunity to format the disk in question and check it for hardware issues. They all passed badblocks and SMART with flying colors. In addition, the volumes were unmounted properly the vast majority of the time (a few instances of human error and power outage) tldr: HFS+ loses your data catastrophically even if you use it properly.
- aroch 12y agoOn the other side of the anecdotal coin, I've been running OSX for a decade now and have in the neighborhood of 100TB of HFS+ formatted media, some in RAID some as single drives. The few times I've catastrophically lost data/drives its been due to the drives physically failing or, in the case of an SSD, a firmware bug. I've lost data to the filesystem (bitrot) but that wouldn't really be prevented by any other FS, they'd just tell me sooner. By the same token, I've experienced much more corruption on my ZFS (~200TB) and btrfs (58TB) arrays over the last few years than I have on HFS for a decade.