3 ms·
I really like the idea behind storing error-correction codes at the application level. Disks should just store and retrieve a chunk of bits and it should be pu
by jrockway 7y ago
I really like the idea behind storing error-correction codes at the application level. Disks should just store and retrieve a chunk of bits and it should be pushed up to the application level to decide whether or not those bits were what we wrote. From there, we can make the right decisions about how durable we really want a file to be.
Instead, people seem to have invested a lot of effort in going the other direction; let the application have less control and let many magic layers in the middle figure out the details. We have had filesystems for decades, which pretend to map names to a list of durable block numbers. Of course, the blocks are not so durable (nor is the database), so people lose their data all the time and act like it's normal.
People have a solution to that problem, though, make the block devices more durable! We can just add software or hardware RAID on top of a bunch of disks, and now suddenly what the filesystem thinks is just one block is now 3 blocks sharded across 3 disks. Some magical curator process then tells me when disks have failed (as though it can detect that with 100% reliability), and I pop in a new drive and magically everything has been repaired. Of course, in reality, the entire disk doesn't just disappear all at once; some blocks just return data with a flipped bit, or all 0s, or an occasional error. My experience in running RAID-1 arrays under Linux (for my workstation) is that I notice some errors in the logs and have to manually figure out which disk contains the correct block (this is why I always use 3 disks) and then dd the block from the good device to the degraded device. By the time the automatic curator shows up to detect and repair this problem for me... all three disks have been RMA'd. It is awful and I'd be surprised to hear a story where someone says "one of my disks failed, I was notified in a timely manner, and my array was repaired when I inserted a new disk". It just doesn't track enough data to actually work, because it ignores the important of error-correction codes. (Don't get me started on RAID-5. That has some parity, but not Reed-Solomon codes, and the parity just makes it unlikely that you could ever manually get the array out of a bad state.)
But even RAID isn't reliable enough, and people don't use it, so of course the disk pretends it is more durable than it actually is. Spinning rust disks have some spare sectors. If their internal heuristics that you cannot tune or read the source code for think something is wrong, it will store some data saying "now block 42 is actually stored at block 2389473" and my RAID controller, filesystem, and application never have to know! Hooray! SSDs are even more egregious; they amplify writes, they garbage collect, they compute proper R-S codes over the data, and have a multi-core CPU running god-knows-what code handling it all.
I think it is all crazy. In my dream world, an SSD would just be a bunch of flash-chips that are not very durable due to the nature of how TLC NAND flash works. Then my application can decide "this is an important photograph, so I'm going to shard it across 8 chips on two different controllers and use error-correction codes that can correct 100% of bit errors". But, that does not exist.
It all makes me mad because when I worked at Google I actually had control, at the application level, over all this. There was the underlying disk API that you could use directly that just stored a block, with the durability of a datacenter disk drive. Then on top of that were filesystems and databases where you had to pick the durability for every operation; you could pick between straight redundancy (write to 3 disks) or whatever level of Reed-Solomon correction you wanted (all these are tradeoffs between read speed, write speed, and actual durability).
Meanwhile in the real world, people build crazy things that still lose their data. ZFS loses data. Cloud block stores lose data. Consumer databases lose data. I know photographers and videographers that have lost all their work because of disk failure... they had multiple disks, but their applications did not uses the disks efficiently in a way that could detect and correct errors. So you come back a year later, and opening your file just says "sorry, error!" and you accept your loss and move on. Someday, I hope this can stop. I never want to watch a YouTube video like this again: https://www.youtube.com/watch?v=gSrnXgAmK8k https://www.youtube.com/watch?v=gSrnXgAmK8k (LTT's Windows-based fileserver with hardware RAID fails randomly and cannot be fixed; all their work gone)
- mntmoss 7y agoI made the observation just a day or two ago to a friend, that when I decided I would invest in a lot of storage, I also had to invest in even more to be able to do backups...and then to perform those backups I have to be able to sync the data effectively, which has some natural scaling limits and introduces ergonomic concerns. After a certain point I threw up my hands and said, "okay, I guess the amount of data I can reasonably keep track of is roughly the amount I can keep synced across multiple devices". So, for the consumer equipment I am using regularly, a few terabytes at most. And that solves the problem in the sense of "I have some form of redundancy", but not "I am confident that it's durable", and definitely not "I am actually curating the data at archival quality". But when I investigated using PAR2 I had another thought, which is that, in a sense, this is a quality-of-service type of issue, and those are always subject to variable standards. And part of my problem is like what you allude to - I can't easily specify how important the data is. If only I could just say, "I only somewhat care about these commercial multi-gigabyte sample libraries, but I care deeply about my password database, so please do the right thing with each." And that's something that - whether it's surfaced to the app or to the user - it really needs a much greater degree of coordination to pull off. As it is, there's a lot of redundant guesswork taking place.
- votepaunchy 7y ago> Then my application can decide "this is an important photograph, so I'm going to shard it across 8 chips on two different controllers and use error-correction codes that can correct 100% of bit errors". But, that does not exist. It’s still not sufficient to backup to a single drive since the device may be lost or destroyed. And once you are storing in the cloud you have all the benefits of distributed redundancy and multi-regional support.
- jrockway 7y agoWell, yes, you lose the data if you lose all the hardware that is storing your data. But that's strictly better than the current case, where RAID controller bugs, SSD firmware bugs, "bit rot", etc. take out your files for absolutely no good reason. The cloud is not made of magic. There are some things that work well, like immutable object stores that are okay with eventual consistency (S3 as an example), but these stores do not work well for things like databases, or logs, or editing video.