4 ms·
I would like to see filesystems benchmarked for robustness. Specifically, robustness to everything around them not performing as required. For example, imagi
by londons_explore 4y ago
I would like to see filesystems benchmarked for robustness.
Specifically, robustness to everything around them not performing as required. For example, imagine an evil SSD which had a 1% chance of rolling a sector back to a previous version, a 1% chance of saying a write failed when it didn't, a 1% chance of writing data to the wrong sector number, a 1% chance of flipping some bits in the written data, and a 1% chance of disconnecting and reconnecting a few seconds later.
Real SSD's have bugs that make them do all of these things.
Given this evil SSD, I want to know how long the filesystem can keep going serving the users usecase.
- colttt 4y agoKent use md-faulty to test such a scenario, you can also see here the test-CI https://evilpiepirate.org/~testdashboard/ci https://evilpiepirate.org/~testdashboard/ci
- 112233 4y agoPlease, where can i read more about this. I remember bricking OCZ drive by setting ATA password, as was fashionable to do back then, but 1% writes going to wrong sector - what are these drives, fake sd cards from aliexpress? Like, which manufacturer goes, like "tests show we cannot write more that 400kB without corrupting drive, let us ship this!" ?
- londons_explore 4y agoWrites going to the wrong sector are usually wear levelling algorithms gone wrong. Specifically, it normally means the information about which logical sector maps to which physical sector was updated not in sync with the actual writing of the data. This is a common performance 'trick' - by delaying and aggregating these bookkeeping writes, and taking them off the critical path, you avoid writing so much data and the user sees lower latency. However, if something like a power failure or firmware crash happens, and the bookkeeping writes never happen, then the end result that the user sees after a reboot is their data written to the wrong sector.
- jlokier 4y ago> Like, which manufacturer goes, like "tests show we cannot write more that 400kB without corrupting drive, let us ship this!" ? According to https://www.sqlite.org/howtocorrupt.html https://www.sqlite.org/howtocorrupt.html there are such drives: 4.2. Fake capacity USB sticks There are many fraudulent USB sticks in circulation that report to have a high capacity (ex: 8GB) but are really only capable of storing a much smaller amount (ex: 1GB). Attempts to write on these devices will often result in unrelated files being overwritten. Any use of a fraudulent flash memory device can easily lead to database corruption, therefore. Internet searches such as "fake capacity usb" will turn up lots of disturbing information about this problem. Bit flips amd overwriting wrong sectors from unrelated files being written are also mentioned. You might think this sort of thing is just cheap USB flash drives, bit I've been told about NVMe SSDs violating their guarantees and causing very strange corruption patterns too. Unfortunately when the cause is a bug in the storage's device own algorithms, the rare corruption event is not necessarily limited to a few bytes or just 1 sector here or there, nor to just the sectors being written. I don't know how prevalent any of these things are really. The sqlite.org says "most" consumer HDDs lie about commiting data to the platter before reporting they've done so, but when I worked with ext3 barriers back in the mid 2000s, the HDDs I tested had timing consistent with flushing write cache correctly, and turning off barriers did in fact lead to observable filesystem corruption on power loss, which was prevented by turning on barriers. The barriers were so important they made the difference between embedded devices that could be reliably power cycled, vs those which didn't reliably recover on boot.
- Dylan16807 4y ago> Fake capacity USB sticks Those drives have a sudden flip from 0% corruption to 95-100% corruption when you hit their limits. I wouldn't count that as the same thing. And you can't reasonably expect anything to work on those. > The sqlite.org says "most" consumer HDDs lie about commiting data to the platter before reporting they've done so, but when I worked with ext3 barriers back in the mid 2000s, the HDDs I tested had timing consistent with flushing write cache correctly Losing a burst of writes every once in a while also manifests extremely differently from steady 1% loss and needs to be handled in a very different way. And if it's at power loss it might be as simple as rolling back the last checkpoint during mount if verification fails.
- crest 4y agoA 1% error rates for corrupting other blocks is prohibitively. A file system would have do extensive forward error correction in addition to checksumming to have a chance to work with this. It would also have to perform a lot of background scrubbing to stay ahead of the rot. While interesting to model and maybe even relevant as a research problem given the steadily worsening bandwidth to capacity ratio of affordable bulk storage I don't expect there are many users willing to accept the overhead required to come even close to a usable file system an a device as bad as the one you described.
- comex 4y agoWell, it’s sort of redundant. According to [1], the raw bit error rate of the flash memory inside today’s SSDs is already in the 0.1%-1% range. And so the controllers inside the SSDs already do forward error correction, more efficiently than the host CPU could do it since they have dedicated hardware for it. Adding another layer of error correction at the filesystem level could help with some of the remaining failure modes, but you would still have to worry about RAM bitflips after the data has already been read into RAM and validated. [1] https://ieeexplore.ieee.org/document/9251942 https://ieeexplore.ieee.org/document/9251942
- simcop2387 4y agoZFS will do this. Give it a RAIDz-{1..3} setup and you've got the FEC/Parity calculations that happen. Every read has it's checksum checked, and when reading if it finds issues it'll start resilvering them asap. You are of course right in that it will eventually start getting to worse and worse performance as it's having to do much more rewriting and full on scrubbing if there are constant amounts of errors happening but it can generally handle things pretty well.
- Dylan16807 4y agoI don't know... Let's say you have 6 drives in raidz2. If you have a 1% silent failure chance per block, then writing a set of 6 blocks has a 0.002% silent failure rate. And ZFS doesn't immediately verify writes, so it won't try again. If that's applied to 4KB blocks, then we have a 0.002% failure rate per 16KB of data. It will take about 36 thousand sets of blocks to reach 50% odds of losing data, which is only half a gigabyte. If we look at the larger block ZFS uses internally then it's a handful of gigabytes. And that's without even adding the feature where writing one block will corrupt other blocks.
- toxik 4y agoWith this level of adversarial problems, you'd better formulate your whole IO stack as a two-player minimax game.
- throw0101a 4y ago> For example, imagine an evil SSD which had a 1% chance of rolling a sector back to a previous version, a 1% chance of saying a write failed when it didn't, a 1% chance of writing data to the wrong sector number, a 1% chance of flipping some bits in the written data, and a 1% chance of disconnecting and reconnecting a few seconds later. There are stories from the ZFS folks of dealing with these issues and things ran just fine. While not directly involved with ZFS development (IIRC), Bryan Cantrill was very 'ZFS-adjacent' since he used Solaris-based systems for a lot of his career, and he has several rants about firmware that you can find online. A video that went viral many years ago, with Cantrill and Brendan Gregg, is "Shouting in the Datacenter": * https://www.youtube.com/watch?v=tDacjrSCeq4 https://www.youtube.com/watch?v=tDacjrSCeq4 * https://www.youtube.com/watch?v=lMPozJFC8g0 https://www.youtube.com/watch?v=lMPozJFC8g0 (making of)
- PhilipRoman 4y agoI have basically the opposite problem. I've been looking for a filesystem that maximizes performance (and minimizes actual disk writes) at the cost of reliability. As long as it loses all my data less than once a week, I can live with it.
- dralley 4y agoCue MongoDB memes
- ilyt 4y agoProbably just using it for cache of some kind
- magicalhippo 4y agoTongue-in-cheek solution: use a ramdisk[1] for dm-writecache in writeback mode[2]? [1]: https://www.kernel.org/doc/Documentation/blockdev/ramdisk.txt https://www.kernel.org/doc/Documentation/blockdev/ramdisk.tx... [2]: https://blog.delouw.ch/2020/01/29/using-lvm-cache-for-storage-tiering/ https://blog.delouw.ch/2020/01/29/using-lvm-cache-for-storag...
- bionade24 4y agoNot sure if this is feasible, but have you considered dumping binary on the raw disk like done with tapes?
- crabbone 4y agoYou could even use partitions as files. You could only have 128 files, but maybe that's enough for OP?
- viraptor 4y agoHave you tried allowing ext4 to ignore all safety? data=writeback, barrier=0, bump up dirty_ratio, tune ^has_journal, maybe disable flushes with https://github.com/stewartsmith/libeatmydata https://github.com/stewartsmith/libeatmydata
- rwmj 4y agonbdkit memory 10G --filter=error error-rate=1% ... and then nbd-loop-mount that as a block device and create your filesystem on top. Notes: We're working on making a ublk interface to nbdkit plugins so the loop mount wouldn't be needed. There are actually better ways to use the error filter, such as triggering it from a file, see the manual: https://www.libguestfs.org/nbdkit-error-filter.1.html https://www.libguestfs.org/nbdkit-error-filter.1.html It's an interesting idea to have an "evil" filter that flips bits at random. I might write that!
- antongribok 4y agoHow does this compare with dm-flakey [0] ? [0]: https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-flakey.html https://www.kernel.org/doc/html/latest/admin-guide/device-ma...
- rwmj 4y agodm-flakey runs inside the kernel so is going to be faster. With nbdkit + NBD loop mounting there's a bit of overhead going through a Unix socket, which is avoided by using ublk + nbdkit plugins, but you still have to go out to userspace for each device access. The advantage is you can write plugins as regular code in lots of different programming languages, with a stable API and ABI: https://www.libguestfs.org/nbdkit.1.html https://www.libguestfs.org/nbdkit.1.html So it's a lot easier to experiment or write plugins for odd devices. Also our sparse RAM disk supports huge disks (up to 16 exabytes IIRC, 2^63-1 bytes whatever that is).
- Dwedit 4y agoSuddenly I'm reminded of the time someone made a Bad Internet simulator (causing packet loss or other problems) and named the program "Comcast".
- klabb3 4y agoI love it.
- seized 4y agoI've had ZFS pools survive (at different times over the span of years): - A RAIDz1 (RAID5) pool with a second disk start failing while rebuilding from an earlier disk failure (data was fine) - A water to air CPU cooler leaking, CPU overheated and the water shorted and killed the HBA running a pool (data was fine) - An SFF-8088 cable half plugged in for months, pool would sometimes hiccup, throw off errors, take a while to list files, but worked fine after plugging it in properly (data was fine after) Then the usual disk failures which are a non-event with ZFS.
- gigatexal 4y agoThis is why I always opt for ZFS.
- ilyt 4y agoI recovered from 3 disk RAID 6 failure (which itself was organization failure driven...) in linux's mdadm... ddrescue to the rescue, I guess I got "lucky" the bad blocks didn't happen in same place on all drives (one died, other started returning bad blocks), but chance for that to happen are infinitesly small in the first place So shrug
- radlad 4y agoYes, I also had a "1 disk failed, second disk failed during rebuild" event (like the parent, not your story) with mdadm & RAID 6 with no issues. People seem to love ZFS but I had no issues running mdadm. I'm now running a ZFS pool and so far it's been more work (and things to learn), requires a lot more RAM, and the benefits are... escaping me.
- jeffbee 4y ago[flagged]
- sangnoir 4y agoIsn't Linux the most deployed OS in industry (by practitioners)? Are the Linux hyperscalers hiding their FS secret-sauce, or perhaps the "aesthetic ideal" filesystems available to Linux good enough?
- jeffbee 4y agoI imagine the hyperscalers are all handling integrity at a higher level where individual filesystems on single hosts are irrelevant to the outcome. In such applications, any old filesystem will do. For people who do not have application-level integrity, the systems that offer robustness in the face of imperfect storage devices are sold by companies like NetApp, which a lot of people would sneer at but they've done the math.
- Datagenerator 4y agoSeen NetApp's boot messages, it's FreeBSD under the hood
- amluto 4y agoISTM one could design a filesystem as a Byzantine-fault-tolerant distributed system that happens to have many nodes (disks) partially sharing hardware (CPU, memory, etc). The result would not look that much like RAID, but would look quite a bit like Ceph and its relatives. Bonus points for making the result efficiently support multiple nodes, each with multiple disks.
- deathanatos 4y agoOn the whole, I'm not sure that a FS can work around such a byzantine drive, at least not if its the only such drive in the system. I'd rather FSes not try to pave over these: these disks are faulty, and we need to demand, with our wallets, better quality hardware. > 1% chance of disconnecting and reconnecting a few seconds later I actually have such an SSD. It's unusable, when it is in that state. The FS doesn't corrupt the data, but it's hard for the OS to make forward progress, and obviously a lot of writes fail at the application level. (It's a shitty USB implementation on the disk: it disconnects if it's on a USB-3 capable port, and too much transfer occurs. It's USB-2, though; connecting it to a USB-2-only port makes it work just fine.)
- mprovost 4y agoAt the point where it disappears for seconds it's a distributed system not an attached disk. At this point you have to start applying the CAP theorem. At least in Unix the assumption is that disks are always attached (and reliable...) so write errors don't typically bubble up to the application layer. This is why losing an NFS mount typically just hangs the system until it recovers.
- deathanatos 4y ago> At least in Unix the assumption is that disks are always attached (and reliable...) I want to say a physical disk being yanked from the system (essentially what was happening, as far as the OS could tell) does cause I/O errors in Linux? I could be wrong though, this isn't exactly something I try to exercise. As for it being a distributed system … I suppose? But that's what an FS's log is for: when the drive reconnects, there will either be a pending WAL entry, or not. If there is, the write can be persisted, otherwise, it is lost. But consistency should still happen. Now, an app might not be ready for that, but that's an app bug. But it can always happen that the power goes out, which in my situation is equivalent to a disk yank. There's also small children, the SO tripping over a cable, etc. But these are different failure modes from some of what the above post listed, such as disks undoing acknowledged writes, or lying about having persisted the write. (Some of the examples are byzantine, some are not.)
- wkat4242 4y agoIf you have an evil SSD which screws up 5% (1 in 20) of your writes, well... You don't need a better FS. You need a new disk right now. This is not the kind of frequency you need to protect against. You just need to detect it and replace the drive ASAP.