3 ms·
I want a multi-device filesystem, offering (typically) the performance of the best device, yet an upper bound on data loss if one device fails. Eg. I could h
by londons_explore 3y ago
I want a multi-device filesystem, offering (typically) the performance of the best device, yet an upper bound on data loss if one device fails.
Eg. I could have an SSD and a slow hard disk. Data is written to the SSD first, and eventually mirrored to the hard disk.
If the SSD fails, the hard disk will still be fully consistent up to the point the mirroring process got to. That mirroring would never be allowed to get more than say 5 minutes behind (writes would be throttled if that ever happened).
This lets you have performance and resilience while paying less money. The scheme can be extended with erasure codes to say 32 drives for a huge database so that you could for example sustain any 3 device failures not losing any fsynced bytes of data, or up to 6 failures but you might lose up to 30 seconds of data.
I think many multi-device filesystems could support this mode, but I don't think any do.
- dymk 3y agoWhat's are the usecases of a filesystem like that? I can't think of a situation where I care that my data is mirrored, but I'm okay with losing 5 minutes of it.
- londons_explore 3y agoYour home desktop PC? I'd be perfectly fine losing 5 minutes of data in case of a disk failure. After all - I can probably recreate it in about 5 minutes. Anyone who does daily backups is effectively saying "I care about my data, but I'm happy to lose up to a days worth if there is a failure".
- dymk 3y agoI suppose I would just throw another SSD in there and mirror with RAID1 (or equivalent), and save myself the headache. SSDs are cheap - A Samsung 980 1TB is $50. Edit: I think you can achieve what you want with current tools, or something pretty close at least, with ZFS. Create a snapshot of your SSD pool every 5 minutes, then stream the delta to an HDD pool in an automated fashion. Delete snapshots more than X hours / days old.
- catiopatio 3y agoThis is exactly how my desktop PC is setup. I use the HDD ZFS pool to maintain effectively infinite non-expiring snapshots of my (single device) SSD ZFS pool. If my SSD fails, I can boot directly from the latest backup on the HDD pool.
- mat_epice 3y agoFrom what I understand, cheap SSDs behave very badly under RAID. For example, Synology actively discourages Samsung EVO SSDs because they cause timeout issues.
- jlokier 3y agoOne of the reasons they say "RAID is not backup" is because the kinds of things that take out a single drive, take out multiple drives together often enough that it surprises people. I've seen more than one person report that they had a RAID NAS at home, thinking their data was safe, only to be horrified one day that every drive failed the same day. An electrical surge from a lightning strike or a faulty power supply, for example, can fry all the drives at once.
- 5e92cb50239222b 3y agoI do something like this with ZFS snapshots and syncoid. It runs every 15 minutes and sends the diff from the latest snapshot onto a second ZFS pool made out of a single HDD (and then updates that snapshot and discards deleted data). Works fine for about a year now.
- bityard 3y agoI'm not sure about erasure codes, but it seems to be possible to do this in a couple different ways already: https://wiki.archlinux.org/title/bcache https://wiki.archlinux.org/title/bcache https://wiki.archlinux.org/title/LVM#Cache https://wiki.archlinux.org/title/LVM#Cache
- saltcured 3y agoYes, the desired behavior is "writeback caching". However, I don't think either of these systems provide a configurable threshold like "max 5 minutes of dirty cache". Instead, you might have to guess as to how large of a cache device you can risk. The larger the cache, the more volatile data might be lost. But, we generally don't have guaranteed writeback throughput either. So you cannot be sure how long a given cache size will take to fully drain to the second storage layer in the worst case. When I have used the LVM cache mechanism, I made a RAID1 mirror of two SSDs for the writeback cache layer, and hoped they were different enough devices so that they wouldn't simultaneously fail from the mirrored traffic patterns. In fact, I used a RAID1 of SSDs to writeback cache a RAID1 of HDDs. But these days, I usually just try to have a large enough SSD to hold my entire data and then use a scheduled backup tool like restic. It copies data to another device but also maintains an archive of multiple snapshots for point-in-time recovery scenarios. I accept up to 24 hours of loss here, using a daily backup procedure. For really important data where I worry about 24 hours, I might manually replicate after making important file updates. E.g. git push to replicate commits to a remote repo, or rsync to replicate specific subdirs to another machine. Edit to add: when worrying about 24 hours, it is also worthwhile to think about the probabilities. The extra effort to maintain a low latency backup might end up costing more time than the unlikely potential day of redundant work after a loss. And how long will the recovery take anyway? Do you keep spare parts on hand or expect to spend time procuring hardware and rebuilding your system after a period of inaction due to a broken system...? The things that I consider too important to wait 24 hours would be non-deterministic things where a recovery method is more difficult than just performing the day's tasks again. E.g. losing recently changed passwords/secrets in an encrypted password store, or losing my notes about other infrequent purchases or bill payments that are in progress. I wouldn't just repeat the steps, but instead have to unravel other recovery procedures for each item.
- RandomBK 3y agoIn the early days of SSD adoption there were 'Hybrid HDD' that effectively acted the way you described, using a small SSD as a read/write cache. Ultimately, the price of SSDs dropped to the level were such solutions filled an ever shrinking niche and they fell out of favor.
- deleted 3y ago[deleted]
- ecnahc515 3y agoNot quite the same, but snapraid can do something sort of like this. It's not real time though, it's scheduled. It's usually combined with something like mergerfs.
- ksec 3y agoSame here except the usage being inside NAS. Having something like 8TB SSD and and two 8TB HDD.
- anotherhue 3y agoZFS SLOG and L2ARC can approximate this.
- martin8412 3y agoRoot on RAM with the data being flushed out to disk?
- viraptor 3y agoMulti-device filesystem is a more fancy solution, but you can do this at block level already. This is where bcache started. Make a virtual device with the slow disks behind some fast ones with bcache, then put any filesystem you want on that device. It should do what you described.
- seized 3y agoDiscord did something that seems to be what you're looking for using md: https://news.ycombinator.com/item?id=32474093 https://news.ycombinator.com/item?id=32474093 https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme-low-latency https://discord.com/blog/how-discord-supercharges-network-di...
- zzo38computer 3y agoI had wanted such a thing too, for reasons you specify and also because I would expect that it would be less loud than multiple spinning hard drives. Either one can fail and if so then the data from the remaining working one can be copied onto a fresh SSD or hard disk, I suppose. Combined with journaling file system can perhaps improve further. With appropriate considerations, you could recover after a power outage. (This might be able to be done without a specialized file system, although a driver and/or controller might be needed for the mirroring and access.)
- tjoff 3y agobacachefs does have some modes that is closely related to your first sdd+hdd scenario. Quite elegant imho, but waiting for it to be mainlined to try it out.