9 ms·
I wound up causing a major outage of my services and destroying my home dir
- teddyh 5y agoTLDR: They got an error from the RAID1 array, but they re-added the broken drive to the array. Also, they did’t have backups. Then, they did fsck -y on the filesystems on the array incorporating the broken drive. What they probably should have done first is remove the broken drive from the array again.
- YATA0 5y agoEvery generation rediscovers why ZFS exists, and why it was designed the way it is.
- ziml77 5y agoOne thing I like the idea of, that ZFS doesn't have, is redundancy broken down to a smaller size than the the whole disk so I don't need matched disk sizes and can easily expand a pool. Btrfs does this. Difficulty aside, I'm wondering if it would be possible for ZFS to implement that as an option without giving up other features.
- magicalhippo 5y agoYou can do it yourself at the cost of (a lot of) performance. Make say 4TB partitions on each drive, and make vdevs across the partitions. Then you can upgrade by putting the partitions on different drives etc. Of course must then be careful not to put two partitions of the same vdev on the same disk. The major downside is that ZFS isn't aware that the underlying devices are the same, so IO which looks nicely scheduled as far as ZFS is concerned might cause massive random IO on the underlying disks. That said, if one could tell ZFS about which zfs-devices share physical hardware, so the IO scheduler would be aware, then that might be a viable strategy. Might be handy in case you have multiple iSCSIs sharing a 10G link as well.
- ziml77 5y agoHuh. I hadn't even considered that it would be possible to assign partitions to the vdevs instead of the entirety of the drive. Though I guess it does make sense that you can, otherwise you wouldn't be able to use a drive participating in a vdev as a boot device. I don't need the feature that badly though. It's just something that's interested me about Unraid (and most consumer NAS devices actually), especially since I'm running 4 2TB drives and 2 3TB drives as a single RAIDZ2 vdev in my home TrueNAS setup.
- magicalhippo 5y ago> I hadn't even considered that it would be possible to assign partitions to the vdevs instead of the entirety of the drive. You can even use files[1]! Mostly available for testing though. It would be nice with a bit more flexibility with regards to the underlying storage, hence this concept. I tested it in a VM and it works fine, just slow when the underlying disks gets trashed. Mostly noticeable on heavy writes and scrubs. Should work fine on SSDs though, and might possibly even be a boon to very fast NVMe drives (increasing concurrency). [1]: https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7.html https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7...
- theamk 5y ago.. and that's how people end up with ZFS. My story was cheap SATA cables (or was it cheap power supplies? I never found out) introducing silent corruption and breaking decades of archived files.
- db48x 5y agoIt’s pretty remarkable actually just how bad that failure mode was. How did mdadm manage to cause such havoc after the broken device was reattached?
- zootboy 5y agoMDADM RAID1 does not seem to have the ability to recover from silent data corruption. In the standard two-disk RAID1, it's actually impossible to recover, since it doesn't know what copy is the uncorrupted one. With three disk RAID1 it would theoretically be possible to use a majority vote, but I don't believe MDADM has such code. Btrfs raid1, on the other hand, uses the checksums of the data stored in its metadata trees to validate which copy of the data is correct and repair the corrupted copy. 3-disk makes this even more robust, as you get an exponential growth of data-metadata pairs to pull from. If any one pair matches, that data can be assumed to be correct.
- butlerm 5y agoNot sure how well this works out in practice, but if one drive returns a read error for a sector and the other returns the corresponding sector without an error, it should be a simple matter to trust the latter. Of course that assumes that storage corruption occurred after uncorrupted data was written to disk. Modern drives have rather large per physical sector checksums to detect the occurrence of these errors and presumably retry a couple times before returning an I/O error if configured correctly. In addition a good I/O bus should have error detection checksums for data in transit, and good hardware should have ECC RAM as well.
- zootboy 5y ago
- hddherman 5y ago> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway, ZFS is a great choice either way. Definitely implement snapshots and a regular backup habit with plain zfs send/recv or syncoid.
- CaliforniaKarl 5y agoI agree with getting SSDs from different vendors, for the following reason: I once had a FreeNAS system fail. It was using two (mirrored) SSDs to hold the OS. One SSD failed; while I was procuring a replacement, the second failed. I later found that both SSDs were from the same batch, which (I think) is why both failures happened so close together. Purchasing same-size SSDs from different vendors (with similar product lines) helps ensure you're using SSDs that should hopefully fail at different times.
- hulitu 5y agoThis was common knowledge on the internet before the "modern web" era. Never use in a RAID disks from the same vendor. When you have to use disks from the same vendor try hard to get different type/ batches. And backup.
- codechad 5y ago> Never use in a RAID disks from the same vendor. Wrong. The advice is to use the same vendor/model but from a different production run (batch).
- wheybags 5y agoI very intentionally bought two Samsung ssd from different countries, so I would get different batches. They show up on my door. Same batch, only a few hundreds apart in their serial number :(
- kova12 5y agothere's so many layers that could have caught the issue, even if you don't use ZFS... What's the point of splurging on EPYC with 256 GB of RAM if you don't want to spend a penny on resilience
- halo37253 5y agoIt smells of Web Dev doing Web Dev things. It works now, it'll be fine. Sure a Docker or VM can be easy to rebuild from scratch. But a Database is not. I learned years ago when a failed HDD took 2TB of data I will never get back.
- imwillofficial 5y agoWelcome to the club. If you haven’t taken down prod at least once you’re not trying hard enough
- fulafel 5y agoTLDR: Lost a lot of diy hosted infra from storage problems. Conclusion: do backups.
- jsiepkes 5y ago> While I think Oracle and the Illumos and OpenZFS contributors should come together to relicense the ZFS codebase under MPLv2 to solve the GPL compatibility problem For starters Oracle would need to relicense their work. Which is unlikely to happen any time soon. If they wanted to relicense it they would have done it by now. All OpenZFS contributers would also need to relicense their code. If I recall correctly the OpenZFS codebase consists for close to 50% out of non-Oracle code. A lot of this code might be contributed by companies where the people who did the work no longer work there. Meaning there is no-one in those companies to get legal to sign-off on it. Short version: Don't hold your breath.
- pabs3 5y agoAll Oracle needs to do is release CDDL 2.0 that says "At your option, you may choose to relicense CDDL 1.0 licensed code under any of the MPLv2, LGPLv2+, GPLv2+, AGPLv3+ licenses" and then anyone can relicense any CDDL 1.0 code from any copyright holder under those licenses, since CDDL 1.0 contains a license upgrade clause. That would solve the situation for OpenZFS contributor's code too and also Oracles code and also dtrace/Solaris/etc code in general.
- jsiepkes 5y agoThat would have consequences reaching far beyond ZFS. A blastzone which no one will be able to oversee. I would therefor say it is extremely unlikely any company (let alone Oracle) would ever sign-off on such a thing.
- pabs3 5y agoPS: I didn't think CDDL was used much outside of Solaris (now proprietary again and the forks of it aren't active I thought) and OpenZFS. Are there other significant CDDL licensed codebases?
- jsiepkes 5y ago> Solaris (now proprietary again and the forks of it aren't active I thought) The Solaris forks (based on Illumos[1]) such as SmartOS[2] are pretty active! > Are there other significant CDDL licensed codebases? A lot of the Java stuff was also CDDL licensed. Though some of that (larger projects such as OpenJDK) were already dual licensed with "GPLv2 + classpath exception". I also doubt it was used much outside of Sun. But even it's just the Sun stuff I still think no corporation would ever dare to make such a license update. [1] https://www.illumos.org/ https://www.illumos.org/ [2] https://wiki.smartos.org/ https://wiki.smartos.org/
- Yerui 5y agoUsing ECC ram on my nas.for ages If I have an amd CPU in my dev server that would also run with ECC. But I also run zfs with snapshots.
- throwaway984393 5y agoYou'll often find angry greybeards on here griping about how you shouldn't do X or Y. Don't run your own mail, don't write your own crypto, don't run K8s. And you'll also find greenhorns talking about how they've been doing X for 10 years and never had a problem. The thing is, both perspectives are valid. It's true that you can sometimes do really complicated things yourself and it can work out fine. And you can also get burned really easily. So the reader needs to figure out for themselves what the risk is and if it's worth it. But there are things we should all do that are just never easy to do right, like.... backups. There's just no cheap and easy and good way. If you want real off-site backups, you're going to have to either pay a small fortune for a managed service (either in subscription fees or network egress fees or both), or physically carry a bunch of drives or tapes somewhere else weekly/monthly. And of course you also have to restore regularly or the whole thing could be a waste. It's just a pain.
- teddyh 5y agoYou seem to conflate backups with off-site backups, and concluding that off-site backups are hard. However, nothing in this story would require off-site backups. A separate backup server on-site would have been fine here. Of course, off-site backups are good, but would not have been actually needed in this case.
- christophilus 5y agoWas going to say the same. Anyway, off-site backups aren’t that expensive, if you go with Backblaze or Wasabi. $5/TB/mo.
- jewel 5y agoIf you have a lot of data then a cheaper choice would be to put a raspberry pi at your mom's house, add a few external drives, forward a port, and send backups with restic.
- hatware 5y ago> don't write your own crypto I'm no greybeard, but this one IS important.
- Ansil849 5y agoThere are a lot of comments here about how the author should've used ZFS, or how well now they know about ZFS, but the post addresses this: > The other big elephant in the room, of course, is “why weren’t you using ZFS in the first place”. While it is true that Alpine has supported ZFS for years, I’ve been hesitant to use it due to the CDDL licensing. In other words, I chose the mantra instilled in me about GPL compatibility since the days when I was using GNU/Linux over pragmatism. It's not as if the author wasn't aware of ZFS, but had issues with the license.
- MayeulC 5y agoI'm pretty sure btrfs would detect these issues as well. It has certainly saved me before thanks to checksumming.
- bullen 5y agoI put critical/unique data on Intel X-25E... 64GB 65nm SLC from 2011 with 100.000 writes per bit, peak SSD!
- hyperman1 5y agoOne part I noticed: The backup strategy. If ariadne fucks up this bad, mere mortals have no chance. I understand her very well, btw, as it took a minor data loss to teach me it will happen to all of us. I just got lucky. But consider the situation as an end user: zillions of options, but no clear strategy. Simply choosing what to back up and how many times is expert level IT. Choosing a software package and storage location is headache inducing for expert level IT. Validating long term all the effort will bear fruit on the worst day of your computers life brings enterprises to their knees.
- Santosh83 5y agoNon-tech end users are expected to backup everything to the Cloud. Not just backup but even store working copies. In fact non-technical users can, at most, plug in an external HDD/SSD and backup their files to it, but unless they take pains to keep a 2nd copy (and both copies in sync), it is no better than having your files on the system drive. Anything beyond this is too technical at the moment for laymen to use. I don't know why that is so. Many programs are more difficult/technical to use than they have to be. It's like some kind of unconscious gatekeeping.
- halo37253 5y agoHonestly I hate builds like this. I'd rock ESXI as hypervisor, and Truenas as the Storage environment. Passthrough the SSD's and any Disk controller to TrueNas. I always recommend HDD Array to backup your primary SSD array. This could have all been avoided with ZFS, along with future worry of bitrot. Then you can run your Primary Linux VM for Docker, and another other standalone Linux VM's you may need. You'll want to keep Truenas on a datastore connected to ESXI, but the rest of the storage back be kept on Truenas and fed back into esxi. Not only keeps your system portable for future upgrade paths, but keeps your system reliable. Opensource is not always the best route to take.