5 ms·
Out of curiosity, why weren't they running the metadata drive in a mirroring raid? If you have PB of data, wouldn't it make sense to spend the ~$100 for a secon
by ajford 11y ago
Out of curiosity, why weren't they running the metadata drive in a mirroring raid? If you have PB of data, wouldn't it make sense to spend the ~$100 for a second 3TB drive to mirror your metadata?
Or was the inode problem not a local disk problem but a problem in the Luster fs? I couldn't quite tell from the article.
- pinewurst 11y agoIt's almost a certainty that the MDS (metadata server) was situated on a mirrored RAID (prob RAID10). I'm guessing that the RAID system itself (software MDRAID or some HW array, DDN or something like a NetApp E-Series) corrupted the data under the FS that the MDS used, which I'm also assuming was XFS. Lustre, for those who don't know it, is a cluster meta-filesystem, with separate metadata and object servers, each sitting on top of host file systems/RAID/storage.
- ople 11y agoThe metadata target (MDT) in the MDS is actually "ldiskfs" which is an enhanced version of ext4. One possibility may be to use ZFS in the future as the support in Lustre seems to be quite stable now. It seems pretty impossible to find out the exact root cause in retrospect as the system was running for a long time without apparent issue. Any ideas are welcome though.
- ople 11y agoIt was filesystem-level corruption in Lustre. The underlying disk arrays and other hardware have comprehensive redundancy.