3 ms·
Lots of fun; while backing up the filesystem prior to wiping and rebuilding it, they ran out of IOPS to do it in a reasonable time frame, so after considering o
by hga 11y ago
Lots of fun; while backing up the filesystem prior to wiping and rebuilding it, they ran out of IOPS to do it in a reasonable time frame, so after considering other options:
One obvious solution would be to use a ramdisk, a virtual disk that actually resides in the memory of a node. The problem was that even our biggest system had 1.5TB of memory while we needed at least 3TB.
As a workaround we created ramdisks on a number of Taito cluster compute nodes, mounted them via iSCSI over the high-speed InfiniBand network to a server and pooled them together to make a sufficiently large filesystem for our needs.
A hack they weren't at all sure would work, but it did nicely.
- powercf 11y agoCouldn't they add 1.5TB of swap to their 1.5TB of memory system and run a ramdisk on that? I'm curious what performance would look like, but given 2-3k IOPS for the on-disk solution, and 20k IOPS for the in-memory I would naively expect at least 11k IOPS for random access, which should have been fast enough without the headache of clustering?
- hga 11y agoMy guess is the "headache of clustering" wasn't a big one for them, they do this for a living, and by that time in the process/downtime, they wanted the job done ASAP.
- ople 11y agoAuthor here: We considered that but as the access pattern was likely pretty much random, the performance would have been terrible. Due to the break we had nearly a 1000 clustered servers sitting idle so it was reasonably quick to do the ramdisk trick.
- icefo 11y agoI'm sorry but I don't understand something. What did you put on that big ramdisk ? The metadata ?
- ople 11y agoWe copied the raw image file of the corrupted metadata filesystem (MDT in Lustre lingo) to the ramdisk. Then we mounted it via loopback and copied the files to tarballs. The bit that was really slow on the spinning disk was reading the millions of files from the metadata FS. The basic process of the file-level backup is documented here: https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.50438207_21638 https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfu...
- garthk 11y agoFor those still not quite getting it: The first copy to RAM was a sequential image copy, thus not bottlenecked on seeks despite spinning platters. The second copy from RAM was a file copy with a lot of random I/O, but not bottlenecked on seeks because it was reading from RAM. Bulk writes tend to be more efficient. They might have made temporary configuration changes to make that end faster, or not if they lacked the appetite for the extra risk.