The fancy RAID controller just shouldn't be there - there's no reason to use it unless you're stuck with it. Sometimes you can flash these LSI controllers with

The fancy RAID controller just shouldn't be there - there's no reason to use it unless you're stuck with it. Sometimes you can flash these LSI controllers with HBA/simple-RAID firmware ("IT" or "IR") to fix this. But really, this had little to do with Ceph (other than the ops mistake of rebooting several machines at once without waiting for an all clear, which is just a bad idea all around when you're upgrading a cluster of any kind). It was an interaction between XFS (not used on modern ceph-volume systems), the kernel, and the RAID controller. I find Ceph to actually be a lot more introspectable than "typical" multi disk storage management filesystems like btrfs and zfs. On those, if something goes wrong, you're often left with a corrupted or unmountable filesystem, or worse, kernel panics or errors. It's all a monolith in the kernel and hard to fix anything. On Ceph, you can dig through the layers, and things are split into separate daemons that can be debugged separately. For example, want to see where the data of a CephFS file is stored? Easy: the inode number is the first part of the object name in the data pool. Append the block number after that and you have your object name. You can just fetch that using the rados tool, bypassing CephFS entirely. Want to go deeper? Ask it to hash the object name to a PG ID. Use `ceph pg dump` to figure out what OSDs that pg lives in. Now you know what disks have your data. OSDs broken? Use the objectstore tool to read the data directly without booting an OSD. Obviously you need to learn all this stuff, but the tooling is there and this is way more introspectable than a bunch of magic structures in a filesystem.