5 ms·

This is also a story about how complexity is at odds with availability on many levels. Ceph, the fancy RAID controller, and XFS are stacked building blocks to g

This is also a story about how complexity is at odds with availability on many levels. Ceph, the fancy RAID controller, and XFS are stacked building blocks to get more 9's of availability in the model where the enemy is hardware failures, but make it harder to understand the whole system well enough so you know you can operate, troubleshoot & recover it.

bsder 6y ago
> fancy RAID controller I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs. The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to put it in the trash and get something that does. If Ceph doesn't really work, RAID hardware is just adding an extra failure point that adds nothing.
- fulafel 6y ago
  They state "The disks were attached as JBOD devices to a ServeRAID M5210 controller (with a stripe size of 512)". I interpret the stripe size to mean a CEPH stripe size. So the fancy RAID controller can bite you in the ass even if you try to lock its risky functionality away in a closet. Maybe it's hard to buy name-brand server hardware with lots of disk bays and a safely dumb controller?
  - bsder 6y ago
    Ah, good catch, I somehow missed that they had it in "JBOD" mode. As you point out, though, it doesn't seem like it's "really" JBOD mode. It seems like that card is bonding the disks together somehow into a larger logical "stripe". Weird.
    - carlhjerpe 6y ago
      JBOD can also mean "raid 0 with different sized disks". It's a great way to lose all your data!
  - marcan_42 6y ago
    They weren't attached as JBOD: JBOD disks have no stripe size. The firmware changelog talks about VDs, which implies this was a RAID volume. So I'm pretty sure what they did is the usual trick of a 1-disk "RAID0" to get "fake JBOD" mode, which is the only way to do it on crappy MegaRAID cards with old firmware without JBOD passthrough. Except, as they found out, this involves all the mess of RAID code in the controller, leading to bugs and other problems. That controller does supposedly support proper JBOD passthrough mode though, so this was a configuration mistake. Either way, the real lesson here is that RAID cards largely suck, and you want to stay as far away from any RAID features as possible if you're trying to run disks for object storage. For these MegaRAID cards, the best option is to flash IT (HBA) firmware if you can, or IR (HBA with "simple" RAID support) firmware. If you have to use MegaRAID firmware (the full fat RAID thing which uses a completely different driver), then get a version that supports true JBOD passthrough. Only use the RAID0 trick if you really, really have no other option.
- Proven 6y ago
  > I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Have you ever worked with decent enterprise hardware?
- luma 6y ago
  RAID controllers are great if you're using them to do the thing that they do. If you are going to host a software defined storage solution on top, then the thing RAID controllers do for you (abstract the hardware storage devices into a block volume device) becomes counter productive. Most modern array controllers can be switched to HBA mode, which disables the RAID bits entirely and passes SAS/SATA commands through directly to the device. The article doesn't say if this was done, but based on the description of events, it wasn't. Running software defined storage on top of hardware RAID functionality is begging for trouble.
- effie 6y ago
  > I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. 1) inertia 2) simpler way to get up and running for people who are not skilled in CLI or other SW RAID tools (HW RAID controllers have nice BIOS setup) 3) HW RAID BIOS setup always boots and is accessible even if OS can't boot 4) HW RAID stays the same for decades so SW updates are less likely to screw something
- m463 6y ago
  I got a hardware raid controller at home, many years ago. And I had a failure. I couldn't get to anything without buying an identical controller. Swore I'd never do again.
  - axiolite 6y ago
    Linux md-raid supports LSI MegaRAID drives just fine. https://askubuntu.com/questions/1310586/megaraid-lsi-raid-to-mdadm https://askubuntu.com/questions/1310586/megaraid-lsi-raid-to...
marcan_42 6y ago
The fancy RAID controller just shouldn't be there - there's no reason to use it unless you're stuck with it. Sometimes you can flash these LSI controllers with HBA/simple-RAID firmware ("IT" or "IR") to fix this. But really, this had little to do with Ceph (other than the ops mistake of rebooting several machines at once without waiting for an all clear, which is just a bad idea all around when you're upgrading a cluster of any kind). It was an interaction between XFS (not used on modern ceph-volume systems), the kernel, and the RAID controller. I find Ceph to actually be a lot more introspectable than "typical" multi disk storage management filesystems like btrfs and zfs. On those, if something goes wrong, you're often left with a corrupted or unmountable filesystem, or worse, kernel panics or errors. It's all a monolith in the kernel and hard to fix anything. On Ceph, you can dig through the layers, and things are split into separate daemons that can be debugged separately. For example, want to see where the data of a CephFS file is stored? Easy: the inode number is the first part of the object name in the data pool. Append the block number after that and you have your object name. You can just fetch that using the rados tool, bypassing CephFS entirely. Want to go deeper? Ask it to hash the object name to a PG ID. Use `ceph pg dump` to figure out what OSDs that pg lives in. Now you know what disks have your data. OSDs broken? Use the objectstore tool to read the data directly without booting an OSD. Obviously you need to learn all this stuff, but the tooling is there and this is way more introspectable than a bunch of magic structures in a filesystem.

bsder 6y ago

> fancy RAID controller I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Hardware RAID controllers always cause problems--whether due to being a throughput bottleneck or being a pile of bad firmware/hardware bugs. The whole point of Ceph is to use commodity hardware and be reliable. Either Ceph works and things are reliable or it doesn't and you need to put it in the trash and get something that does. If Ceph doesn't really work, RAID hardware is just adding an extra failure point that adds nothing.

fulafel 6y ago
They state "The disks were attached as JBOD devices to a ServeRAID M5210 controller (with a stripe size of 512)". I interpret the stripe size to mean a CEPH stripe size. So the fancy RAID controller can bite you in the ass even if you try to lock its risky functionality away in a closet. Maybe it's hard to buy name-brand server hardware with lots of disk bays and a safely dumb controller?
- bsder 6y ago
  Ah, good catch, I somehow missed that they had it in "JBOD" mode. As you point out, though, it doesn't seem like it's "really" JBOD mode. It seems like that card is bonding the disks together somehow into a larger logical "stripe". Weird.
  - carlhjerpe 6y ago
    JBOD can also mean "raid 0 with different sized disks". It's a great way to lose all your data!
- marcan_42 6y ago
  They weren't attached as JBOD: JBOD disks have no stripe size. The firmware changelog talks about VDs, which implies this was a RAID volume. So I'm pretty sure what they did is the usual trick of a 1-disk "RAID0" to get "fake JBOD" mode, which is the only way to do it on crappy MegaRAID cards with old firmware without JBOD passthrough. Except, as they found out, this involves all the mess of RAID code in the controller, leading to bugs and other problems. That controller does supposedly support proper JBOD passthrough mode though, so this was a configuration mistake. Either way, the real lesson here is that RAID cards largely suck, and you want to stay as far away from any RAID features as possible if you're trying to run disks for object storage. For these MegaRAID cards, the best option is to flash IT (HBA) firmware if you can, or IR (HBA with "simple" RAID support) firmware. If you have to use MegaRAID firmware (the full fat RAID thing which uses a completely different driver), then get a version that supports true JBOD passthrough. Only use the RAID0 trick if you really, really have no other option.
Proven 6y ago
> I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. Have you ever worked with decent enterprise hardware?
luma 6y ago
RAID controllers are great if you're using them to do the thing that they do. If you are going to host a software defined storage solution on top, then the thing RAID controllers do for you (abstract the hardware storage devices into a block volume device) becomes counter productive. Most modern array controllers can be switched to HBA mode, which disables the RAID bits entirely and passes SAS/SATA commands through directly to the device. The article doesn't say if this was done, but based on the description of events, it wasn't. Running software defined storage on top of hardware RAID functionality is begging for trouble.
effie 6y ago
> I simply don't understand why people use hardware RAID controllers. Anything above JBOD is asking for disaster. 1) inertia 2) simpler way to get up and running for people who are not skilled in CLI or other SW RAID tools (HW RAID controllers have nice BIOS setup) 3) HW RAID BIOS setup always boots and is accessible even if OS can't boot 4) HW RAID stays the same for decades so SW updates are less likely to screw something
m463 6y ago
I got a hardware raid controller at home, many years ago. And I had a failure. I couldn't get to anything without buying an identical controller. Swore I'd never do again.
- axiolite 6y ago
  Linux md-raid supports LSI MegaRAID drives just fine. https://askubuntu.com/questions/1310586/megaraid-lsi-raid-to-mdadm https://askubuntu.com/questions/1310586/megaraid-lsi-raid-to...

marcan_42 6y ago

The fancy RAID controller just shouldn't be there - there's no reason to use it unless you're stuck with it. Sometimes you can flash these LSI controllers with HBA/simple-RAID firmware ("IT" or "IR") to fix this. But really, this had little to do with Ceph (other than the ops mistake of rebooting several machines at once without waiting for an all clear, which is just a bad idea all around when you're upgrading a cluster of any kind). It was an interaction between XFS (not used on modern ceph-volume systems), the kernel, and the RAID controller. I find Ceph to actually be a lot more introspectable than "typical" multi disk storage management filesystems like btrfs and zfs. On those, if something goes wrong, you're often left with a corrupted or unmountable filesystem, or worse, kernel panics or errors. It's all a monolith in the kernel and hard to fix anything. On Ceph, you can dig through the layers, and things are split into separate daemons that can be debugged separately. For example, want to see where the data of a CephFS file is stored? Easy: the inode number is the first part of the object name in the data pool. Append the block number after that and you have your object name. You can just fetch that using the rados tool, bypassing CephFS entirely. Want to go deeper? Ask it to hash the object name to a PG ID. Use `ceph pg dump` to figure out what OSDs that pg lives in. Now you know what disks have your data. OSDs broken? Use the objectstore tool to read the data directly without booting an OSD. Obviously you need to learn all this stuff, but the tooling is there and this is way more introspectable than a bunch of magic structures in a filesystem.