7 ms·

I think the title is somewhat unnecessarily editorialized from: > Bug 2208553 - xfs metadata corruption after upgrade to 6.3.3 kernel [NEEDINFO] There are rep

by e12e 3y ago

I think the title is somewhat unnecessarily editorialized from: > Bug 2208553 - xfs metadata corruption after upgrade to 6.3.3 kernel [NEEDINFO] There are reported problems, but also: > I note that all these reported failures seem to be using hardware raid and with stripe (whether either is relevant is unknown). > I am also using XFS (without issue) on a number of other systems with a 6.3 kernel, but none of them are using hardware raid or stripes. Ed: That said - it's quite worrying to see corruption on xfs as the filesystem should be really rock solid by now.

jeffbee 3y ago
So it only affects users who care about their data? Heck of a take.
- ofrzeta 3y ago
  Do you imply that hardware RAID is safer than software RAID?
  - mschuster91 3y ago
    If it's battery-backed, yes.
    - mnw21cam 3y ago
      It shouldn't be safer. However, using a battery-backed write cache in hardware RAID allows the OS to just dump the writes into the cache, get the message that they're safe very quickly, and then move on to the next thing. So, a battery-backed write cache allows it to be faster while still being safe. You can still get the same level of safety without it, and you can get very close to the same speed by switching off all the protections (don't do that), or by logging writes to a separate fast storage device (like a SSD or a battery-backed RAM).
- dralley 3y ago
  I was under the impression that hardware RAID is generally considered unreliable.
  - oneplane 3y ago
    Hardware RAID is pretty much a niche at this stage. And unreliable and unsafe in general. Using it is like having an open fire pit inside your home because it's nice and cozy, and assuming that your house not having burned down yet means it's a good idea. So far the only deployments where I've still seen it somewhat relevant is single-node Windows machines without virtualisation (so the balance between trusting windows with your block storage vs. trusting a RAID controller swings towards the controller), and large proprietary SANs where you don't really have a choice. Everything else is either local software RAID (or ZFS), or a cluster filesystem with block storage support for SAN consumption. And then there is the worst of all worlds: firmware-enabled software RAID like the ones you get with consumer mainboards. As far as I know, that only exists because of windows and the legacy inability to boot from anything except a single-disk magic method on MBR or single ESP partition based booting in UEFI. It's all just crutches for bad systems to not have to do it correctly.
    - justsomehnguy 3y ago
      > So far the only deployments where I've still seen it somewhat relevant I'm glad what all your deployments have a competent sysadmins supervisng them. Sadly, this is not how it works in the real life, so I doubt your experience cover 100% of use cases.
  - Volundr 3y ago
    It's complicated. For consumer level users I'd recommend software RAID because knowing what your getting with hardware RAID is tricky. If your motherboard offers a built-in RAID it's almost certainly crap and your better off disabling it first thing. In general these aren't actually hardware RAIDs anyway, it's actually implemented in the drivers. Low end RAID cards are sometimes better, but mdraid or windows RAID will still probably be more reliable. It's really only expensive enterprise-grade RAID cards that meaningfully compete with software RAID and you'll want to look at the tradeoffs of each.
- e12e 3y ago
  It might not be xfs - it could be the hw raid drivers. I get that the poster wanted to get the word out - but this (for now) only affects opt-in testers; some due diligence is expected. Additionally, that doesn't change the fact that hn frowns on editorializing titles.
  - ericbarrett 3y ago
    > It might not be xfs - it could be the hw raid drivers. It could be that XFS users see the corruption first because of its CRC-verified metadata; you'd expect ZFS to see this quickly too, but another commenter mentioned that OpenZFS isn't even out yet for 6.3. Or perhaps because XFS happens to put its metadata where the corruption occurs. Unclear at this point.
  - rincebrain 3y ago
    At least one person in there reported it with no HW RAID controller in the mix.
  - itvision 3y ago
    > only affects opt-in testers That's a falsehood. Kernel releases on kernel.org are _not_ considered "unstable". That's a myth open source aficionados have to drop ASAP.
    - mroche 3y ago
      The "opt-in testers" is referring to the context of this Bugzilla report against Fedora. The only Fedora users who will have the 6.3 kernel series on their systems are those who willingly and intentionally enable the updates-testing repository. It is not a reference to the Linux ecosystem at large. Putting that aside... this probably should have been caught by upstream before releasing, though. I find it a bit hard to believe no kernel developer, particularly those working on the storage subsystems, don't have some kind of system using XFS within a hardware or software RAID configuration.
      - itvision 3y ago
itvision 3y ago
> I note that all these reported failures seem to be using hardware raid and with stripe (whether either is relevant is unknown). That's not true. Later in the comments there is a person without HW raid who also has been affected. There's nothing indicating that's limited to 6.3.3 exactly. In fact in the same discussion kernel 6.3.1 was reported to having had this issue. Also, a developer chimed it and said it's probably specific to the entire 6.3.x series because patches for 6.3.1-6.3.4 did not introduce any serious XFS specific changes.
- deeter72 3y ago
  Damn here I was hoping I was safe on Opensuse TW with Kernel 6.3.2 Thankfully I should be able to rollback with Opensuse's snapper. This is the key advantage of Opensuse TW in my opinion, The BTRFS filesystem alongside snapper helps maintaining a reliable system while on the bleeding edge.
parasense 3y ago
There is also LVM2 in the picture, but I'm not pointing any fingers. My gut feeling is this smells like some cache type thing waking up to sync/flush buffers and the pointer they follow lands them in the middle of some unwanted region. https://en.wikipedia.org/wiki/Dm-cache https://en.wikipedia.org/wiki/Dm-cache But I didn't see any evidence of caching in the BZ, so it's a potential redherring. The data spanning the corrupted regions was an RSA key, which is fine... some folks might get their kackles up, but meh... I'm not sure why a key file would be cached, unless it's a read cache, and then the normal vfs cache would be where.

by e12e 3y ago

jeffbee 3y ago

So it only affects users who care about their data? Heck of a take.

ofrzeta 3y ago
Do you imply that hardware RAID is safer than software RAID?
- mschuster91 3y ago
  If it's battery-backed, yes.
  - mnw21cam 3y ago
    It shouldn't be safer. However, using a battery-backed write cache in hardware RAID allows the OS to just dump the writes into the cache, get the message that they're safe very quickly, and then move on to the next thing. So, a battery-backed write cache allows it to be faster while still being safe. You can still get the same level of safety without it, and you can get very close to the same speed by switching off all the protections (don't do that), or by logging writes to a separate fast storage device (like a SSD or a battery-backed RAM).
dralley 3y ago
I was under the impression that hardware RAID is generally considered unreliable.
- oneplane 3y ago
  Hardware RAID is pretty much a niche at this stage. And unreliable and unsafe in general. Using it is like having an open fire pit inside your home because it's nice and cozy, and assuming that your house not having burned down yet means it's a good idea. So far the only deployments where I've still seen it somewhat relevant is single-node Windows machines without virtualisation (so the balance between trusting windows with your block storage vs. trusting a RAID controller swings towards the controller), and large proprietary SANs where you don't really have a choice. Everything else is either local software RAID (or ZFS), or a cluster filesystem with block storage support for SAN consumption. And then there is the worst of all worlds: firmware-enabled software RAID like the ones you get with consumer mainboards. As far as I know, that only exists because of windows and the legacy inability to boot from anything except a single-disk magic method on MBR or single ESP partition based booting in UEFI. It's all just crutches for bad systems to not have to do it correctly.
  - justsomehnguy 3y ago
    > So far the only deployments where I've still seen it somewhat relevant I'm glad what all your deployments have a competent sysadmins supervisng them. Sadly, this is not how it works in the real life, so I doubt your experience cover 100% of use cases.
- Volundr 3y ago
  It's complicated. For consumer level users I'd recommend software RAID because knowing what your getting with hardware RAID is tricky. If your motherboard offers a built-in RAID it's almost certainly crap and your better off disabling it first thing. In general these aren't actually hardware RAIDs anyway, it's actually implemented in the drivers. Low end RAID cards are sometimes better, but mdraid or windows RAID will still probably be more reliable. It's really only expensive enterprise-grade RAID cards that meaningfully compete with software RAID and you'll want to look at the tradeoffs of each.
e12e 3y ago
It might not be xfs - it could be the hw raid drivers. I get that the poster wanted to get the word out - but this (for now) only affects opt-in testers; some due diligence is expected. Additionally, that doesn't change the fact that hn frowns on editorializing titles.
- ericbarrett 3y ago
  > It might not be xfs - it could be the hw raid drivers. It could be that XFS users see the corruption first because of its CRC-verified metadata; you'd expect ZFS to see this quickly too, but another commenter mentioned that OpenZFS isn't even out yet for 6.3. Or perhaps because XFS happens to put its metadata where the corruption occurs. Unclear at this point.
- rincebrain 3y ago
  At least one person in there reported it with no HW RAID controller in the mix.
- itvision 3y ago
  > only affects opt-in testers That's a falsehood. Kernel releases on kernel.org are _not_ considered "unstable". That's a myth open source aficionados have to drop ASAP.
  - mroche 3y ago
    The "opt-in testers" is referring to the context of this Bugzilla report against Fedora. The only Fedora users who will have the 6.3 kernel series on their systems are those who willingly and intentionally enable the updates-testing repository. It is not a reference to the Linux ecosystem at large. Putting that aside... this probably should have been caught by upstream before releasing, though. I find it a bit hard to believe no kernel developer, particularly those working on the storage subsystems, don't have some kind of system using XFS within a hardware or software RAID configuration.
    - itvision 3y ago

itvision 3y ago

> I note that all these reported failures seem to be using hardware raid and with stripe (whether either is relevant is unknown). That's not true. Later in the comments there is a person without HW raid who also has been affected. There's nothing indicating that's limited to 6.3.3 exactly. In fact in the same discussion kernel 6.3.1 was reported to having had this issue. Also, a developer chimed it and said it's probably specific to the entire 6.3.x series because patches for 6.3.1-6.3.4 did not introduce any serious XFS specific changes.

deeter72 3y ago
Damn here I was hoping I was safe on Opensuse TW with Kernel 6.3.2 Thankfully I should be able to rollback with Opensuse's snapper. This is the key advantage of Opensuse TW in my opinion, The BTRFS filesystem alongside snapper helps maintaining a reliable system while on the bleeding edge.

parasense 3y ago

There is also LVM2 in the picture, but I'm not pointing any fingers. My gut feeling is this smells like some cache type thing waking up to sync/flush buffers and the pointer they follow lands them in the middle of some unwanted region. https://en.wikipedia.org/wiki/Dm-cache https://en.wikipedia.org/wiki/Dm-cache But I didn't see any evidence of caching in the BZ, so it's a potential redherring. The data spanning the corrupted regions was an RSA key, which is fine... some folks might get their kackles up, but meh... I'm not sure why a key file would be cached, unless it's a read cache, and then the normal vfs cache would be where.