7 ms·

XFS metadata corruption after upgrade to 6.3.3 kernel

130 points by itvision 3y ago

chasil 3y ago
Interesting that ElRepo is on 6.3.4. http://elrepo.org/tiki/kernel-ml http://elrepo.org/tiki/kernel-ml
- LinuxBender 3y ago
  "-ml" is always the latest upstream. "-lt" [1] will have the latest LTS kernel if that is what you were looking for. Using the kernel-ml may expose your system to security, performance and/or data corruption issues. [1] - https://elrepo.org/tiki/kernel-lt https://elrepo.org/tiki/kernel-lt
mholt 3y ago
Apparently the 6.3 kernel made significant changes to some relevant APIs -- I noticed OpenZFS is not released for 6.3 yet. https://github.com/openzfs/zfs/issues/14622 https://github.com/openzfs/zfs/issues/14622
- whoisthis4chan 3y ago
  > So far, it looks like this will be a major undertaking. Whoever was responsible for the namespace support is going to have to rebase the entire dang thing with alternate code paths for everything, because as of Linux 6.3, they replaced a whole mess of APIs so they either newly accept struct mnt_idmap * instead of struct user_namespace *, or accept the former in addition to the previous parameters. This has far reaching implications, I think. Good luck to whoever attempts the task. ouch
  - Volundr 3y ago
    Ouch indeed. I've been wondering why it seems to be taking longer than usual for ZFS to build against the new kernel. Guess I'll be on LTS kernels for awhile yet. For once at least I feel like these changes aren't just the Linux devs breaking ZFS for funsies.
    - rincebrain 3y ago
      Support is already merged in git tip, a stable release just hasn't been cut with those changes backported yet. I don't think it'd be that problematic to backport, from what I saw, just nobody's done it.
  - koverstreet 3y ago
    the struct mnt_idmap stuff was badly needed, before struct user_namespace was being passed around for two different purposes - it was just asking for confusion and bugs.
- speed_spread 3y ago
  I keep having this kinky fantasy of a GPL'd Rust ZFS re-implementation. Yeah, I know.
  - dralley 3y ago
    You'd be more likely to get one for bcachefs. Kent has expressed some interest in Rust although he is (correctly) just focusing on finishing the filesystem first.
    - koverstreet 3y ago
      yep, just waiting on gcc-rust support to land and then we'll be moving our Rust interfaces into the kernel
      - dralley 3y ago
        gcc-rs will take a few years, rustc_codegen_gcc is pretty close to being ready to go.
    - speed_spread 3y ago
      I'm curious about which would stabilize more quickly, an all new filesystem or the reimplementation of an existing one in a different language? The reimpl has the advantage of having the original to compare and test with, but the fresh design can use a staged approach. Yet more hypothetical Friday-style questions.
      - koverstreet 3y ago
        Incremental file by file rewrite, for sure.
        Someone 3y ago
        There’s a lot of unexplored stuff w.r.t. the GPL license, but I doubt that would legally work for a rewrite of CDDL-licensed ZFS to a GPL license.
e12e 3y ago
I think the title is somewhat unnecessarily editorialized from: > Bug 2208553 - xfs metadata corruption after upgrade to 6.3.3 kernel [NEEDINFO] There are reported problems, but also: > I note that all these reported failures seem to be using hardware raid and with stripe (whether either is relevant is unknown). > I am also using XFS (without issue) on a number of other systems with a 6.3 kernel, but none of them are using hardware raid or stripes. Ed: That said - it's quite worrying to see corruption on xfs as the filesystem should be really rock solid by now.
- jeffbee 3y ago
  So it only affects users who care about their data? Heck of a take.
  - ofrzeta 3y ago
    Do you imply that hardware RAID is safer than software RAID?
    - mschuster91 3y ago
      If it's battery-backed, yes.
      - mnw21cam 3y ago
        It shouldn't be safer. However, using a battery-backed write cache in hardware RAID allows the OS to just dump the writes into the cache, get the message that they're safe very quickly, and then move on to the next thing. So, a battery-backed write cache allows it to be faster while still being safe. You can still get the same level of safety without it, and you can get very close to the same speed by switching off all the protections (don't do that), or by logging writes to a separate fast storage device (like a SSD or a battery-backed RAM).
  - dralley 3y ago
    I was under the impression that hardware RAID is generally considered unreliable.
fbhabbed 3y ago
And this is why I was never a fan of rolling release distros
- deeter72 3y ago
  Ignorant take

130 points by itvision 3y ago

chasil 3y ago

Interesting that ElRepo is on 6.3.4. http://elrepo.org/tiki/kernel-ml http://elrepo.org/tiki/kernel-ml

LinuxBender 3y ago
"-ml" is always the latest upstream. "-lt" [1] will have the latest LTS kernel if that is what you were looking for. Using the kernel-ml may expose your system to security, performance and/or data corruption issues. [1] - https://elrepo.org/tiki/kernel-lt https://elrepo.org/tiki/kernel-lt

mholt 3y ago

Apparently the 6.3 kernel made significant changes to some relevant APIs -- I noticed OpenZFS is not released for 6.3 yet. https://github.com/openzfs/zfs/issues/14622 https://github.com/openzfs/zfs/issues/14622

whoisthis4chan 3y ago
> So far, it looks like this will be a major undertaking. Whoever was responsible for the namespace support is going to have to rebase the entire dang thing with alternate code paths for everything, because as of Linux 6.3, they replaced a whole mess of APIs so they either newly accept struct mnt_idmap * instead of struct user_namespace *, or accept the former in addition to the previous parameters. This has far reaching implications, I think. Good luck to whoever attempts the task. ouch
- Volundr 3y ago
  Ouch indeed. I've been wondering why it seems to be taking longer than usual for ZFS to build against the new kernel. Guess I'll be on LTS kernels for awhile yet. For once at least I feel like these changes aren't just the Linux devs breaking ZFS for funsies.
  - rincebrain 3y ago
    Support is already merged in git tip, a stable release just hasn't been cut with those changes backported yet. I don't think it'd be that problematic to backport, from what I saw, just nobody's done it.
- koverstreet 3y ago
  the struct mnt_idmap stuff was badly needed, before struct user_namespace was being passed around for two different purposes - it was just asking for confusion and bugs.
speed_spread 3y ago
I keep having this kinky fantasy of a GPL'd Rust ZFS re-implementation. Yeah, I know.
- dralley 3y ago
  You'd be more likely to get one for bcachefs. Kent has expressed some interest in Rust although he is (correctly) just focusing on finishing the filesystem first.
  - koverstreet 3y ago
    yep, just waiting on gcc-rust support to land and then we'll be moving our Rust interfaces into the kernel
    - dralley 3y ago
      gcc-rs will take a few years, rustc_codegen_gcc is pretty close to being ready to go.
  - speed_spread 3y ago
    I'm curious about which would stabilize more quickly, an all new filesystem or the reimplementation of an existing one in a different language? The reimpl has the advantage of having the original to compare and test with, but the fresh design can use a staged approach. Yet more hypothetical Friday-style questions.
    - koverstreet 3y ago
      Incremental file by file rewrite, for sure.
      - Someone 3y ago
        There’s a lot of unexplored stuff w.r.t. the GPL license, but I doubt that would legally work for a rewrite of CDDL-licensed ZFS to a GPL license.

e12e 3y ago

I think the title is somewhat unnecessarily editorialized from: > Bug 2208553 - xfs metadata corruption after upgrade to 6.3.3 kernel [NEEDINFO] There are reported problems, but also: > I note that all these reported failures seem to be using hardware raid and with stripe (whether either is relevant is unknown). > I am also using XFS (without issue) on a number of other systems with a 6.3 kernel, but none of them are using hardware raid or stripes. Ed: That said - it's quite worrying to see corruption on xfs as the filesystem should be really rock solid by now.

jeffbee 3y ago
So it only affects users who care about their data? Heck of a take.
- ofrzeta 3y ago
  Do you imply that hardware RAID is safer than software RAID?
  - mschuster91 3y ago
    If it's battery-backed, yes.
    - mnw21cam 3y ago
      It shouldn't be safer. However, using a battery-backed write cache in hardware RAID allows the OS to just dump the writes into the cache, get the message that they're safe very quickly, and then move on to the next thing. So, a battery-backed write cache allows it to be faster while still being safe. You can still get the same level of safety without it, and you can get very close to the same speed by switching off all the protections (don't do that), or by logging writes to a separate fast storage device (like a SSD or a battery-backed RAM).
- dralley 3y ago
  I was under the impression that hardware RAID is generally considered unreliable.

fbhabbed 3y ago

And this is why I was never a fan of rolling release distros

deeter72 3y ago
Ignorant take