6 ms·

As someone who is heavily dependent on zfs, this is a little bit of reassurance. Sad that data corruption exists in any file system that is shipped too many peo

As someone who is heavily dependent on zfs, this is a little bit of reassurance. Sad that data corruption exists in any file system that is shipped too many people, but reassuring that it happens to the most stable and least "interesting" in the newness sense file systems. ZFS's own recent file system corruption issue is in roughly the same category of edge case, but accessible to reasonable if niche workloads.

upon_drumhead 3y ago
Yeah, all the threads full of folks bashing on ZFS because their trusty ext won’t corrupt data should be eating crow right about now, but I’m sure they’ll hop in and explain why this bug is different.
- megous 3y ago
  It's not an extfs bug, it's a "stable kernel" process bug. Linux master branch/mainline was never affected by this bug. Someone cherry-picked a wrong patch into stable without realizing the consequences, and it was caught too late. Stable process is cherry pincking thousands to tens of thousands of patches from the current master kernel branch into years old kernel branches, spraying tens of thousands of emails at original patch authors, hoping (With some limited testing on top) that the resulting frankenkernels will still work and all these patches will have satisfied dependencies applied, too. I don't trust this process very much, and just run the latest stable branch for the latest kernel release, only. Staying with older stable release branch for too long seems too risky, unless you're some bigcorp that can afford the testing required, or you're running some highly mainstream setup that is probably covered by tests done by the stable team, and testing teams they cooperate with.
  - uselpa 3y ago
    I understand your reasoning, but running a popular « stable » distro in a popular configuration will help detect bugs much faster. I’m not sure this would be the case for the very latest kernel.
    - megous 3y ago
      A bunch of popular distros run the latest kernel.
  - keep_reading 3y ago
    Nobody ships vanilla kernels. Sure the reference implementation was unaffected but users never experience vanilla Linux kernels
    - dezgeg 3y ago
      Distros like Arch and NixOS use the vanilla kernel with very small amount (single digits) of patches on top. For the purposes of ext4, it's very likely to be as-is with vanilla. (I am counting the official linux-stable releases from kernel.org as 'vanilla' here.)
      - megous 3y ago
        Fedora, openSUSE tumbleweed, Manjaro,...
        keep_reading 3y ago
        > Fedora You know it's not that hard to debunk this: > You'll then be left with a kernel-6.X.? directory, containing both an unpatched 'vanilla-6.X.?' dir, and a linux-6.X.?-noarch hardlinked dir which has the Fedora patches applied. [1] WOW fedora patches! Sounds to me like they're not shipping vanilla. [1] https://fedoraproject.org/wiki/Kernel https://fedoraproject.org/wiki/Kernel
        megous 3y ago
        That's not what my original argument was about. So not sure what you're debunking. I have problem with backaptching 10s of thousands of changes to years old kernels by people who don't really understand the changes or consequences, like 5.4, 5.15, 6.1 or whatever. Not patching up 6.6 or 6.7 kernel with a few out of tree patches, where they mostly understand what they're doing and can test the limited set of changes they're applying.
        keep_reading 3y ago
        If they change anything it's no longer vanilla. Anything at all. No matter how insignificant.
- uxp8u61q 3y ago
  This isn't a sports tournament, nobody is cheering or bashing "teams". If ZFS has bugs and caveats, they should be called out. Same for ext4.
  - GauntletWizard 3y ago
    [flagged]
    - baz00 3y ago
      Well said. Twisted partisanism. I’m the middle ground. It all sucks in one way or another but we’re generally trying to our best.
    - reanimus 3y ago
      > When a core rust developer is exposed as a psychotic pedophile, there's crickets - it's not news because it happens every month. bro what on earth are you on about
Lt_Riza_Hawkeye 3y ago
The problem here was Debian's distribution process. Any distro compiling linux from the releases on kernel.org was not affected.
- Blikkentrekker 3y ago
  So this is another bug introduced by Debian itself by patching things? I remember that there was a fairly severe one which was caused by patching OpenSSL I think? But I remember the change they made being fairly weird and no one understood why but it was easy to see that it would introduce a vulnerability.
  - mordae 3y ago
    No, the issue was caused by backporting a patch A but not backporting a patch B. Sadly in this case the overall behavior after applying just A was broken when issuing direct IO writes.
  - keep_reading 3y ago
    I remember when they made 2 CVEs back to back in Apache by incorrect back porting of changes
  - remram 3y ago
    No, it exists in the upstream release.
- asdfgasdfh 3y ago
  [dead]
- uselpa 3y ago
  Are you sure about that (genuine question)? The linked discussion involves a Suse engineer and a request to the kernel maintainer directly, not to a Debian packager-
  - burtness 3y ago
    No, its an upstream bug being discussed in the debian bug tracker
404mm 3y ago
You’re forgetting the btrfs folks. — someone who lost data due to btrfs bug
nolist_policy 3y ago
Note though that unlike zfs, ext4 has a battle-tested fsck. Combine ext4's dumb but robust approach to journaling and robust metadata layout (for example inodes are statically allocated) with fsck.ext4 (which got refined for years) and you can recover from any situation. To give you an example, fsck.ext4 will happily carve a working filesystem out of random data, as long as there is a valid superblock. Seriously - try it yourself: # create a working filesystem image dd if=/dev/zero of=test1.img bs=1M count=256 /sbin/mkfs.ext4 test1.img # write file with random data dd if=/dev/urandom of=test2.img bs=1M count=256 # copy superblock into random file dd if=test1.img of=test2.img bs=1024 count=4 seek=1 skip=1 conv=notrunc /sbin/fsck.ext4 -fy test2.img sudo mount test2.img /mnt/somewhere
- mustache_kimono 3y ago
  I'm trying to understand what exactly this is testing/demonstrating. Like: "Wow, but why should I care?" I'm not sure being able to fsck a random disk image shows resiliency. Doing this could do all sorts of nasty things to data you actually care about.
  - grrandalf 3y ago
    (not the author) The metadata is a contiguous range of disk blocks. I think the intuition is that such layouts are likely to require simpler filesystem code to manipulate. [versus (iirc) ZFS which may have metadata blocks scattered throughout the disk, probably requiring more intricate code to keep track of].
    - mustache_kimono 3y ago
      > [versus (iirc) ZFS which may have metadata blocks scattered throughout the disk, probably requiring more intricate code to keep track of]. So -- you think redundant metadata is a bad thing? Try wiping your metadata and then trying to fsck that random disk image. Again, has this been the source of data corruption you're aware of? This seems like a "Maybe, it could be this way, but I don't know" kind of take.
- 3y ago

upon_drumhead 3y ago

Yeah, all the threads full of folks bashing on ZFS because their trusty ext won’t corrupt data should be eating crow right about now, but I’m sure they’ll hop in and explain why this bug is different.

megous 3y ago
It's not an extfs bug, it's a "stable kernel" process bug. Linux master branch/mainline was never affected by this bug. Someone cherry-picked a wrong patch into stable without realizing the consequences, and it was caught too late. Stable process is cherry pincking thousands to tens of thousands of patches from the current master kernel branch into years old kernel branches, spraying tens of thousands of emails at original patch authors, hoping (With some limited testing on top) that the resulting frankenkernels will still work and all these patches will have satisfied dependencies applied, too. I don't trust this process very much, and just run the latest stable branch for the latest kernel release, only. Staying with older stable release branch for too long seems too risky, unless you're some bigcorp that can afford the testing required, or you're running some highly mainstream setup that is probably covered by tests done by the stable team, and testing teams they cooperate with.
- uselpa 3y ago
  I understand your reasoning, but running a popular « stable » distro in a popular configuration will help detect bugs much faster. I’m not sure this would be the case for the very latest kernel.
  - megous 3y ago
    A bunch of popular distros run the latest kernel.
- keep_reading 3y ago
  Nobody ships vanilla kernels. Sure the reference implementation was unaffected but users never experience vanilla Linux kernels
  - dezgeg 3y ago
    Distros like Arch and NixOS use the vanilla kernel with very small amount (single digits) of patches on top. For the purposes of ext4, it's very likely to be as-is with vanilla. (I am counting the official linux-stable releases from kernel.org as 'vanilla' here.)
    - megous 3y ago
      Fedora, openSUSE tumbleweed, Manjaro,...
      - keep_reading 3y ago
        > Fedora You know it's not that hard to debunk this: > You'll then be left with a kernel-6.X.? directory, containing both an unpatched 'vanilla-6.X.?' dir, and a linux-6.X.?-noarch hardlinked dir which has the Fedora patches applied. [1] WOW fedora patches! Sounds to me like they're not shipping vanilla. [1] https://fedoraproject.org/wiki/Kernel https://fedoraproject.org/wiki/Kernel
        megous 3y ago
        That's not what my original argument was about. So not sure what you're debunking. I have problem with backaptching 10s of thousands of changes to years old kernels by people who don't really understand the changes or consequences, like 5.4, 5.15, 6.1 or whatever. Not patching up 6.6 or 6.7 kernel with a few out of tree patches, where they mostly understand what they're doing and can test the limited set of changes they're applying.
        keep_reading 3y ago
        If they change anything it's no longer vanilla. Anything at all. No matter how insignificant.
uxp8u61q 3y ago
This isn't a sports tournament, nobody is cheering or bashing "teams". If ZFS has bugs and caveats, they should be called out. Same for ext4.
- GauntletWizard 3y ago
  [flagged]
  - baz00 3y ago
    Well said. Twisted partisanism. I’m the middle ground. It all sucks in one way or another but we’re generally trying to our best.
  - reanimus 3y ago
    > When a core rust developer is exposed as a psychotic pedophile, there's crickets - it's not news because it happens every month. bro what on earth are you on about

Lt_Riza_Hawkeye 3y ago

The problem here was Debian's distribution process. Any distro compiling linux from the releases on kernel.org was not affected.

Blikkentrekker 3y ago
So this is another bug introduced by Debian itself by patching things? I remember that there was a fairly severe one which was caused by patching OpenSSL I think? But I remember the change they made being fairly weird and no one understood why but it was easy to see that it would introduce a vulnerability.
- mordae 3y ago
  No, the issue was caused by backporting a patch A but not backporting a patch B. Sadly in this case the overall behavior after applying just A was broken when issuing direct IO writes.
- keep_reading 3y ago
  I remember when they made 2 CVEs back to back in Apache by incorrect back porting of changes
- remram 3y ago
  No, it exists in the upstream release.
asdfgasdfh 3y ago
[dead]
uselpa 3y ago
Are you sure about that (genuine question)? The linked discussion involves a Suse engineer and a request to the kernel maintainer directly, not to a Debian packager-
- burtness 3y ago
  No, its an upstream bug being discussed in the debian bug tracker

404mm 3y ago

You’re forgetting the btrfs folks. — someone who lost data due to btrfs bug

nolist_policy 3y ago

Note though that unlike zfs, ext4 has a battle-tested fsck. Combine ext4's dumb but robust approach to journaling and robust metadata layout (for example inodes are statically allocated) with fsck.ext4 (which got refined for years) and you can recover from any situation. To give you an example, fsck.ext4 will happily carve a working filesystem out of random data, as long as there is a valid superblock. Seriously - try it yourself: # create a working filesystem image dd if=/dev/zero of=test1.img bs=1M count=256 /sbin/mkfs.ext4 test1.img # write file with random data dd if=/dev/urandom of=test2.img bs=1M count=256 # copy superblock into random file dd if=test1.img of=test2.img bs=1024 count=4 seek=1 skip=1 conv=notrunc /sbin/fsck.ext4 -fy test2.img sudo mount test2.img /mnt/somewhere

mustache_kimono 3y ago
I'm trying to understand what exactly this is testing/demonstrating. Like: "Wow, but why should I care?" I'm not sure being able to fsck a random disk image shows resiliency. Doing this could do all sorts of nasty things to data you actually care about.
- grrandalf 3y ago
  (not the author) The metadata is a contiguous range of disk blocks. I think the intuition is that such layouts are likely to require simpler filesystem code to manipulate. [versus (iirc) ZFS which may have metadata blocks scattered throughout the disk, probably requiring more intricate code to keep track of].
  - mustache_kimono 3y ago
    > [versus (iirc) ZFS which may have metadata blocks scattered throughout the disk, probably requiring more intricate code to keep track of]. So -- you think redundant metadata is a bad thing? Try wiping your metadata and then trying to fsck that random disk image. Again, has this been the source of data corruption you're aware of? This seems like a "Maybe, it could be this way, but I don't know" kind of take.
3y ago