4 ms·

Ext4 data corruption in 6.1 stable

70 points by turrini 3y ago

jeroen79 3y ago
so everything before 6.1 is ok, and after 6.5 is also ok?
- altairprime 3y ago
  6.1.64+ affected, per reply on thread.
  - kvemkon 3y ago
    Should be fixed in 6.1.66. iomap: update ki_pos a little later in iomap_dio_complete https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.1.y&id=ea574927fc0bc343016ea3337fcfc0b3fb26fe08 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
    - csdvrx 3y ago
      Is what you've linked the patch that should be backported to affected kernels? As the potentially problematic patch causing the corruption, I found https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.1.y&id=91562895f803 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux... but it's not present in the 6.2.2 I run. I'm very conservative in the versions or critical components like kernels or dkms modules, that didn't protect me against the ZFS bug, but it seems to have avoided me this EXT4 bug!
- pixl97 3y ago
  >The commit got merged in 6.5-rc1 so all stable kernels that have 91562895f803 ("ext4: properly sync file size update after O_SYNC direct IO") before 6.5 are corrupting data - I've noticed at least 6.1 is still carrying the problematic commit. Greg, please take out the commit from all stable kernels before 6.5 as soon as possible, we'll figure out proper backport once user data are not being corrupted anymore. Thanks!
- spyc 3y ago
  The way I read https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuhc3@quack3/ https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuh... 6.5+ should be okay. PS: Please correct me if not!
RainbowFriends 3y ago
I guess this falls into the class of Linux bugs that causes regressions on the LTS kernel, where you'd be safer running stable.
Modified3019 3y ago
Is this triggered under certain workloads, or has it been chewing data for everyone, and how much?
- csdvrx 3y ago
  > Is this triggered under certain workloads, or has it been chewing data for everyone, and how much? Unless you are checksumming your files through a daemon run on a schedule (IDS style) AND manually review the results every now and then, you can't know.
  - uselpa 3y ago
    If you have versioned backups, you can diff against update_day-1 without an additional tool.
    - csdvrx 3y ago
      I have versioned backups for a few months. Most people don't and can't (it requires a lot of storage) Even if they have them, the diffing needs to be done. Given the past problems with ZFS, I think I'll write a summarizer, that creates a database of metadata for each backup: it will be small to keep around, and may be use more easily to determine which backup contains the uncorrupted file.
      - uselpa 3y ago
        I use borgbackup so the storage requirements are fairly low if the data does not change too much (which is my case). Also diffs are just a « borg diff » away.
- o11c 3y ago
  It appears to require O_DIRECT, which is notoriously buggy and no sane program should ever use.
  - spyc 3y ago
snvzz 3y ago
When the Linux kernel is huge (MLoCs!), complex and lacks stable internal APIs, this should not surprise anyone. There are better ways to do it, and it is being done. Refer to the microkernel multiserver architecture, with efforts such as Genode (on seL4) or Fuchsia.
- diegocg 3y ago
  How on earth does running a file system in userspace suddenly makes file systems safe? Also, how does putting N millions of LoC in N projects solve problems magically?
uselpa 3y ago
I had already updated to the problematic kernel, so I resorted to installing 6.5 from backports. Hope this helps.
jacobgorm 3y ago
I’ve defaulted to XFS since it became available for Linux, and never had any problems. Only once, some time around 2009, did I try ext3 and it forgot to flush a lot of data that were lost. I don’t see why anyone would use ext over XFS.
- davidgerard 3y ago
  XFS is all but unmaintained, which may be a good reason to be cautious. Very stable code, but the world may change out from under it.
  - snvzz 3y ago
    In a sane system, the world would not "change out from under it". Linux is a special category of mess.
    - crote 3y ago
      Why would the world not change? When XFS was designed, the storage ecosystem was spinning rust attached using PATA and SCSI. We've developed quite a bit since then, with things like SAS, NVMe, and SSDs. Storage have gotten over an order of magnitude more throughput, over an order of magnitude less latency, and several orders of magnitude more IOPS. Running a filesystem designed for ancient hardware means you won't be taking advantage of several decades of innovation.
      - snvzz 3y ago
        The world here means the APIs XFS is built on top of. Yes, it is possible to maintain stable APIs and even offer multiple versions of. Other systems (even AmigaOS in the 80s) manage to do so. Linux is a special case.
      - jacobgorm 3y ago
        In what ways would you say that ext4 takes advantage of the latest several decades of innovation, where XFS does not? Both XFS and ext2 were first released around 1993, and it is safe to say that XFS was more advanced than ext2 at that time, given that it had journaling and ext2 did not. If I wanted something actually novel, I would go for ZFS or one of its clones.
abrookewood 3y ago
Any of the people who were freaking out over a rare ZFS corruption care to comment?
- ThePowerOfFuet 3y ago
  What about it?

70 points by turrini 3y ago

jeroen79 3y ago

so everything before 6.1 is ok, and after 6.5 is also ok?

altairprime 3y ago
6.1.64+ affected, per reply on thread.
- kvemkon 3y ago
  Should be fixed in 6.1.66. iomap: update ki_pos a little later in iomap_dio_complete https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.1.y&id=ea574927fc0bc343016ea3337fcfc0b3fb26fe08 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
  - csdvrx 3y ago
    Is what you've linked the patch that should be backported to affected kernels? As the potentially problematic patch causing the corruption, I found https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-6.1.y&id=91562895f803 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux... but it's not present in the 6.2.2 I run. I'm very conservative in the versions or critical components like kernels or dkms modules, that didn't protect me against the ZFS bug, but it seems to have avoided me this EXT4 bug!
pixl97 3y ago
>The commit got merged in 6.5-rc1 so all stable kernels that have 91562895f803 ("ext4: properly sync file size update after O_SYNC direct IO") before 6.5 are corrupting data - I've noticed at least 6.1 is still carrying the problematic commit. Greg, please take out the commit from all stable kernels before 6.5 as soon as possible, we'll figure out proper backport once user data are not being corrupted anymore. Thanks!
spyc 3y ago
The way I read https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuhc3@quack3/ https://lore.kernel.org/stable/20231205122122.dfhhoaswsfscuh... 6.5+ should be okay. PS: Please correct me if not!

RainbowFriends 3y ago

I guess this falls into the class of Linux bugs that causes regressions on the LTS kernel, where you'd be safer running stable.

Modified3019 3y ago

Is this triggered under certain workloads, or has it been chewing data for everyone, and how much?

csdvrx 3y ago
> Is this triggered under certain workloads, or has it been chewing data for everyone, and how much? Unless you are checksumming your files through a daemon run on a schedule (IDS style) AND manually review the results every now and then, you can't know.
- uselpa 3y ago
  If you have versioned backups, you can diff against update_day-1 without an additional tool.
  - csdvrx 3y ago
    I have versioned backups for a few months. Most people don't and can't (it requires a lot of storage) Even if they have them, the diffing needs to be done. Given the past problems with ZFS, I think I'll write a summarizer, that creates a database of metadata for each backup: it will be small to keep around, and may be use more easily to determine which backup contains the uncorrupted file.
    - uselpa 3y ago
      I use borgbackup so the storage requirements are fairly low if the data does not change too much (which is my case). Also diffs are just a « borg diff » away.
o11c 3y ago
It appears to require O_DIRECT, which is notoriously buggy and no sane program should ever use.
- spyc 3y ago

snvzz 3y ago

When the Linux kernel is huge (MLoCs!), complex and lacks stable internal APIs, this should not surprise anyone. There are better ways to do it, and it is being done. Refer to the microkernel multiserver architecture, with efforts such as Genode (on seL4) or Fuchsia.

diegocg 3y ago
How on earth does running a file system in userspace suddenly makes file systems safe? Also, how does putting N millions of LoC in N projects solve problems magically?

uselpa 3y ago

I had already updated to the problematic kernel, so I resorted to installing 6.5 from backports. Hope this helps.

jacobgorm 3y ago

I’ve defaulted to XFS since it became available for Linux, and never had any problems. Only once, some time around 2009, did I try ext3 and it forgot to flush a lot of data that were lost. I don’t see why anyone would use ext over XFS.

davidgerard 3y ago
XFS is all but unmaintained, which may be a good reason to be cautious. Very stable code, but the world may change out from under it.
- snvzz 3y ago
  In a sane system, the world would not "change out from under it". Linux is a special category of mess.
  - crote 3y ago
    Why would the world not change? When XFS was designed, the storage ecosystem was spinning rust attached using PATA and SCSI. We've developed quite a bit since then, with things like SAS, NVMe, and SSDs. Storage have gotten over an order of magnitude more throughput, over an order of magnitude less latency, and several orders of magnitude more IOPS. Running a filesystem designed for ancient hardware means you won't be taking advantage of several decades of innovation.
    - snvzz 3y ago
      The world here means the APIs XFS is built on top of. Yes, it is possible to maintain stable APIs and even offer multiple versions of. Other systems (even AmigaOS in the 80s) manage to do so. Linux is a special case.
    - jacobgorm 3y ago
      In what ways would you say that ext4 takes advantage of the latest several decades of innovation, where XFS does not? Both XFS and ext2 were first released around 1993, and it is safe to say that XFS was more advanced than ext2 at that time, given that it had journaling and ext2 did not. If I wanted something actually novel, I would go for ZFS or one of its clones.

abrookewood 3y ago

Any of the people who were freaking out over a rare ZFS corruption care to comment?

ThePowerOfFuet 3y ago
What about it?