5 ms·

Ext4 data corruption trouble

67 points by tshtf 14y ago

cokernel_hacker 14y ago
This is symptomatic with one of my two big problems with journalling oriented file-systems. My problems with journalling are two fold: 1) They are very slow: 1a) You have a nice big sequential write into the journal, which is OK. 1b) A flush track cache to make sure it is actually in the journal. This can sync whatever has accumulated in the track cache which might not just be journal data. 1c) The actual overwrites that spew data randomly over the drive. 1d) Writes to update the journal header/terminate the transaction. 1e) A final flush track cache which will sync who-knows-what onto the platters/flash. 2) Replay behavior of the journal log is _very_ fragile code. You need to handle lots of terrible cases, the most awful of which is to ensure that you don't play older transaction on top of newer transactions. You might say "hey, that shouldn't happen!" but it happens. It happens because the code is not trivial to write and detecting these bad cases aren't trivial. Even if you do get the code write, you are still screwed. Why? Because if your drive does not support the flush track cache mechanism you are in for a world of pain. You can have a journal and journal header that is ancient if it just stuck in some cache... The ext* family of filesystems do not appear to have natural resiliency to this sort of problem. Instead, it appears to be a coordinated, concerted effort between various parts of the journaling code.
- riledhel 14y ago
  What is your main use case? What is your choice instead of ext*?
  - cokernel_hacker 14y ago
    I like write-anywhere/COW filesystems. Unfortunately the open source ones kinda blow due to interesting performance penalties (I'm looking at you, btrfs back ref management) or crazy memory usage/non-extent based systems/read-modify-write induced mania (I'm looking at you, ZFS block tree)
    - mcpherrinm 14y ago
      What non-open source ones do you like, then?
      - sliverstorm 14y ago
        Oh, you've probably never heard of them.
    - zanny 14y ago
      I've been playing with btrfs under the 3.7 prerelease kernel on an arch kvm with a virtio disks, and with lzo compression btrfs is kicking some serious read / write butt vs every other major filesystem I can think of. Given sufficient hardware, of course. Haven't tried it on an ssd though, only on 7200rpm turnstables.
  - lmm 14y ago
    I still use reiserfs for all my linux machines (ext2 for /boot); ZFS on freebsd. I haven't experienced the corruption I've seen under ext3 with either of them.
- rbanffy 14y ago
  > The ext* family of filesystems do not appear to have natural resiliency to this sort of problem. I believe the fact data corruption on ext4 is rare shows it's not really a huge problem. The only time I lost data on an ext4 filesystem was when I mistyped a wildcard for rm.
  - cokernel_hacker 14y ago
    I would rather not have byzantine relations between fragments of code make up the policy of my file system's metadata resiliency thank-you-very-much. I would rather prefer that stale journal replays be no-ops, even at the expense of making journal replay slower.
    - rbanffy 14y ago
      tune2fs -O ^has_journal /dev/sda1 You're welcome.
- gjm11 14y ago
  > This is symptomatic with one of my two big problems with journalling oriented file-systems. Personally, I think it's more symptomatic of the problems of filesystem-oriented journalists. See the comment quoted by sciurus. (Irrelevant note: I really don't recommend reversing all the arrows in a Unix kernel. But I like your username anyway.)
  - cokernel_hacker 14y ago
    My apologies but what sciurus might be addressing and what I am talking about are addressing two very different things. There is a bug in ext4, https://lkml.org/lkml/2012/10/25/521 https://lkml.org/lkml/2012/10/25/521 Note that this is dated after the "Wed, 24 Oct 2012 17:31:29 -0400" post and talks about changing code related to a truncated journal. My post does not say anything about the likelihood of the dataloss due to such bugs, I mention what happens in every design in a WAL journaling system. Talking about the impact of a particular ext bug is up to the people who post on the LKML. To be honest, I don't really care how severe the bug is, I care about why there are bugs and journal truncation is a source of them.
- tytso 14y ago
  >Because if your drive does not support the flush track cache mechanism you are in for a world of pain. If your drive does not support CACHE_FLUSH mechanism, then pretty much any file system that are supposed to handle unclean shutdowns (ZFS, btrfs), etc., are going to be screwed. More generally, any time you have a file system, there will always be very delicate code. Fundamentally file systems are _hard_. It has to handle a huge amount of concurrent operations, and users want speed, so we use fine-grained locking, and there's a reason why file systems take 100 man years or so to become production ready. The btrfs folks started in 2007, and I warned them it would probably be at least 5-7 years, minimum before it would be ready for the enterprise. And now here it is five years later, and the community distributions (not the enterprise distros) are just starting to adopt btrfs. ZFS took seven years to develop before Sun announced it, and then it took a few more years before people trusted it on production servers. No one believes how hard it is when they start, but the history is pretty clear.
bcl 14y ago
Update here: https://lwn.net/Articles/521090/ https://lwn.net/Articles/521090/
sciurus 14y ago
Best comment on that thread (from the bug reporter) "Those following along at home is probably half the human race, now we have posts on Phoronix, Slashdot and Heise. Who the hell submits things like this to random-terrified-user media outlets before we've even characterized the bloody problem? Every one of those posts is inaccurate, of course, through no fault of their own but merely because we didn't yet know what the problem was ourselves, merely that I and one other person were seeing corruption: we obviously started by assuming that it was something obvious and thus fairly serious, but that didn't mean we expected that to be true: I certainly expected the final problem to be more subtle, if still capable of causing serious disk corruption (my warning here was just in case it was not). But now there's a wave of self-sustaining accidental lies spreading across the net, damaging the reputation of ext4 unwarrantedly, and I started it without wanting to. It's times like this when I start to understand why some companies have closed bug trackers."
- saurik 14y ago
  Also (same overall gist but with more technical detail), https://lwn.net/Articles/521090/ https://lwn.net/Articles/521090/
batgaijin 14y ago
What's the status with BTRFS? Also, is ZFS Linux performance still terrible, or did they find a way to fix that?
- meaty 14y ago
  btrfs is shipping with OpenSuSE and Oracle Linux apparently as stable. Not sure I trust it yet. If they ship it with Debian stable I will consider it. ZFS still uses FUSE AFAIK so is going to suck.
  - skeletonjelly 14y ago
    Doubt ZFS is going to be anything but FUSE. Mandated by licenses. ZFS can be used with *BSD of course.
    - Andys 14y ago
      The linux ZFS kernel modules work well and are not prohibited by license. It simply will never be included in the Linux kernel.
  - batgaijin 14y ago
    I thought they had a kernel module or something bypassing the need for fuse.
    - dmpk2k 14y ago
      Indeed there is: http://zfsonlinux.org/ http://zfsonlinux.org/ There's more than one ZFS port to Linux, which might be confusing some people. There's also the more dramatic option of using FreeBSD or one of the illumos derivatives, which work fine in my experience too.
  - kzrdude 14y ago
    Btrfs will be in the next stable, and it will be Linux 3.2. Not a good idea to use that old version of linux if you want btrfs!
Florin_Andrei 14y ago
Oh great. Ubuntu 12.04 Server crashes randomly due to some obscure bug in 3.2 I upgraded the kernel to 3.6.3 specifically to stop the machine from crashing. Now let's hope that INDEED it doesn't crash, else I might get hit by this newfangled Ext4 bug. Sounds like the accelerated kernel development is hitting various limits, they should go back to more stodgy stable series like before.
- rbanffy 14y ago
  > Ubuntu 12.04 Server crashes randomly due to some obscure bug in 3.2 Must be really obscure. With more than 100 servers under my watch, I never saw anything like that.
  - Florin_Andrei 14y ago
    I'm guessing all those 100 servers are the same hardware, or a few variations.

67 points by tshtf 14y ago

This is symptomatic with one of my two big problems with journalling oriented file-systems. My problems with journalling are two fold: 1) They are very slow: 1a) You have a nice big sequential write into the journal, which is OK. 1b) A flush track cache to make sure it is actually in the journal. This can sync whatever has accumulated in the track cache which might not just be journal data. 1c) The actual overwrites that spew data randomly over the drive. 1d) Writes to update the journal header/terminate the transaction. 1e) A final flush track cache which will sync who-knows-what onto the platters/flash. 2) Replay behavior of the journal log is _very_ fragile code. You need to handle lots of terrible cases, the most awful of which is to ensure that you don't play older transaction on top of newer transactions. You might say "hey, that shouldn't happen!" but it happens. It happens because the code is not trivial to write and detecting these bad cases aren't trivial. Even if you do get the code write, you are still screwed. Why? Because if your drive does not support the flush track cache mechanism you are in for a world of pain. You can have a journal and journal header that is ancient if it just stuck in some cache... The ext* family of filesystems do not appear to have natural resiliency to this sort of problem. Instead, it appears to be a coordinated, concerted effort between various parts of the journaling code.

riledhel 14y ago
What is your main use case? What is your choice instead of ext*?
- cokernel_hacker 14y ago
  I like write-anywhere/COW filesystems. Unfortunately the open source ones kinda blow due to interesting performance penalties (I'm looking at you, btrfs back ref management) or crazy memory usage/non-extent based systems/read-modify-write induced mania (I'm looking at you, ZFS block tree)
  - mcpherrinm 14y ago
    What non-open source ones do you like, then?
    - sliverstorm 14y ago
      Oh, you've probably never heard of them.
  - zanny 14y ago
    I've been playing with btrfs under the 3.7 prerelease kernel on an arch kvm with a virtio disks, and with lzo compression btrfs is kicking some serious read / write butt vs every other major filesystem I can think of. Given sufficient hardware, of course. Haven't tried it on an ssd though, only on 7200rpm turnstables.
- lmm 14y ago
  I still use reiserfs for all my linux machines (ext2 for /boot); ZFS on freebsd. I haven't experienced the corruption I've seen under ext3 with either of them.
rbanffy 14y ago
> The ext* family of filesystems do not appear to have natural resiliency to this sort of problem. I believe the fact data corruption on ext4 is rare shows it's not really a huge problem. The only time I lost data on an ext4 filesystem was when I mistyped a wildcard for rm.
- cokernel_hacker 14y ago
  I would rather not have byzantine relations between fragments of code make up the policy of my file system's metadata resiliency thank-you-very-much. I would rather prefer that stale journal replays be no-ops, even at the expense of making journal replay slower.
  - rbanffy 14y ago
    tune2fs -O ^has_journal /dev/sda1 You're welcome.
gjm11 14y ago
> This is symptomatic with one of my two big problems with journalling oriented file-systems. Personally, I think it's more symptomatic of the problems of filesystem-oriented journalists. See the comment quoted by sciurus. (Irrelevant note: I really don't recommend reversing all the arrows in a Unix kernel. But I like your username anyway.)
- cokernel_hacker 14y ago
  My apologies but what sciurus might be addressing and what I am talking about are addressing two very different things. There is a bug in ext4, https://lkml.org/lkml/2012/10/25/521 https://lkml.org/lkml/2012/10/25/521 Note that this is dated after the "Wed, 24 Oct 2012 17:31:29 -0400" post and talks about changing code related to a truncated journal. My post does not say anything about the likelihood of the dataloss due to such bugs, I mention what happens in every design in a WAL journaling system. Talking about the impact of a particular ext bug is up to the people who post on the LKML. To be honest, I don't really care how severe the bug is, I care about why there are bugs and journal truncation is a source of them.
tytso 14y ago
>Because if your drive does not support the flush track cache mechanism you are in for a world of pain. If your drive does not support CACHE_FLUSH mechanism, then pretty much any file system that are supposed to handle unclean shutdowns (ZFS, btrfs), etc., are going to be screwed. More generally, any time you have a file system, there will always be very delicate code. Fundamentally file systems are _hard_. It has to handle a huge amount of concurrent operations, and users want speed, so we use fine-grained locking, and there's a reason why file systems take 100 man years or so to become production ready. The btrfs folks started in 2007, and I warned them it would probably be at least 5-7 years, minimum before it would be ready for the enterprise. And now here it is five years later, and the community distributions (not the enterprise distros) are just starting to adopt btrfs. ZFS took seven years to develop before Sun announced it, and then it took a few more years before people trusted it on production servers. No one believes how hard it is when they start, but the history is pretty clear.

bcl 14y ago

Update here: https://lwn.net/Articles/521090/ https://lwn.net/Articles/521090/

sciurus 14y ago

Best comment on that thread (from the bug reporter) "Those following along at home is probably half the human race, now we have posts on Phoronix, Slashdot and Heise. Who the hell submits things like this to random-terrified-user media outlets before we've even characterized the bloody problem? Every one of those posts is inaccurate, of course, through no fault of their own but merely because we didn't yet know what the problem was ourselves, merely that I and one other person were seeing corruption: we obviously started by assuming that it was something obvious and thus fairly serious, but that didn't mean we expected that to be true: I certainly expected the final problem to be more subtle, if still capable of causing serious disk corruption (my warning here was just in case it was not). But now there's a wave of self-sustaining accidental lies spreading across the net, damaging the reputation of ext4 unwarrantedly, and I started it without wanting to. It's times like this when I start to understand why some companies have closed bug trackers."

saurik 14y ago
Also (same overall gist but with more technical detail), https://lwn.net/Articles/521090/ https://lwn.net/Articles/521090/

batgaijin 14y ago

What's the status with BTRFS? Also, is ZFS Linux performance still terrible, or did they find a way to fix that?

meaty 14y ago
btrfs is shipping with OpenSuSE and Oracle Linux apparently as stable. Not sure I trust it yet. If they ship it with Debian stable I will consider it. ZFS still uses FUSE AFAIK so is going to suck.
- skeletonjelly 14y ago
  Doubt ZFS is going to be anything but FUSE. Mandated by licenses. ZFS can be used with *BSD of course.
  - Andys 14y ago
    The linux ZFS kernel modules work well and are not prohibited by license. It simply will never be included in the Linux kernel.
- batgaijin 14y ago
  I thought they had a kernel module or something bypassing the need for fuse.
  - dmpk2k 14y ago
    Indeed there is: http://zfsonlinux.org/ http://zfsonlinux.org/ There's more than one ZFS port to Linux, which might be confusing some people. There's also the more dramatic option of using FreeBSD or one of the illumos derivatives, which work fine in my experience too.
- kzrdude 14y ago
  Btrfs will be in the next stable, and it will be Linux 3.2. Not a good idea to use that old version of linux if you want btrfs!

Florin_Andrei 14y ago

Oh great. Ubuntu 12.04 Server crashes randomly due to some obscure bug in 3.2 I upgraded the kernel to 3.6.3 specifically to stop the machine from crashing. Now let's hope that INDEED it doesn't crash, else I might get hit by this newfangled Ext4 bug. Sounds like the accelerated kernel development is hitting various limits, they should go back to more stodgy stable series like before.

rbanffy 14y ago
> Ubuntu 12.04 Server crashes randomly due to some obscure bug in 3.2 Must be really obscure. With more than 100 servers under my watch, I never saw anything like that.
- Florin_Andrei 14y ago
  I'm guessing all those 100 servers are the same hardware, or a few variations.