9 ms·
Linux: Ext4 data corruption in 6.1.64-1
- neilv 3y agoMirror: https://web.archive.org/web/20231210053348/https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057843 https://web.archive.org/web/20231210053348/https://bugs.debi... Related: "Debian 12.3 image release delayed" https://lists.debian.org/debian-announce/2023/msg00005.html https://lists.debian.org/debian-announce/2023/msg00005.html
- GauntletWizard 3y agoAs someone who is heavily dependent on zfs, this is a little bit of reassurance. Sad that data corruption exists in any file system that is shipped too many people, but reassuring that it happens to the most stable and least "interesting" in the newness sense file systems. ZFS's own recent file system corruption issue is in roughly the same category of edge case, but accessible to reasonable if niche workloads.
- upon_drumhead 3y agoYeah, all the threads full of folks bashing on ZFS because their trusty ext won’t corrupt data should be eating crow right about now, but I’m sure they’ll hop in and explain why this bug is different.
- megous 3y agoIt's not an extfs bug, it's a "stable kernel" process bug. Linux master branch/mainline was never affected by this bug. Someone cherry-picked a wrong patch into stable without realizing the consequences, and it was caught too late. Stable process is cherry pincking thousands to tens of thousands of patches from the current master kernel branch into years old kernel branches, spraying tens of thousands of emails at original patch authors, hoping (With some limited testing on top) that the resulting frankenkernels will still work and all these patches will have satisfied dependencies applied, too. I don't trust this process very much, and just run the latest stable branch for the latest kernel release, only. Staying with older stable release branch for too long seems too risky, unless you're some bigcorp that can afford the testing required, or you're running some highly mainstream setup that is probably covered by tests done by the stable team, and testing teams they cooperate with.
- uselpa 3y agoI understand your reasoning, but running a popular « stable » distro in a popular configuration will help detect bugs much faster. I’m not sure this would be the case for the very latest kernel.
- megous 3y agoA bunch of popular distros run the latest kernel.
- keep_reading 3y agoNobody ships vanilla kernels. Sure the reference implementation was unaffected but users never experience vanilla Linux kernels
- dezgeg 3y agoDistros like Arch and NixOS use the vanilla kernel with very small amount (single digits) of patches on top. For the purposes of ext4, it's very likely to be as-is with vanilla. (I am counting the official linux-stable releases from kernel.org as 'vanilla' here.)
- megous 3y agoFedora, openSUSE tumbleweed, Manjaro,...
- keep_reading 3y ago> Fedora You know it's not that hard to debunk this: > You'll then be left with a kernel-6.X.? directory, containing both an unpatched 'vanilla-6.X.?' dir, and a linux-6.X.?-noarch hardlinked dir which has the Fedora patches applied. [1] WOW fedora patches! Sounds to me like they're not shipping vanilla. [1] https://fedoraproject.org/wiki/Kernel https://fedoraproject.org/wiki/Kernel
- 3y ago
- uxp8u61q 3y agoThis isn't a sports tournament, nobody is cheering or bashing "teams". If ZFS has bugs and caveats, they should be called out. Same for ext4.
- GauntletWizard 3y ago[flagged]
- baz00 3y agoWell said. Twisted partisanism. I’m the middle ground. It all sucks in one way or another but we’re generally trying to our best.
- reanimus 3y ago> When a core rust developer is exposed as a psychotic pedophile, there's crickets - it's not news because it happens every month. bro what on earth are you on about
- Lt_Riza_Hawkeye 3y agoThe problem here was Debian's distribution process. Any distro compiling linux from the releases on kernel.org was not affected.
- Blikkentrekker 3y agoSo this is another bug introduced by Debian itself by patching things? I remember that there was a fairly severe one which was caused by patching OpenSSL I think? But I remember the change they made being fairly weird and no one understood why but it was easy to see that it would introduce a vulnerability.
- mordae 3y agoNo, the issue was caused by backporting a patch A but not backporting a patch B. Sadly in this case the overall behavior after applying just A was broken when issuing direct IO writes.
- keep_reading 3y agoI remember when they made 2 CVEs back to back in Apache by incorrect back porting of changes
- remram 3y agoNo, it exists in the upstream release.
- asdfgasdfh 3y ago[dead]
- uselpa 3y agoAre you sure about that (genuine question)? The linked discussion involves a Suse engineer and a request to the kernel maintainer directly, not to a Debian packager-
- burtness 3y agoNo, its an upstream bug being discussed in the debian bug tracker
- 404mm 3y agoYou’re forgetting the btrfs folks. — someone who lost data due to btrfs bug
- nolist_policy 3y agoNote though that unlike zfs, ext4 has a battle-tested fsck. Combine ext4's dumb but robust approach to journaling and robust metadata layout (for example inodes are statically allocated) with fsck.ext4 (which got refined for years) and you can recover from any situation. To give you an example, fsck.ext4 will happily carve a working filesystem out of random data, as long as there is a valid superblock. Seriously - try it yourself: # create a working filesystem image dd if=/dev/zero of=test1.img bs=1M count=256 /sbin/mkfs.ext4 test1.img # write file with random data dd if=/dev/urandom of=test2.img bs=1M count=256 # copy superblock into random file dd if=test1.img of=test2.img bs=1024 count=4 seek=1 skip=1 conv=notrunc /sbin/fsck.ext4 -fy test2.img sudo mount test2.img /mnt/somewhere
- mustache_kimono 3y agoI'm trying to understand what exactly this is testing/demonstrating. Like: "Wow, but why should I care?" I'm not sure being able to fsck a random disk image shows resiliency. Doing this could do all sorts of nasty things to data you actually care about.
- grrandalf 3y ago(not the author) The metadata is a contiguous range of disk blocks. I think the intuition is that such layouts are likely to require simpler filesystem code to manipulate. [versus (iirc) ZFS which may have metadata blocks scattered throughout the disk, probably requiring more intricate code to keep track of].
- mustache_kimono 3y ago> [versus (iirc) ZFS which may have metadata blocks scattered throughout the disk, probably requiring more intricate code to keep track of]. So -- you think redundant metadata is a bad thing? Try wiping your metadata and then trying to fsck that random disk image. Again, has this been the source of data corruption you're aware of? This seems like a "Maybe, it could be this way, but I don't know" kind of take.
- 3y ago
- userbinator 3y agoSeverity: grave I'm not sure if that's a standard value, but it made me chuckle a little.
- arglebargle123 3y agoIt would have been really funny if they reported that one as: Severity: `
- asystole 3y agoAm I missing something? grave means very serious, which seems like an appropriate category.
- layer8 3y agoMaybe the GP read it as a noun.
- chrismorgan 3y agohttps://www.debian.org/Bugs/Developer#severities https://www.debian.org/Bugs/Developer#severities
- Animats 3y ago"Causes non-serious data loss". What does that mean. Only affects cat videos?
- minikomi 3y agoYou lose your data but, like, it's just a prank man.
- pestatije 3y agoyeah, practical joke for using Linux...have a laugh and deal with it
- viraptor 3y agoI believe the Debian policy for reports says that for corruption you can recover from. Serious data loss would be losing your metadata and not being able to figure out where any of the files are, or corrupting all copies of the disk encryption key.
- userbinator 3y agoI think "corruption you can recover from" seems far too vague; you can recover from any corruption with backups.
- piperswe 3y agoThe way I'd interpret it is that it's non-serious if fsck can repair it. Not sure if that's accurate though, just the way it sounds to me.
- viraptor 3y agoIf you're trying to be padantic and missing forest for the trees... Sure.
- LoganDark 3y agoSure, and you can recover from any hardware failure with a separate computer. In fact, you can recover from death simply by cloning yourself. What's the issue? The issue is that some people don't have the time, money, and/or willpower to invest in backups. Some people only have one volume, or one copy of each volume at least, and need to recover their data using only that one volume if something gets corrupted. That's what "corruption you can recover from" means. Using a backup is not recovering from corruption; it's replacing the corruption with the pre-corrupted data. Recovering from corruption involves actually transforming the corrupted data back into non-corrupted data.
- esbeeb 3y agoIn Debian bookworm: "sudo systemctl stop unattended-upgrades.service" ...wasn't able to prevent unattended-upgrades from going on ahead and just upgrading (to this problematic kernel) anyway. Unintuitively, the "right" way to disable unattended-upgrades is: "sudo dpkg-reconfigure unattended-upgrades" ...and choose "No" when asked.
- megous 3y agoMost likley because you stopped the wrong service. You should have stopped the relevant timer service, not the service that the timer starts.
- bsd_source 3y ago[flagged]
- esbeeb 3y agoCan you give example code to stop that timer?
- Timber-6539 3y agoUsually, it's the same service name but with .timer at the end instead of .service.
- KAMSPioneer 3y agoIt's the same, but you disable the timer instead of the service. `sudo systemctl stop unattended-upgrades.timer`
- yrro 3y agoIt will still start when you next boot up. 'systemctl disable --now unattended-upgrades.timer' will stop it now _and_ remove the symlink that starts it at boot.
- 3y ago
- sampa 3y agoHWE kernel (currently 6.2.0-37-generic #38~22.04.1-Ubuntu) on LTS Ubuntu 22.04 is not affected
- mgdhar 3y agoCurious why it doesn’t happen on GNU? Context: This is seen in the following environments: > > > * dragonboard-845c > > > * juno-64k_page_size > > > * qemu-arm64 > > > * qemu-armv7 > > > * qemu-i386 > > > * qemu-x86_64 > > > * x86_64-clang
- n3storm 3y agoThree months using kernel 6.6.5 from Zabbly (for hardware support) and having no issues: https://github.com/zabbly/linux https://github.com/zabbly/linux Quote "As those kernels aren't signed by a trusted distribution key, you may need to turn off Secure Boot on your system in order to boot this kernel."
- pjmlp 3y agoThe irony of seeing this, after Linus remarks how serious and thorough the file system kernel developers are, versus the other device drivers developers.
- tarruda 3y agoI wonder if Linus will publicly scold the developer responsible for this, like the famous "Mauro, STFU!" e-mail
- u_name 3y agoAren't this days over?
- mike_hock 3y agoIf they are, then that's why the quality is declining.
- gcr 3y ago“Verbal abuse is a necessary component of effective software reliability engineering!” “If my file system developer hasn’t been through the Full Metal Jacket experience, I don’t trust the code they write!” “Linux is going down the toilet! What we really need is leadership from a guy who’s VERY angry all the time!” Sure, giving good candid feedback is a gift, but degradation isn’t a necessary part of honesty.
- remram 3y agoFunny. You don't even know if that's the case, but somehow you know it's the cause?
- mike_hock 3y agoWeird. I had a few upvotes from people with a sense of humor before it went back down to 1 and these comments appeared.
- Kunix 3y agoHave any guidelines been posted somewhere for systems already updated to linux-image-amd64 6.1.64-1?
- HankB99 3y agoMy desktop and laptop are both root on ZFS so I dodged this bug. (Actually I saw the press on this before upgrading anyway so I'm holding off until I see a fix released.)
- Kunix 3y agoI saw the message yesterday late in the evening, I still had 8 hours before my automated updates ran and I thought the package would be revoked by then. Mistake :/
- shagrath 3y agoI also manually upgraded yesterday (bad luck, just a few hours after the update) and just now reverted to the previous kernel. Not sure this was the right thing to do but didn't want to take any chances
- AnAaaaardvark 3y agoMeanwhile my ntfs partitions are bugging pretty much all the time lately.
- fulafel 3y agoIs there some widely used software that does O_DIRECT writes? MariaDB?
- hedora 3y agoO_DIRECT is almost always the wrong choice. sync_file_range gives you much better control over scheduling of the write backs, and madvise gives you better control over caching policy. There were some old UNIX variants where O_DIRECT actually bypassed the filesystem cache, but Linux's cache is coherent, so reading a file immediately after an O_DIRECT write completes is guaranteed to give the new value. That is more sane than the cache-incoherent approach (how do you know your write is going to a page that is clean in OS cache in the other UNIXes?), but also eliminates most of the code-path-length benefit of O_DIRECT. Also, if I remember right, O_DIRECT doesn't bypass the I/O scheduler on Linux. That, and bypassing the cache are the two main benefits of the old API, and you get neither. As a bonus, filesystems like tmpfs passive-aggressively return error if you try to use O_DIRECT.
- jcalvinowens 3y ago> but Linux's cache is coherent Linux achieves that coherency by making O_DIRECT invalidate the page in the page cache. I think it paints a more accurate picture to say the caching is disabled with O_DIRECT, not that it is coherent (although it certainly is).
- jacooper 3y agoIt seems they have frozen the roll out of 6.1.0-14 too? Did they just freeze all Kernel Updates using `HTTP 403 - broken package ` as a response?
- Narishma 3y agoThat version is the one (or one of the ones) affected. I'm guessing that's why they froze it, though why they didn't replace it with a newer one, or even the old safe version I'm not sure.
- jacooper 3y agoHonestly the handling of this bug is horrendous to say the least, why can't they just retract the update? Why do people have to actively avoid updating to avoid this? This doesn't help debians reputation.
- leni536 3y agoSeems like I dodged this bullet by using Debian unstable, how ironic.
- frankjr 3y agoHere's my understanding so far: In the upstream Linux kernel there were two fixes posted months from each other, one for direct io [0] and the other one for ext4 [1]. The ext4 one was marked for backport to stable (CC: stable@vger.kernel.org), the other was not. The problem is that these commits depend on each other for things to work properly. If you have both, you're fine. If you have only the backported one, you have a problem. Which versions are affected? We know for sure that 6.1.64 is affected, 6.1.55 is not (because it doesn't have the commit). As of right now, 6.1.64 is still marked as "stable" in Debian [2] but if you actually try to install it from the official mirrors (deb.debian.org), you will get error 403. The fix is included in version 6.1.66 which will soon be available. The issue seems to be only highlighted in the context of Debian but it is not specific to it. The issue is/was in the official upstream release. [0] https://github.com/torvalds/linux/commit/936e114a245b6e38e0dbf706a67e7611fc993da1 https://github.com/torvalds/linux/commit/936e114a245b6e38e0d... [1] https://github.com/torvalds/linux/commit/91562895f8030cb9a0470b1db49de79346a69f91 https://github.com/torvalds/linux/commit/91562895f8030cb9a04... [2] https://packages.debian.org/source/stable/linux https://packages.debian.org/source/stable/linux
- tgsovlerkhgsel 3y agoWere any Ubuntu versions affected?
- frankjr 3y agoI think not unless you've explicitly installed the 6.1 series (none of the official supported releases ship with it by default as far as I can tell).
- frankjr 3y ago6.1.65 is affected as well.
- frankjr 3y agoDid you update to 6.1.66? Sucker! Your data might be safe(r) but your Wi-Fi is not working! The exact same thing happened again ([0] needs [1] but that one was not backported). Fixed in 6.1.67 [2] released this morning. [0] https://github.com/torvalds/linux/commit/7e7efdda6adb385fbdfd6f819d76bc68c923c394 https://github.com/torvalds/linux/commit/7e7efdda6adb385fbdf... [1] https://github.com/torvalds/linux/commit/076fc8775dafe995e94c106bb732bf2d42dedcea https://github.com/torvalds/linux/commit/076fc8775dafe995e94... [2] https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.67 https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.6...
- 6581 3y agoDupe: https://news.ycombinator.com/item?id=38585546 https://news.ycombinator.com/item?id=38585546
- userabchn 3y agoWhy wasn't automatic updating to this halted immediately? The "grave" bug report was yesterday, but today unattended-upgrades updated me to the affected kernel and I restarted before I heard about the problem.
- Kunix 3y agoI found a few things surprising about this bug too, but this might be due to my own ignorance about how these processes work: - I was expecting the package to either be revoked, or to have a new `6.1.64-2` version being published with the previously good known state. Maybe it wasn't done because it was not possible (mirrors being write-only, and maybe other complications in publishing a 6.1.64-2). - I was expecting some guidelines to be published for affected machines. I've seen questions being asked [1], but no answers to them yet (so users are not sure if it would be safe to rollback to the previous kernel version (6.1.55-1) or not). Maybe it's because the problem is not well understood yet, or maybe there are not enough people available to answer such questions on a weekend? If someone can provide some context about "how these things work", it would be interesting to learn more about this. [^1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057843#33 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057843#33
- Denvercoder9 3y agoThe `6.1.66-1` release with a fix was uploaded to Debian's infrastructure about 2 hours after the bug was filed, but it takes a bit for it to be compiled for all Debian's architectures (the mipsel builder e.g. needed 9 hours) and for it to propgate through the mirror network.
- Denvercoder9 3y agoThe way the Debian update process works doesn't make it possible to halt updates immediately. The servers you're getting the updates from aren't controlled by the Debian project itself, but are operated by third parties. They sync their data from a master machine (which is operated by Debian) 4 times per day. If you're using a mirror that's not under the debian.org domain, there might even be another machine or two in between, each adding some delay. So even if Debian pulled the update immediately, which their archive software isn't really setup to do, it'd take a bit for that to propgate through to the mirrors. If this all seems archaic, keep in mind that all this has been designed 20+ years ago, when bandwidth was quite a bit less abundant.
- ikawe 3y agoTo get past apt errors in the immediate (copied from the bottom of the bug report): > This should block just the buggy kernel. Which might help with unattended upgrades problem or just being forgetful. It might even uninstall the buggy kernel, though I didn't test that. And it shouldn't impact upgrading to 6.1.66 when it's available. > create a file: /etc/apt/preferences.d/buggy-kernel > with the contents: # avoid kernel with ext4 bug # 1057843 Package: linux-image-\* Pin: version 6.1.64-1 Pin-Priority: -1 > (the comment isn't required but is helpful for remembering why this file is around in 6 months.)
- jackyard86 3y agoThe reason I'm using Debian stable was to avoid these kind of problems. I should just move on to FreeBSD.
- lproven 3y agoDupe here: https://news.ycombinator.com/item?id=38585546 https://news.ycombinator.com/item?id=38585546