19 ms·

Regression in Linux 4.14 – using bcache can destroy the file system

201 points by panny 9y ago

rodgerd 9y ago
Reading the bug it appears it may affect more filesystems than merely bcache. Nasty.
- xwvvvvwx 9y ago
  pretty impressive response time though.
mtgx 9y ago
Linus worried so much about the new security features that he missed this one?
- jlgaddis 9y ago
  Surely you don't believe that Linus personally reviews and extensively tests every single patch that gets merged?
- __david__ 9y ago
  Here is the commit that introduces the bug: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=74d46992e0d9 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... Can you spot the error? Here is the commit that fixes the bug: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=62530ed8b1d https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
  - lomnakkus 9y ago
    Why is this a single commit? I would expect renaming like that to be done in completely separate commits and then subsequent commits to do any fixups. Is it just a merge commit? I imagine it's probably unfair to blame Linus for this (assuming it's a huge merge), but it would also weird for you to point to a merge commit as evidence of the opposite. Color me confused.
    - vog 9y ago
      Not sure why you are downvoted. I thought exactly the same thing after looking at the commit. As a software developer, the habit committing smaller changes has saved me a lot of time. Even more so, a clean distinction between refactorings (e.g. moving or renaming things) and functional changes (e.g. adding or fixing things) is vital to any large code base. As the saying goes: "for each desired change, make the change easy (warning: this may be hard), then make the easy change" (Kent Beck)
      - blueflow 9y ago
        I wonder what the creators of git think about your opinion on how to use it properly... Jokes aside, what you see it usually just the squashed commit submitted for merge. With 6 million git objects, you reconsider twice having each change in a single commit.
        bonzini 9y ago
        That's absolutely wrong. Linux never squashes merge commits. With my Linux maintainer hat on, I would have never accepted that commit.
        deleted 9y ago
        [deleted]
        lomnakkus 9y ago
        > I wonder what the creators of git think about your opinion on how to use it properly... Are you the creator of git? If not, how exactly are you qualified to judge "proper" usage -- by YOUR standard you're not qualified yourself, so...? (I realized it was a joke, but I offer the above as a counterpoint to the satirical/ironic element of the joke. My opinion is that the originator doesn't get special treatment. But then the point wasn't git, it was what/how things get merged into the Linux kernel. The mechanism is entirely immaterial.) Squashing commits for merge into Linux is not standard practice AFAIK... and happily I see an Linux committer has async'ly backed me up on this. Generally the preference I see, where git is concerned (not sure about the kernel), is that everything is rebased right into the master branch to avoid merge graphs getting absurd.
  - StreamBright 9y ago
    You are just proving the point that opensource wont protect against security bugs because most of the software engineers cannot spot a security bug in code?
    - mort96 9y ago
      I'm not sure if anyone has ever claimed that open source is a guaranteed protection against every security bug. Besides, this wasn't a security bug, was it? It corrupted the filesystem, it didn't give anyone else any undesired access did it?
    - PopsiclePete 9y ago
      And you are "proving the point" that humans are fallible and make mistakes and "open source" never meant, and can't mean, that the code is magically of better quality?
    - yjftsjthsd-h 9y ago
      Given the response time, I would say that this is an amazing example of the protection that open source provides.
    - chris_wot 9y ago
      What security bug? It was a corruption bug.
  - StillBored 9y ago
    A part of this is the "Blessed maintainer" issue, where large patchsets like this that a mere moral submits would end up going through dozens of review/testing cycles before even being considered by the maintainers of most subsystems. OTHO, the maintainer of said subsystem can write these huge patches, get one of his buddies to ack it, and its in the tree without much comment/testing. As this bug points out, being a maintainer doesn't suddenly make all your code bug free.. EDIT: OTOH, most of the Linus released kernels go through months of bug fixing before they show up in any of the stable distributions. Its the "rolling release" distro's which get hit by these bugs because they have this strange idea that once Linus (or whatever package) releases something its done cooking.
baobrien 9y ago
I lost data to this one after building a 4.14 kernel.
- harry8 9y ago
  Be positive. You learned an important lesson about using very recently released system software without having backups. You're a better engineer for it. Apply the lesson elsewhere and it's probably significantly more valuable than the lost data. "Experience is what you get when you didn't get what you want."
- Twirrim 9y ago
  If you operate on the bleeding edge, sometimes you're going to get cut.
xwvvvvwx 9y ago
Curious if anyone knows how the kernel is tested?
- klodolph 9y ago
  …I mean, yes? This question is well documented and simple Google searches will turn up a large number of high-quality results, such as https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested https://stackoverflow.com/questions/3177338/how-is-the-linux....
  - xwvvvvwx 9y ago
    Don't be boring. The nice thing about asking on hacker news is you get answers from experts, and you can often get insider knowledge that really isn't available anywhere on google. But thanks for the link, it's a good one ;)
    - klodolph 9y ago
      Please don't accuse me of being a boring person. I'm not sure why you think that is acceptable behavior. The nice thing about doing a minute of preliminary research when you have a question is that the questions you ask afterwards will be more interesting.
- simcop2387 9y ago
  As best as they can, and by a lot of people running release candidates on test systems. Unfortunately there are basically an infinite (not really, just incredibly impossibly large) number of configurations of software and hardware that can lead to a bug like this not triggering so it's not always possible to catch this ahead of time.
  - xwvvvvwx 9y ago
    How often does a bug of this severity slip through the net?
Jach 9y ago
I really love Gentoo's mask system. I upgraded to 4.12.12 not too long ago, it is the latest stable: https://packages.gentoo.org/packages/sys-kernel/gentoo-sources https://packages.gentoo.org/packages/sys-kernel/gentoo-sourc... You have to explicitly decide to use masked versions, at your own peril. (Of course, many Gentoo users love the bleeding edge, and I too have a number of things I manually unmasked... But the kernel? Stable only please.)
- whacker 9y ago
  Debian/Ubuntu has a similar mechanism. http://manpages.ubuntu.com/manpages/xenial/man8/apt-mark.8.html http://manpages.ubuntu.com/manpages/xenial/man8/apt-mark.8.h...
  - dfc 9y ago
    apt-mark is nothing like masking from gentoo. A-M is just for making modifications to the database that remembers whether a package was intentionally installed or was automatically pulled in as part of a dependency. This database is how apt generates the list of "packages that were automatically installed and no longer needed." The closest thing that I can think of to masking in Debi an is pulling in a package from experimental or a PPA that you added. There are no packages in the repo (stable, testing, unstable) you are using that require anything other than "apt-get install package" to install them.
    - anderskaseorg 9y ago
      You have described apt-mark {auto|manual}, but there is also apt-mark {hold|unhold}.
      - leni536 9y ago
        Damn, I always abused apt-preferences to achieve the same effect as apt-mark hold|unhold. Thanks! Does 'apt-get dist-upgrade' respect held packages?
X86BSD 9y ago
I... WOW. This one is epic. I don’t even... wow.
- chris_wot 9y ago
  Yeah, epic in how fast it got fixed!
- snvzz 9y ago
  On a Linux .0 release, it's not that uncommon for fs corruption bugs. I remember quite a few of these. Best to only upgrade Linux when the next release is already out (e.g. 4.15 out to consider 4.14). Or just use something else.
jnwatson 9y ago
In case anyone is wondering, Bcache allows one to use an SSD as a read/write cache (in writeback mode) or read cache (writethrough or writearound) for another blockdevice (generally a rotating HDD or array).
- SteveNuts 9y ago
  Is it possible to use bcache in front of an NFS volume?
  - khc 9y ago
    No but you can already use FS-cache?
    - ptman 9y ago
      fs-cache / cachefilesd seems to still be buggy. was it even deprecated?
      - khc 9y ago
        Not that I know of. You can also try https://github.com/kahing/catfs https://github.com/kahing/catfs which I wrote recently but I don't promise it's not buggy :-)
  - kogepathic 9y ago
    > Is it possible to use bcache in front of an NFS volume? No, because bcache only works for a block device, whereas NFS presents itself as a pseudo filesystem (NFS) to the target. You could however use bcache as a local accelerator for a remote iSCSI target.
pcunite 9y ago
So, and single line of code change? This is why the language choice is not the biggest issue here. Keeping all the necessary state in a programmer's head is the real issue. Using <insert fav lang>Rust</> would not have helped. bio->bi_partno = bio_src->bi_partno; https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=62530ed8b1d https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- khuey 9y ago
  A language with copy constructors could have prevented this.
  - std_throwaway 9y ago
    C++ would be a logical language upgrade for the kernel.
    - MBCook 9y ago
      Why is that?
    - cdmckay 9y ago
      Read Linus' rant about why the Linux kernel shouldn't use C++: https://lwn.net/Articles/249460/ https://lwn.net/Articles/249460/
      - steveklabnik 9y ago
        Thu, 6 Sep 2007 18:50:28 +0100 (BST) C++ has changed quite a bit in the last 10 years.
        iforgotpassword 9y ago
        Yep, adding more and more bs so it's impossible to fully understand it ever. Understanding C thoroughly is already hard, but C++ is a whole new level. "But then only use the new and modern parts of it" No! You cannot simply ignore all the complicated cruft just because you don't like it because sooner or later it will come and bite you in the ass.
pkaye 9y ago
How come none of the static analyzers that seems to be run on the Linux source code not catch this?
- caf 9y ago
  Because the field wasn't formally uninitialised - it was initialised to zero by bio_init(). It was instead a logic error, which are not as amenable to static analysis.
nategri 9y ago
Slightly OT but I'm happy to see breaking news from a gentoo.org domain. I long ago decided installing Gentoo was a young person's pursuit, but I'll always have fond feelings for it.
- dguaraglia 9y ago
  You can always try Arch, it's like the more-adult version of Gentoo. Still a lot of work though.
  - mohaine 9y ago
    Agreed but Arch isn't much work after the initial install. Well worth it for the rolling updates. Arch mostly just works. The big issue I run into is that the upgrades are _almost_ flawless, which bites you once a year or so when you least expect it.
    - nerflad 9y ago
      It becomes slightly more work if you depend on AUR packages (especially if you end up maintaining the packages you depend on!), but overall I think it's a great daily-driver system.
    - dguaraglia 9y ago
      I might give it another try when I buy my next laptop. Last time I tried there was way too much text-file tweaking for my taste (years of OS X have made me soft.)
  - hawski 9y ago
    When I was a teenager I used Gentoo for a couple of years. I'm glad that I did, because I learned a lot. Of course it became tiresome after some time. However now I think about Gentoo as a more-adult one. It's really a meta-distribution. It was and is misunderstood by a lot of people. Teenage me included. Gentoo is deceivingly easy to setup - documentation is excellent and tooling mature. I think that the best way to use Gentoo is to really set all the USE flags and all the unmasking to one's will, but not for a single computer. There should be a binhost - a binary package server available for whole fleet of computers. You can also cross compile for a totally different architecture. Then the customization comes in handy. Also now it's certainly easier than when I've done it. I used Gentoo on a desktop with a whooping 512MB RAM. Now compile times are significantly lower. To sum it up: use Gentoo to build a custom Linux distribution for fleet of computers or for embedded devices. That's it's true power. Google chose Portage for ChromeOS building for a reason.

201 points by panny 9y ago

rodgerd 9y ago

Reading the bug it appears it may affect more filesystems than merely bcache. Nasty.

xwvvvvwx 9y ago
pretty impressive response time though.

mtgx 9y ago

Linus worried so much about the new security features that he missed this one?

jlgaddis 9y ago
Surely you don't believe that Linus personally reviews and extensively tests every single patch that gets merged?
__david__ 9y ago
Here is the commit that introduces the bug: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=74d46992e0d9 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... Can you spot the error? Here is the commit that fixes the bug: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=62530ed8b1d https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- lomnakkus 9y ago
  Why is this a single commit? I would expect renaming like that to be done in completely separate commits and then subsequent commits to do any fixups. Is it just a merge commit? I imagine it's probably unfair to blame Linus for this (assuming it's a huge merge), but it would also weird for you to point to a merge commit as evidence of the opposite. Color me confused.
  - vog 9y ago
    Not sure why you are downvoted. I thought exactly the same thing after looking at the commit. As a software developer, the habit committing smaller changes has saved me a lot of time. Even more so, a clean distinction between refactorings (e.g. moving or renaming things) and functional changes (e.g. adding or fixing things) is vital to any large code base. As the saying goes: "for each desired change, make the change easy (warning: this may be hard), then make the easy change" (Kent Beck)
    - blueflow 9y ago
      I wonder what the creators of git think about your opinion on how to use it properly... Jokes aside, what you see it usually just the squashed commit submitted for merge. With 6 million git objects, you reconsider twice having each change in a single commit.
      - bonzini 9y ago
        That's absolutely wrong. Linux never squashes merge commits. With my Linux maintainer hat on, I would have never accepted that commit.
        deleted 9y ago
        [deleted]
      - lomnakkus 9y ago
        > I wonder what the creators of git think about your opinion on how to use it properly... Are you the creator of git? If not, how exactly are you qualified to judge "proper" usage -- by YOUR standard you're not qualified yourself, so...? (I realized it was a joke, but I offer the above as a counterpoint to the satirical/ironic element of the joke. My opinion is that the originator doesn't get special treatment. But then the point wasn't git, it was what/how things get merged into the Linux kernel. The mechanism is entirely immaterial.) Squashing commits for merge into Linux is not standard practice AFAIK... and happily I see an Linux committer has async'ly backed me up on this. Generally the preference I see, where git is concerned (not sure about the kernel), is that everything is rebased right into the master branch to avoid merge graphs getting absurd.
- StreamBright 9y ago
  You are just proving the point that opensource wont protect against security bugs because most of the software engineers cannot spot a security bug in code?
  - mort96 9y ago
    I'm not sure if anyone has ever claimed that open source is a guaranteed protection against every security bug. Besides, this wasn't a security bug, was it? It corrupted the filesystem, it didn't give anyone else any undesired access did it?
  - PopsiclePete 9y ago
    And you are "proving the point" that humans are fallible and make mistakes and "open source" never meant, and can't mean, that the code is magically of better quality?
  - yjftsjthsd-h 9y ago
    Given the response time, I would say that this is an amazing example of the protection that open source provides.
  - chris_wot 9y ago
    What security bug? It was a corruption bug.
- StillBored 9y ago
  A part of this is the "Blessed maintainer" issue, where large patchsets like this that a mere moral submits would end up going through dozens of review/testing cycles before even being considered by the maintainers of most subsystems. OTHO, the maintainer of said subsystem can write these huge patches, get one of his buddies to ack it, and its in the tree without much comment/testing. As this bug points out, being a maintainer doesn't suddenly make all your code bug free.. EDIT: OTOH, most of the Linus released kernels go through months of bug fixing before they show up in any of the stable distributions. Its the "rolling release" distro's which get hit by these bugs because they have this strange idea that once Linus (or whatever package) releases something its done cooking.

baobrien 9y ago

I lost data to this one after building a 4.14 kernel.

harry8 9y ago
Be positive. You learned an important lesson about using very recently released system software without having backups. You're a better engineer for it. Apply the lesson elsewhere and it's probably significantly more valuable than the lost data. "Experience is what you get when you didn't get what you want."
Twirrim 9y ago
If you operate on the bleeding edge, sometimes you're going to get cut.

xwvvvvwx 9y ago

Curious if anyone knows how the kernel is tested?

klodolph 9y ago
…I mean, yes? This question is well documented and simple Google searches will turn up a large number of high-quality results, such as https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested https://stackoverflow.com/questions/3177338/how-is-the-linux....
- xwvvvvwx 9y ago
  Don't be boring. The nice thing about asking on hacker news is you get answers from experts, and you can often get insider knowledge that really isn't available anywhere on google. But thanks for the link, it's a good one ;)
  - klodolph 9y ago
    Please don't accuse me of being a boring person. I'm not sure why you think that is acceptable behavior. The nice thing about doing a minute of preliminary research when you have a question is that the questions you ask afterwards will be more interesting.
simcop2387 9y ago
As best as they can, and by a lot of people running release candidates on test systems. Unfortunately there are basically an infinite (not really, just incredibly impossibly large) number of configurations of software and hardware that can lead to a bug like this not triggering so it's not always possible to catch this ahead of time.
- xwvvvvwx 9y ago
  How often does a bug of this severity slip through the net?

Jach 9y ago

I really love Gentoo's mask system. I upgraded to 4.12.12 not too long ago, it is the latest stable: https://packages.gentoo.org/packages/sys-kernel/gentoo-sources https://packages.gentoo.org/packages/sys-kernel/gentoo-sourc... You have to explicitly decide to use masked versions, at your own peril. (Of course, many Gentoo users love the bleeding edge, and I too have a number of things I manually unmasked... But the kernel? Stable only please.)

whacker 9y ago
Debian/Ubuntu has a similar mechanism. http://manpages.ubuntu.com/manpages/xenial/man8/apt-mark.8.html http://manpages.ubuntu.com/manpages/xenial/man8/apt-mark.8.h...
- dfc 9y ago
  apt-mark is nothing like masking from gentoo. A-M is just for making modifications to the database that remembers whether a package was intentionally installed or was automatically pulled in as part of a dependency. This database is how apt generates the list of "packages that were automatically installed and no longer needed." The closest thing that I can think of to masking in Debi an is pulling in a package from experimental or a PPA that you added. There are no packages in the repo (stable, testing, unstable) you are using that require anything other than "apt-get install package" to install them.
  - anderskaseorg 9y ago
    You have described apt-mark {auto|manual}, but there is also apt-mark {hold|unhold}.
    - leni536 9y ago
      Damn, I always abused apt-preferences to achieve the same effect as apt-mark hold|unhold. Thanks! Does 'apt-get dist-upgrade' respect held packages?

X86BSD 9y ago

I... WOW. This one is epic. I don’t even... wow.

chris_wot 9y ago
Yeah, epic in how fast it got fixed!
snvzz 9y ago
On a Linux .0 release, it's not that uncommon for fs corruption bugs. I remember quite a few of these. Best to only upgrade Linux when the next release is already out (e.g. 4.15 out to consider 4.14). Or just use something else.

jnwatson 9y ago

In case anyone is wondering, Bcache allows one to use an SSD as a read/write cache (in writeback mode) or read cache (writethrough or writearound) for another blockdevice (generally a rotating HDD or array).

SteveNuts 9y ago
Is it possible to use bcache in front of an NFS volume?
- khc 9y ago
  No but you can already use FS-cache?
  - ptman 9y ago
    fs-cache / cachefilesd seems to still be buggy. was it even deprecated?
    - khc 9y ago
      Not that I know of. You can also try https://github.com/kahing/catfs https://github.com/kahing/catfs which I wrote recently but I don't promise it's not buggy :-)
- kogepathic 9y ago
  > Is it possible to use bcache in front of an NFS volume? No, because bcache only works for a block device, whereas NFS presents itself as a pseudo filesystem (NFS) to the target. You could however use bcache as a local accelerator for a remote iSCSI target.

pcunite 9y ago

So, and single line of code change? This is why the language choice is not the biggest issue here. Keeping all the necessary state in a programmer's head is the real issue. Using <insert fav lang>Rust</> would not have helped. bio->bi_partno = bio_src->bi_partno; https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=62530ed8b1d https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

khuey 9y ago
A language with copy constructors could have prevented this.
- std_throwaway 9y ago
  C++ would be a logical language upgrade for the kernel.
  - MBCook 9y ago
    Why is that?
  - cdmckay 9y ago
    Read Linus' rant about why the Linux kernel shouldn't use C++: https://lwn.net/Articles/249460/ https://lwn.net/Articles/249460/
    - steveklabnik 9y ago
      Thu, 6 Sep 2007 18:50:28 +0100 (BST) C++ has changed quite a bit in the last 10 years.
      - iforgotpassword 9y ago
        Yep, adding more and more bs so it's impossible to fully understand it ever. Understanding C thoroughly is already hard, but C++ is a whole new level. "But then only use the new and modern parts of it" No! You cannot simply ignore all the complicated cruft just because you don't like it because sooner or later it will come and bite you in the ass.

pkaye 9y ago

How come none of the static analyzers that seems to be run on the Linux source code not catch this?

caf 9y ago
Because the field wasn't formally uninitialised - it was initialised to zero by bio_init(). It was instead a logic error, which are not as amenable to static analysis.

nategri 9y ago

Slightly OT but I'm happy to see breaking news from a gentoo.org domain. I long ago decided installing Gentoo was a young person's pursuit, but I'll always have fond feelings for it.

dguaraglia 9y ago
You can always try Arch, it's like the more-adult version of Gentoo. Still a lot of work though.
- mohaine 9y ago
  Agreed but Arch isn't much work after the initial install. Well worth it for the rolling updates. Arch mostly just works. The big issue I run into is that the upgrades are _almost_ flawless, which bites you once a year or so when you least expect it.
  - nerflad 9y ago
    It becomes slightly more work if you depend on AUR packages (especially if you end up maintaining the packages you depend on!), but overall I think it's a great daily-driver system.
  - dguaraglia 9y ago
    I might give it another try when I buy my next laptop. Last time I tried there was way too much text-file tweaking for my taste (years of OS X have made me soft.)
- hawski 9y ago
  When I was a teenager I used Gentoo for a couple of years. I'm glad that I did, because I learned a lot. Of course it became tiresome after some time. However now I think about Gentoo as a more-adult one. It's really a meta-distribution. It was and is misunderstood by a lot of people. Teenage me included. Gentoo is deceivingly easy to setup - documentation is excellent and tooling mature. I think that the best way to use Gentoo is to really set all the USE flags and all the unmasking to one's will, but not for a single computer. There should be a binhost - a binary package server available for whole fleet of computers. You can also cross compile for a totally different architecture. Then the customization comes in handy. Also now it's certainly easier than when I've done it. I used Gentoo on a desktop with a whooping 512MB RAM. Now compile times are significantly lower. To sum it up: use Gentoo to build a custom Linux distribution for fleet of computers or for embedded devices. That's it's true power. Google chose Portage for ChromeOS building for a reason.