7 ms·

Sometimes it actually is a kernel bug: bind() in Linux 6.0.16

162 points by zdw 4y ago

vfclists 4y ago
How did this happen? Which commit caused it?
- smashed 4y ago
  Best explanation I found is here: https://lore.kernel.org/stable/CAFsF8vL4CGFzWMb38_XviiEgxoKX0GYup=JiUFXUOmagdk9CRg@mail.gmail.com/ https://lore.kernel.org/stable/CAFsF8vL4CGFzWMb38_XviiEgxoKX... A patch was backported to the 6.0 branch from the main branch, but they forgot a line of code, leading to a buggy behavior.
  - yjftsjthsd-h 4y ago
    So that might suggest that it's actually better to just track the latest version, rather than worrying about backports?
    - CJefferson 4y ago
      In my (limited) experience, the only reason to use backports is because you have closed source kernel modules which you can't update (that of course ends up covering most Android phones, and many SOCs)
      - touisteur 4y ago
        There's also official support of vmm things like firecracker, which officially supports only 5.10 and maybe latest but don't send bugs?
    - ilyt 4y ago
      Distros standardize on version not because it is more stable but because tooling (which might include 3rd party modules for the kernel) can then rely to work on that version without recompile. If you don't have that constraint yeah, not much reason.
    - eklitzke 4y ago
      The vast majority of the time fixes like this that are being backported are straightforward fixes for bugs (security or otherwise) that require very little manual conflict resolution, especially if the fix is just being backported one or two kernel releases. The developer can often just cherry-pick the commit into a few recent release branches and most of the time git will just automatically do the merge correctly, or if there is a manual merge conflict it's something really simple. In fact, if there's a complicated merge conflict often the change won't be backported at all unless the bug is actually serious enough to warrant X hours of someone's time to do it and get code review etc. Most of the time this process works correctly, but obviously there's room for error and mistakes can happen. There's a tradeoff between the risk of running an older kernel that has known bugs, upgrading to the latest new kernel which has bug fixes but may introduce new bugs, and getting backports for known bugs to your known working kernel. Most of the time the last option is reasonable but it definitely depends on your use case and what you're optimizing for.
  - vfclists 4y ago
    This soooo BASIC!! Why aren't the processes either manual or automated in place to check for things like this? Aren't there some tests in place to check for such basic functionality errors? Doesn't kernel development process mandate much facilities? I'm sure the NSA, Unit 8200 and GCHQ have tests like this in place but don't share their findings. Is it a matter of funding or leadership philosophy and priorities?
    - cdelsolar 4y ago
      yeah right? doesn't the kernel have a test suite?
      - bonzini 4y ago
        Every kernel subsystem has its own testsuite. Running all of them would requires hundreds of different pieces of hardware, so it's not really possible for a single release manager to do so. For Linus's releases this is easily solved by slowing down progressively the pace of development towards a release, so that cross-subsystem issues where maintainer A breaks maintainer B's subsystem become progressively less likely over the two months of the release cycle. For stable releases this is much harder to do because of the short cycle. The stable branches in the end are a mostly automated collection of patches based on both maintainer input and the output of a machine learning model. The quality of stable branches is generally pretty good, or screwups such as this one would not make a headline; but that's more a result of discipline of mainline kernel development, rather than a virtue of the stable kernel release process.
        vfclists 4y ago
        > Running all of them would requires hundreds of different pieces of hardware, so it's not really possible for a single release manager to do so. The issue is this bug is not hardware related. Its a pure software issue. Hardware bugs are an entirely different kettle of fish. BTW is that bonzini of GNU Smalltalk fame?
        4y ago
  - bravetraveler 4y ago
    I'll open by saying I'll forever be thankful for GKH... but this response kills me: > As 6.0.y is now end-of-life, is there anything keeping you on that kernel tree? Uh, several distributions. It wasn't EOL enough to prevent breaking it, so fix it. Don't even technically need their input, Git and all. I'll buy this EOL thing if they revert the change that caused this and stop releasing under 6.0. There were at least two more after this
    - Arnavion 4y ago
      >Uh, several distributions. That is the kernel bug tracker, not the distributions bug tracker. >It wasn't EOL enough to prevent breaking it, so fix it. It wasn't EOL at the time the patch was backported. It's EOL now. >I'll buy this EOL thing if they revert the change that caused this and stop releasing under 6.0. There were at least two more after this Not sure what "this" in "two more after this" is, but there have been no 6.0 releases since it was EOLed.
      - bravetraveler 4y ago
        > That is the kernel bug tracker, not the distributions bug tracker. The point is that the contributors reasoning for being on the tree is irrelevant. Like you said, they just made it EOL Distributions are the continuous/constant answer as to why countless people will be. This isn't an ancient release, something from the grave. Is the expectation, then, that distributions would have to patch out the regression - or take on a more major upgrade (6.1 / 6.2), likely breaking something else? Neither of these are particularly tenable. I'm glad GKH was willing to accept further changes to make it correct, but reverting is also applicable. Breaking something, calling it EOL, and not fixing it is closer to dead than end of life.
        Arnavion 4y ago
        >Is the expectation, then, that distributions would have to patch out the regression - or take on a more major upgrade (6.1 / 6.2), likely breaking something else? Correct. >Neither of these are particularly tenable. Yes they are. >Breaking something, calling it EOL, and not fixing it is closer to dead than end of life. You're awfully confident about how things should work, even though you don't understand how they already work.
sph 4y ago
Not the first annoying networking bug introduced in 6.0.16 — if you have a CIFS mount, you get a kernel panic. There's a confirmed bug report about it on Red Hat's and the kernel's bugzilla. They don't seem to be related, so it's just coincidence.
mjw1007 4y ago
Allegedly this was the patch introducing the bug: https://lore.kernel.org/stable/20221228144337.512799851@linuxfoundation.org/ https://lore.kernel.org/stable/20221228144337.512799851@linu... It seems to me that for a long time now stable releases haven't been trying very hard to follow the stated policy [1], in particular the parts that say > It must fix a real bug that bothers people (not a, "This could be a problem..." type thing). and > It must fix a problem that causes a build error ([...]), an oops, a hang, data corruption, a real security issue, or some "oh, that’s not good" issue. In short, something critical. [1]: https://docs.kernel.org/process/stable-kernel-rules.html https://docs.kernel.org/process/stable-kernel-rules.html
- bravetraveler 4y ago
  You know, this is a really good point. The last three 'stable' releases have contained an annoying number of refactors and fixes for amdgpu in particular Play with PowerPlay tables in staging, I haven't been able to upgrade
  - bravetraveler 4y ago
    I can't edit now but a note for posterity... when I say 'staging' I actually suggest mainline or linux-next, whichever is actually more appropriate Stable seems an odd place /shrug Those 'fixes' may have fixed what they looked at, but I'm still broken. Feels more like rearranging chairs than actually fixing
- 77pt77 4y ago
  That's an absolutely awful policy! It should fix an actual problem, even if not found until now.
  - CodesInChaos 4y ago
    Why do you think fixes for minor problems should be backported to stable kernels?
    - 4y ago
dmarlow 4y ago
How is testing done in kernels? Is there unit testing, integration, end to end? I'm unfamiliar, but curious.
- seanhunter 4y ago
  Not in the Linux kernel. In the Linux kernel it has traditionally always been done by the developer sticking lots of printk statements into their code until they are happy with the outputs and then removing most of them. Make of that what you will.
  - bonzini 4y ago
    LOL that is so false that is not even worth correcting.
    - marcellus23 4y ago
      Actually could you correct it? I don't know anything about the Linux kernel so I'm curious.
      - seanhunter 4y ago
        I'll correct myself. That was how it was done circa 2000-2005, but my info was way out of date because I haven't followed kernel development much for a while. Here's some stuff about how it's done now. https://www.kernel.org/doc/html/latest/dev-tools/testing-overview.html https://www.kernel.org/doc/html/latest/dev-tools/testing-ove... Basically there is a system called KUnit for writing white box unit tests and there is code coverage to determine coverage as you would expect, then there are systems for static analysis and verifying assertions.
    - deleted 4y ago
      [deleted]
  - vfclists 4y ago
    This is so wrong!!
helmsb 4y ago
“ There's a common saying and rule of thumb in programming (possibly originating in the C world) that it's never a compiler bug, it's going to be a bug in your code even if it looks crazy or impossible.” I say, it’s “rarely a compiler bug.” When teaching new developers I remind them that you should assume the bug is in the code you just wrote before jumping to the bug being a compiler, framework or OS issue. It’s just a matter of probabilities.
- plumefar 4y ago
  In my +- 25 years of experience, I've encountered exactly 1 compiler bug. That cost me a full week of debug. Unfortunately, it happened during my very first year as a developer, and that triggered sub-optimal debugging practices (ie: is that the compiler fault again?) At the end of the day, you want to optimise for debug time. There's the probability that it's a compiler/developer bug (respectively very low/very high) and the time it takes to rule it out. It's of course best NOT to start by investigating the compiler bug.
  - bondant 4y ago
    I think it's really dependent of the domain you are working in. Some stuff is better tested than others. In my 5 years careers I found bugs (found and confirmed by compiler authors) in gfortran, xlf and pgfortran. So yes bugs in compilers are rare but sometimes you just keep running into them.
    - dtech 4y ago
      those are quite niche languages though, most people use compilers which are much more widely used.
      - capitalsigma 4y ago
        Clang support for AVX-512 intrinsics is pretty broken
  - CameronNemo 4y ago
mrtweetyhack 4y ago
[dead]
nikanj 4y ago
The biggest mindset adjustment I ever had to make was moving from C++ to Node.js. You can usually trust that select() is not broken, but select.js is deprecated, select-kitchensink.js is not compatible with your toolchain and unicyclect.ts leaks memory like a sieve
koala_man 4y ago
> it's never a compiler bug I wish this was true. It would have made my job of writing compilers so much easier.
- moloch-hai 4y ago
  Out of all the compiler bugs I have failed reports against over the years, only one was fixed. And the fix was applied only in the "trunk" branch, not any of the other ones still "maintained". I learned that Gcc has a policy not to fix performance regressions on any but the development branch. Back in the late '80s we spent fully half of each working day tracking down the trigger and workaround for cfront crashes. Kids today have it easy.
- lostmsu 4y ago
  I think you misunderstand. It is never the bug in the compiler of your compiler.
tedunangst 4y ago
It's encouraging to see so many new volunteers signing up to help maintain and test the stable branch!
netheril96 4y ago
For curiosity I upgraded the kernel of my VPS running Debian 11 from 5.10 to 6.1. The networking became visibly slower. All TCP receive and send rate droped to at least 1/4 of the original. I had to restore the original kernel and the network became fine again. I don't think it is a bug in the kernel. More likely the userspace or the host system needs to be reconfigured to work well with the bleeding edge kernel. Still I am curious, what exactly may cause this dramatic drop?
yencabulator 4y ago
> it's never a compiler bug Back in the days of ISO-8859-1, you could make gcc crash by having the string literal "ä" in the source code...

162 points by zdw 4y ago

vfclists 4y ago

How did this happen? Which commit caused it?

smashed 4y ago
Best explanation I found is here: https://lore.kernel.org/stable/CAFsF8vL4CGFzWMb38_XviiEgxoKX0GYup=JiUFXUOmagdk9CRg@mail.gmail.com/ https://lore.kernel.org/stable/CAFsF8vL4CGFzWMb38_XviiEgxoKX... A patch was backported to the 6.0 branch from the main branch, but they forgot a line of code, leading to a buggy behavior.
- yjftsjthsd-h 4y ago
  So that might suggest that it's actually better to just track the latest version, rather than worrying about backports?
  - CJefferson 4y ago
    In my (limited) experience, the only reason to use backports is because you have closed source kernel modules which you can't update (that of course ends up covering most Android phones, and many SOCs)
    - touisteur 4y ago
      There's also official support of vmm things like firecracker, which officially supports only 5.10 and maybe latest but don't send bugs?
  - ilyt 4y ago
    Distros standardize on version not because it is more stable but because tooling (which might include 3rd party modules for the kernel) can then rely to work on that version without recompile. If you don't have that constraint yeah, not much reason.
  - eklitzke 4y ago
    The vast majority of the time fixes like this that are being backported are straightforward fixes for bugs (security or otherwise) that require very little manual conflict resolution, especially if the fix is just being backported one or two kernel releases. The developer can often just cherry-pick the commit into a few recent release branches and most of the time git will just automatically do the merge correctly, or if there is a manual merge conflict it's something really simple. In fact, if there's a complicated merge conflict often the change won't be backported at all unless the bug is actually serious enough to warrant X hours of someone's time to do it and get code review etc. Most of the time this process works correctly, but obviously there's room for error and mistakes can happen. There's a tradeoff between the risk of running an older kernel that has known bugs, upgrading to the latest new kernel which has bug fixes but may introduce new bugs, and getting backports for known bugs to your known working kernel. Most of the time the last option is reasonable but it definitely depends on your use case and what you're optimizing for.
- vfclists 4y ago
  This soooo BASIC!! Why aren't the processes either manual or automated in place to check for things like this? Aren't there some tests in place to check for such basic functionality errors? Doesn't kernel development process mandate much facilities? I'm sure the NSA, Unit 8200 and GCHQ have tests like this in place but don't share their findings. Is it a matter of funding or leadership philosophy and priorities?
  - cdelsolar 4y ago
    yeah right? doesn't the kernel have a test suite?
    - bonzini 4y ago
      Every kernel subsystem has its own testsuite. Running all of them would requires hundreds of different pieces of hardware, so it's not really possible for a single release manager to do so. For Linus's releases this is easily solved by slowing down progressively the pace of development towards a release, so that cross-subsystem issues where maintainer A breaks maintainer B's subsystem become progressively less likely over the two months of the release cycle. For stable releases this is much harder to do because of the short cycle. The stable branches in the end are a mostly automated collection of patches based on both maintainer input and the output of a machine learning model. The quality of stable branches is generally pretty good, or screwups such as this one would not make a headline; but that's more a result of discipline of mainline kernel development, rather than a virtue of the stable kernel release process.
      - vfclists 4y ago
        > Running all of them would requires hundreds of different pieces of hardware, so it's not really possible for a single release manager to do so. The issue is this bug is not hardware related. Its a pure software issue. Hardware bugs are an entirely different kettle of fish. BTW is that bonzini of GNU Smalltalk fame?
        4y ago
- bravetraveler 4y ago
  I'll open by saying I'll forever be thankful for GKH... but this response kills me: > As 6.0.y is now end-of-life, is there anything keeping you on that kernel tree? Uh, several distributions. It wasn't EOL enough to prevent breaking it, so fix it. Don't even technically need their input, Git and all. I'll buy this EOL thing if they revert the change that caused this and stop releasing under 6.0. There were at least two more after this
  - Arnavion 4y ago
    >Uh, several distributions. That is the kernel bug tracker, not the distributions bug tracker. >It wasn't EOL enough to prevent breaking it, so fix it. It wasn't EOL at the time the patch was backported. It's EOL now. >I'll buy this EOL thing if they revert the change that caused this and stop releasing under 6.0. There were at least two more after this Not sure what "this" in "two more after this" is, but there have been no 6.0 releases since it was EOLed.
    - bravetraveler 4y ago
      > That is the kernel bug tracker, not the distributions bug tracker. The point is that the contributors reasoning for being on the tree is irrelevant. Like you said, they just made it EOL Distributions are the continuous/constant answer as to why countless people will be. This isn't an ancient release, something from the grave. Is the expectation, then, that distributions would have to patch out the regression - or take on a more major upgrade (6.1 / 6.2), likely breaking something else? Neither of these are particularly tenable. I'm glad GKH was willing to accept further changes to make it correct, but reverting is also applicable. Breaking something, calling it EOL, and not fixing it is closer to dead than end of life.
      - Arnavion 4y ago
        >Is the expectation, then, that distributions would have to patch out the regression - or take on a more major upgrade (6.1 / 6.2), likely breaking something else? Correct. >Neither of these are particularly tenable. Yes they are. >Breaking something, calling it EOL, and not fixing it is closer to dead than end of life. You're awfully confident about how things should work, even though you don't understand how they already work.

sph 4y ago

Not the first annoying networking bug introduced in 6.0.16 — if you have a CIFS mount, you get a kernel panic. There's a confirmed bug report about it on Red Hat's and the kernel's bugzilla. They don't seem to be related, so it's just coincidence.

mjw1007 4y ago

Allegedly this was the patch introducing the bug: https://lore.kernel.org/stable/20221228144337.512799851@linuxfoundation.org/ https://lore.kernel.org/stable/20221228144337.512799851@linu... It seems to me that for a long time now stable releases haven't been trying very hard to follow the stated policy [1], in particular the parts that say > It must fix a real bug that bothers people (not a, "This could be a problem..." type thing). and > It must fix a problem that causes a build error ([...]), an oops, a hang, data corruption, a real security issue, or some "oh, that’s not good" issue. In short, something critical. [1]: https://docs.kernel.org/process/stable-kernel-rules.html https://docs.kernel.org/process/stable-kernel-rules.html

bravetraveler 4y ago
You know, this is a really good point. The last three 'stable' releases have contained an annoying number of refactors and fixes for amdgpu in particular Play with PowerPlay tables in staging, I haven't been able to upgrade
- bravetraveler 4y ago
  I can't edit now but a note for posterity... when I say 'staging' I actually suggest mainline or linux-next, whichever is actually more appropriate Stable seems an odd place /shrug Those 'fixes' may have fixed what they looked at, but I'm still broken. Feels more like rearranging chairs than actually fixing
77pt77 4y ago
That's an absolutely awful policy! It should fix an actual problem, even if not found until now.
- CodesInChaos 4y ago
  Why do you think fixes for minor problems should be backported to stable kernels?
  - 4y ago

dmarlow 4y ago

How is testing done in kernels? Is there unit testing, integration, end to end? I'm unfamiliar, but curious.

seanhunter 4y ago
Not in the Linux kernel. In the Linux kernel it has traditionally always been done by the developer sticking lots of printk statements into their code until they are happy with the outputs and then removing most of them. Make of that what you will.
- bonzini 4y ago
  LOL that is so false that is not even worth correcting.
  - marcellus23 4y ago
    Actually could you correct it? I don't know anything about the Linux kernel so I'm curious.
    - seanhunter 4y ago
      I'll correct myself. That was how it was done circa 2000-2005, but my info was way out of date because I haven't followed kernel development much for a while. Here's some stuff about how it's done now. https://www.kernel.org/doc/html/latest/dev-tools/testing-overview.html https://www.kernel.org/doc/html/latest/dev-tools/testing-ove... Basically there is a system called KUnit for writing white box unit tests and there is code coverage to determine coverage as you would expect, then there are systems for static analysis and verifying assertions.
  - deleted 4y ago
    [deleted]
- vfclists 4y ago
  This is so wrong!!

helmsb 4y ago

“ There's a common saying and rule of thumb in programming (possibly originating in the C world) that it's never a compiler bug, it's going to be a bug in your code even if it looks crazy or impossible.” I say, it’s “rarely a compiler bug.” When teaching new developers I remind them that you should assume the bug is in the code you just wrote before jumping to the bug being a compiler, framework or OS issue. It’s just a matter of probabilities.

plumefar 4y ago
In my +- 25 years of experience, I've encountered exactly 1 compiler bug. That cost me a full week of debug. Unfortunately, it happened during my very first year as a developer, and that triggered sub-optimal debugging practices (ie: is that the compiler fault again?) At the end of the day, you want to optimise for debug time. There's the probability that it's a compiler/developer bug (respectively very low/very high) and the time it takes to rule it out. It's of course best NOT to start by investigating the compiler bug.
- bondant 4y ago
  I think it's really dependent of the domain you are working in. Some stuff is better tested than others. In my 5 years careers I found bugs (found and confirmed by compiler authors) in gfortran, xlf and pgfortran. So yes bugs in compilers are rare but sometimes you just keep running into them.
  - dtech 4y ago
    those are quite niche languages though, most people use compilers which are much more widely used.
    - capitalsigma 4y ago
      Clang support for AVX-512 intrinsics is pretty broken
- CameronNemo 4y ago

mrtweetyhack 4y ago

[dead]

nikanj 4y ago

The biggest mindset adjustment I ever had to make was moving from C++ to Node.js. You can usually trust that select() is not broken, but select.js is deprecated, select-kitchensink.js is not compatible with your toolchain and unicyclect.ts leaks memory like a sieve

koala_man 4y ago

> it's never a compiler bug I wish this was true. It would have made my job of writing compilers so much easier.

moloch-hai 4y ago
Out of all the compiler bugs I have failed reports against over the years, only one was fixed. And the fix was applied only in the "trunk" branch, not any of the other ones still "maintained". I learned that Gcc has a policy not to fix performance regressions on any but the development branch. Back in the late '80s we spent fully half of each working day tracking down the trigger and workaround for cfront crashes. Kids today have it easy.
lostmsu 4y ago
I think you misunderstand. It is never the bug in the compiler of your compiler.

tedunangst 4y ago

It's encouraging to see so many new volunteers signing up to help maintain and test the stable branch!

netheril96 4y ago

For curiosity I upgraded the kernel of my VPS running Debian 11 from 5.10 to 6.1. The networking became visibly slower. All TCP receive and send rate droped to at least 1/4 of the original. I had to restore the original kernel and the network became fine again. I don't think it is a bug in the kernel. More likely the userspace or the host system needs to be reconfigured to work well with the bleeding edge kernel. Still I am curious, what exactly may cause this dramatic drop?

yencabulator 4y ago

> it's never a compiler bug Back in the days of ISO-8859-1, you could make gcc crash by having the string literal "ä" in the source code...