6 ms·

Kernel quality control, or the lack thereof

174 points by based2 8y ago

exmadscientist 8y ago
This is nothing new. I followed kernel development pretty closely about a decade ago and I was honestly very disappointed in what I saw. It's all about features, features, features, and there's almost no testing beyond "well, it works on my machine". It's kind of ironic that Linux has become the C++ of kernels, given Linus's feelings about C++: a giant pile of features, many of which nobody really understands, that can have some really weird corner case interactions. I switched to FreeBSD as a result of my time reading LKML. The BSDs seem, at least to me, to be more designed rather than thrown together; but maybe I just haven't spent enough time watching their sausage get made. Hopefully now that a senior kernel developer (Chinner) is saying some of these things publicly, things can get better. But a culture change of the magnitude needed will not come quickly.
- m45t3r 8y ago
  I don't think it is surprise that Linux is about adding more features in an organic way, instead of being well thought like *BSDs. However saying that no testing happens in Linux kernel is dishonest, to say at least: there is automated tests maintained by big corporations like LTP [1] or autotest [2], thousands of people run different versions of unstable/mainline kernels with different configurations, security researches does multiple tests like running fuzzers and reporting issues, multiple opensource projects run tests in current versions of Linux kernel (that in the end also serves as a test of kernel itself), etc etc. Linux is basically the kind of project that is big enough and impactful enough that naturally gets testing for free from the community. [1]: https://github.com/linux-test-project/ltp https://github.com/linux-test-project/ltp [2]: https://github.com/autotest/autotest https://github.com/autotest/autotest
  - maltalex 8y ago
    > Linux is basically the kind of project that is big enough and impactful enough that naturally gets testing for free from the community. That makes intuitive sense, but is it really true? Is Linux being throughly tested by the community?
    - qlk1123 8y ago
      How do you define "throughly" anyway? Just like a comment in the original post, >>But anything less than 100% coverage guarantees that some part of the code is not tested... > And anything less than 100% race coverage similarly guarantees a hole in your testing. As does anything less than 100% configuration-combination coverage. As does anything less than 100% input coverage. As does anything less than 100% hardware-configuration testing. As does ...
    - notacoward 8y ago
      This has always been my problem with the "many eyes make all bugs shallow" theory. It's a random walk. The most common features and configurations will get tested over and over and over well beyond the point of usefulness. Less common features might not get tested at all. In fact it's worse than a random walk, because usage is even more concentrated in a few areas. Yes, the fuzzers etc. do find some bugs, but relative to the size and importance of the project itself they're smaller than they would be on most other kinds of projects. There's just no substitute for rigorous unit and/or functional tests, constructed with an eye toward what can happen instead of just what commonly does happen in the most vanilla usage. Unfortunately, UNIX kernel development - not just Linux - has never been strong on that kind of testing. Part of the reason is the inherent difficulty of doing those sorts of tests on such a foundational component. Part of it is ignorance, which I mean as simply "not knowing" and not as an insult. Part of it is macho kernel-hacker culture. Whatever the reasons, it needs to change.
      - m45t3r 8y ago
        > The most common features and configurations will get tested over and over and over well beyond the point of usefulness. Less common features might not get tested at all. Not that I wouldn't want a featured Unix-like kernel with a comprehensive[1] test suite (and I don't know any major OS that have this, not Windows, macOS or Linux), however I think this is fine. Common setups works fine, uncommon configurations may have some problems, with possible workarounds. Most people will not run mainline kernels anyway, so a workaround is acceptable. [1]: What I mean by comprehensive is basically having tests for every possible configuration, that is basically impossible anyway. Probably the closest thing we can get is formally proven OS, however I don't think we will even have a general purpose OS formally proven.
  - Shorel 8y ago
    Ok, I will check these links to try to generate log files. Because the Linux kernel for sure is going crazy in my new laptop. The lack of AMD support is mindblowing, to say the least.
    - kev009 8y ago
      AMD laid off a lot of their Linux devs in 2012, and it still hasn't recovered [1] [1] https://linux.slashdot.org/story/12/11/07/1634237/amd-closes-osrc-lays-off-several-linux-kernel-developers https://linux.slashdot.org/story/12/11/07/1634237/amd-closes...
    - int_19h 8y ago
      What do you mean by "lack of AMD support"? CPU? GPU?
      - Shorel 8y ago
        I wish I knew. Using a Dell Inspiron with AMD Ryzen CPU and AMD Vega 8 GPU. Sometimes it just does not boot. Sometimes it works fine for some hours, and totally locks up when I click a button in Firefox.
- nly 8y ago
  > "well, it works on my machine". I don't know why you'd be surprised by this. The top kernel on kernel.org's front page is literally just Linus's personal tree. "Upstream" is context-sensitive but "mainline" has always been Linus's personal tree. In reality, what matters to most users is what goes in to Debian stable or RHEL, and those kernels are far from just "works for me" tested.
  - slondr 8y ago
    Linux has always been treated like it's Linus's personal project. That's why more than a few of us believed that Linus taking a 'break' a few months ago was the beginning of the end. It turns out that not much changed, but the culture is still there.
- cageface 8y ago
  Could this be part of the motivation for Google to develop Fuchsia?
  - ori_b 8y ago
    If it's anything like most of Google's code, it's also going to be features features features, with weird corner cases and interactions.
- newnewpdro 8y ago
  If FreeBSD had received a fraction of the resources Linux has received over the last decades, it would have its own share of scars left by the growing pains. For years FreeBSD development was very stagnant while Linux exploded in popularity and support. The comparison is silly.
- nwmcsween 8y ago
  Eh FreeBSD and *BSD suffers from the shiny disease as well.
- ksec 8y ago
  It is Intelligent Design and Order (BSD*) versus Natural Selection and Chaos (GNU/Linux). Not saying which one is better, I guess everyone has preference. Although I really wish FreeBSD is more widely deployed and used.
michaelt 8y ago
Does anyone know what kernel developers do if someone sends them a patch adding support for hardware they don't have - a new wireless dongle, say? Does the maintainer merge the code after just reading over the code and checking it compiles, never having executed it? My assumption has always been there's little testing in such a situation - but I can't square that with the fact the kernel seems to work so well :)
- wtallis 8y ago
  In practice, such code dumps usually get rejected out of hand because most unknown first-time contributors are unable to produce a fully functional driver without violating the coding practices of that subsystem and re-inventing the wheel in some fashion. The maintainers always prefer companies to do their driver development in a more open fashion so that there can be two-way feedback throughout the process to ensure the end result is something worthy of merging. Developers who play by the rules and ensure that their driver both fits into existing infrastructure and doesn't break anything else are usually developers who can be trusted to deliver code that works on the new hardware that's not yet public.
- cesarb 8y ago
  Having sent the kernel maintainers a patch adding support for hardware they don't have (in my case, it was a network card): they merge the code after just reading over the code and checking it compiles. Keep in mind, however, that they usually have decades of experience in their particular area of the kernel, so they often can tell at a glance when you're doing something wrong or unusual.
- megous 8y ago
  Yes. Maintainers themsevels are also fully capable of breaking support for HW they don't have. It's a collaborative project. If support for your HW gets broken, you can bug someone to fix it or fix it yourself. What I do to keep my HW working is fetch -rc1, test it, and send fixes.
notacoward 8y ago
"We trusted people" is a classic denial of responsibility. The people shoving in the new-feature changes helped create the problem. It's great that at least one of them seems to have had a change of heart, but starting by blaming (unspecified) others suggests that the change might not last long. Expect reversion to form in a few months.
shereadsthenews 8y ago
Syzkaller finds bugs in linux kernel all day every day. The fact that some ioctls don't work right should surprise nobody.
hannob 8y ago
Not surprised to read this about XFS. I did some fuzzing on filesystem tools a few years ago. xfsprogs was... "interesting". Interesting in that it was seemingly impossible to find anyone to report bugs to. They had a bug tracker, but it didn't work, submitting bugs resulted in an error. I think there was also a mail address that bounced. I think in the end my bug reports reached noone who could care about them.
- int0x80 8y ago
  There is xfs.org and the XFS mailing list. You can just send an email there. Maybe you were sending HTML emails or something that made the message bounce?
lazka 8y ago
I've filed bugs for some of the mentioned copy_file_range() issues more than two years ago: - https://bugzilla.kernel.org/show_bug.cgi?id=135461 https://bugzilla.kernel.org/show_bug.cgi?id=135461 - https://bugzilla.kernel.org/show_bug.cgi?id=135451 https://bugzilla.kernel.org/show_bug.cgi?id=135451 No response...
- craftyguy 8y ago
  I'm sure they would have welcomed patches.
  - lazka 8y ago
    What makes you so sure?
    - craftyguy 8y ago
      Users complaining that things are broken < Users complaining that things are broken and then submitting a patch (or at least some attempt to unbreak things) It's exceedingly rare that a maintainer of anything would scoff at all patches from users to help fix things.
      - lazka 8y ago
        Got it, so you're just guessing.
        craftyguy 8y ago
        No, experience.
  - jodrellblank 8y ago
    What does that have to do with anything?
    - craftyguy 8y ago
Mister_Snuggles 8y ago
There was one bug for a while where IPSec would not handle TCP packets[0]. This was a big one since sending TCP packets over an IPSec tunnel is a somewhat common scenario. I had to keep an old kernel for quite a while before that one was fixed. It seems that a lot of the quality control happens with the distributions, not with the upstream software itself. I doubt that SLES would have seen this bug, but because I was running Tumbleweed I have to expect breakage like that. [0] http://lkml.iu.edu/hypermail/linux/kernel/1704.3/02043.html http://lkml.iu.edu/hypermail/linux/kernel/1704.3/02043.html
buserror 8y ago
From my experience dealing with upstreaming, it has become more of some sort of priesthood that is required to go thru the proper mantras and speak the proper language. And most of the priest no longer have any real idea of what's the real world is doing with the kernel. I've spent a few months upstreaming a subsystem earlier last year, something that had been tested in the field, at customers, and I had to literally gut it to fit the priesthood's way of doing things. Ultimately the result that went into the kernel was technically /inferior/ to the original source, and any rant about /that/ was pointedly ignored. I'm not even going to mention the device tree binding, which has become as bad as the high days of XML where the format took a life of it's own and requires it's own maintainers. It's completely bonkers. I think since a lot of maintainers became 'professionals' they no longer use linux. They just juggle patches all day and talk between themselves and their clique. And as long as it fits the big tech company that pay them, it doesn't matter if it's actually /useful/ to anyone else.

174 points by based2 8y ago

This is nothing new. I followed kernel development pretty closely about a decade ago and I was honestly very disappointed in what I saw. It's all about features, features, features, and there's almost no testing beyond "well, it works on my machine". It's kind of ironic that Linux has become the C++ of kernels, given Linus's feelings about C++: a giant pile of features, many of which nobody really understands, that can have some really weird corner case interactions. I switched to FreeBSD as a result of my time reading LKML. The BSDs seem, at least to me, to be more designed rather than thrown together; but maybe I just haven't spent enough time watching their sausage get made. Hopefully now that a senior kernel developer (Chinner) is saying some of these things publicly, things can get better. But a culture change of the magnitude needed will not come quickly.

m45t3r 8y ago
I don't think it is surprise that Linux is about adding more features in an organic way, instead of being well thought like *BSDs. However saying that no testing happens in Linux kernel is dishonest, to say at least: there is automated tests maintained by big corporations like LTP [1] or autotest [2], thousands of people run different versions of unstable/mainline kernels with different configurations, security researches does multiple tests like running fuzzers and reporting issues, multiple opensource projects run tests in current versions of Linux kernel (that in the end also serves as a test of kernel itself), etc etc. Linux is basically the kind of project that is big enough and impactful enough that naturally gets testing for free from the community. [1]: https://github.com/linux-test-project/ltp https://github.com/linux-test-project/ltp [2]: https://github.com/autotest/autotest https://github.com/autotest/autotest
- maltalex 8y ago
  > Linux is basically the kind of project that is big enough and impactful enough that naturally gets testing for free from the community. That makes intuitive sense, but is it really true? Is Linux being throughly tested by the community?
  - qlk1123 8y ago
    How do you define "throughly" anyway? Just like a comment in the original post, >>But anything less than 100% coverage guarantees that some part of the code is not tested... > And anything less than 100% race coverage similarly guarantees a hole in your testing. As does anything less than 100% configuration-combination coverage. As does anything less than 100% input coverage. As does anything less than 100% hardware-configuration testing. As does ...
  - notacoward 8y ago
    This has always been my problem with the "many eyes make all bugs shallow" theory. It's a random walk. The most common features and configurations will get tested over and over and over well beyond the point of usefulness. Less common features might not get tested at all. In fact it's worse than a random walk, because usage is even more concentrated in a few areas. Yes, the fuzzers etc. do find some bugs, but relative to the size and importance of the project itself they're smaller than they would be on most other kinds of projects. There's just no substitute for rigorous unit and/or functional tests, constructed with an eye toward what can happen instead of just what commonly does happen in the most vanilla usage. Unfortunately, UNIX kernel development - not just Linux - has never been strong on that kind of testing. Part of the reason is the inherent difficulty of doing those sorts of tests on such a foundational component. Part of it is ignorance, which I mean as simply "not knowing" and not as an insult. Part of it is macho kernel-hacker culture. Whatever the reasons, it needs to change.
    - m45t3r 8y ago
      > The most common features and configurations will get tested over and over and over well beyond the point of usefulness. Less common features might not get tested at all. Not that I wouldn't want a featured Unix-like kernel with a comprehensive[1] test suite (and I don't know any major OS that have this, not Windows, macOS or Linux), however I think this is fine. Common setups works fine, uncommon configurations may have some problems, with possible workarounds. Most people will not run mainline kernels anyway, so a workaround is acceptable. [1]: What I mean by comprehensive is basically having tests for every possible configuration, that is basically impossible anyway. Probably the closest thing we can get is formally proven OS, however I don't think we will even have a general purpose OS formally proven.
- Shorel 8y ago
  Ok, I will check these links to try to generate log files. Because the Linux kernel for sure is going crazy in my new laptop. The lack of AMD support is mindblowing, to say the least.
  - kev009 8y ago
    AMD laid off a lot of their Linux devs in 2012, and it still hasn't recovered [1] [1] https://linux.slashdot.org/story/12/11/07/1634237/amd-closes-osrc-lays-off-several-linux-kernel-developers https://linux.slashdot.org/story/12/11/07/1634237/amd-closes...
  - int_19h 8y ago
    What do you mean by "lack of AMD support"? CPU? GPU?
    - Shorel 8y ago
      I wish I knew. Using a Dell Inspiron with AMD Ryzen CPU and AMD Vega 8 GPU. Sometimes it just does not boot. Sometimes it works fine for some hours, and totally locks up when I click a button in Firefox.
nly 8y ago
> "well, it works on my machine". I don't know why you'd be surprised by this. The top kernel on kernel.org's front page is literally just Linus's personal tree. "Upstream" is context-sensitive but "mainline" has always been Linus's personal tree. In reality, what matters to most users is what goes in to Debian stable or RHEL, and those kernels are far from just "works for me" tested.
- slondr 8y ago
  Linux has always been treated like it's Linus's personal project. That's why more than a few of us believed that Linus taking a 'break' a few months ago was the beginning of the end. It turns out that not much changed, but the culture is still there.
cageface 8y ago
Could this be part of the motivation for Google to develop Fuchsia?
- ori_b 8y ago
  If it's anything like most of Google's code, it's also going to be features features features, with weird corner cases and interactions.
newnewpdro 8y ago
If FreeBSD had received a fraction of the resources Linux has received over the last decades, it would have its own share of scars left by the growing pains. For years FreeBSD development was very stagnant while Linux exploded in popularity and support. The comparison is silly.
nwmcsween 8y ago
Eh FreeBSD and *BSD suffers from the shiny disease as well.
ksec 8y ago
It is Intelligent Design and Order (BSD*) versus Natural Selection and Chaos (GNU/Linux). Not saying which one is better, I guess everyone has preference. Although I really wish FreeBSD is more widely deployed and used.

michaelt 8y ago

Does anyone know what kernel developers do if someone sends them a patch adding support for hardware they don't have - a new wireless dongle, say? Does the maintainer merge the code after just reading over the code and checking it compiles, never having executed it? My assumption has always been there's little testing in such a situation - but I can't square that with the fact the kernel seems to work so well :)

wtallis 8y ago
In practice, such code dumps usually get rejected out of hand because most unknown first-time contributors are unable to produce a fully functional driver without violating the coding practices of that subsystem and re-inventing the wheel in some fashion. The maintainers always prefer companies to do their driver development in a more open fashion so that there can be two-way feedback throughout the process to ensure the end result is something worthy of merging. Developers who play by the rules and ensure that their driver both fits into existing infrastructure and doesn't break anything else are usually developers who can be trusted to deliver code that works on the new hardware that's not yet public.
cesarb 8y ago
Having sent the kernel maintainers a patch adding support for hardware they don't have (in my case, it was a network card): they merge the code after just reading over the code and checking it compiles. Keep in mind, however, that they usually have decades of experience in their particular area of the kernel, so they often can tell at a glance when you're doing something wrong or unusual.
megous 8y ago
Yes. Maintainers themsevels are also fully capable of breaking support for HW they don't have. It's a collaborative project. If support for your HW gets broken, you can bug someone to fix it or fix it yourself. What I do to keep my HW working is fetch -rc1, test it, and send fixes.

notacoward 8y ago

"We trusted people" is a classic denial of responsibility. The people shoving in the new-feature changes helped create the problem. It's great that at least one of them seems to have had a change of heart, but starting by blaming (unspecified) others suggests that the change might not last long. Expect reversion to form in a few months.

shereadsthenews 8y ago

Syzkaller finds bugs in linux kernel all day every day. The fact that some ioctls don't work right should surprise nobody.

hannob 8y ago

Not surprised to read this about XFS. I did some fuzzing on filesystem tools a few years ago. xfsprogs was... "interesting". Interesting in that it was seemingly impossible to find anyone to report bugs to. They had a bug tracker, but it didn't work, submitting bugs resulted in an error. I think there was also a mail address that bounced. I think in the end my bug reports reached noone who could care about them.

int0x80 8y ago
There is xfs.org and the XFS mailing list. You can just send an email there. Maybe you were sending HTML emails or something that made the message bounce?

lazka 8y ago

I've filed bugs for some of the mentioned copy_file_range() issues more than two years ago: - https://bugzilla.kernel.org/show_bug.cgi?id=135461 https://bugzilla.kernel.org/show_bug.cgi?id=135461 - https://bugzilla.kernel.org/show_bug.cgi?id=135451 https://bugzilla.kernel.org/show_bug.cgi?id=135451 No response...

craftyguy 8y ago
I'm sure they would have welcomed patches.
- lazka 8y ago
  What makes you so sure?
  - craftyguy 8y ago
    Users complaining that things are broken < Users complaining that things are broken and then submitting a patch (or at least some attempt to unbreak things) It's exceedingly rare that a maintainer of anything would scoff at all patches from users to help fix things.
    - lazka 8y ago
      Got it, so you're just guessing.
      - craftyguy 8y ago
        No, experience.
- jodrellblank 8y ago
  What does that have to do with anything?
  - craftyguy 8y ago

Mister_Snuggles 8y ago

There was one bug for a while where IPSec would not handle TCP packets[0]. This was a big one since sending TCP packets over an IPSec tunnel is a somewhat common scenario. I had to keep an old kernel for quite a while before that one was fixed. It seems that a lot of the quality control happens with the distributions, not with the upstream software itself. I doubt that SLES would have seen this bug, but because I was running Tumbleweed I have to expect breakage like that. [0] http://lkml.iu.edu/hypermail/linux/kernel/1704.3/02043.html http://lkml.iu.edu/hypermail/linux/kernel/1704.3/02043.html

buserror 8y ago

From my experience dealing with upstreaming, it has become more of some sort of priesthood that is required to go thru the proper mantras and speak the proper language. And most of the priest no longer have any real idea of what's the real world is doing with the kernel. I've spent a few months upstreaming a subsystem earlier last year, something that had been tested in the field, at customers, and I had to literally gut it to fit the priesthood's way of doing things. Ultimately the result that went into the kernel was technically /inferior/ to the original source, and any rant about /that/ was pointedly ignored. I'm not even going to mention the device tree binding, which has become as bad as the high days of XML where the format took a life of it's own and requires it's own maintainers. It's completely bonkers. I think since a lot of maintainers became 'professionals' they no longer use linux. They just juggle patches all day and talk between themselves and their clique. And as long as it fits the big tech company that pay them, it doesn't matter if it's actually /useful/ to anyone else.