16 ms·

It is eternally frustrating how valuable kernel features like grsec, the bfq and bfs schedulers (for disk and cpu respectively), and many more hugely useful but

by zanny 10y ago

It is eternally frustrating how valuable kernel features like grsec, the bfq and bfs schedulers (for disk and cpu respectively), and many more hugely useful but not commercially backed (or at least proposed) projects wallow in side trees of Linux mainline. It is just baffling how the grsec patches are ~13? years old now and yet remain out of tree rather than, say, default-off on kconfig, which even then makes no sense - users of the kernel should be getting maximally secure and conservative software first, and then turning off checks to increase performance, not the other way around - having a wild and exploitable kernel you need to use out of tree patches to get under reasonable control.

CrLf 10y ago
If there's something that defines the Linux kernel development process is that large/wide changes are never accepted in bulk. Changes must be broken up into independent pieces that make sense by themselves and can be well understood by the people doing the merges. For example, if you want to introduce a new filesystem that requires VFS features not yet present, you submit those features first and make a case for their usefulness beyond your own use case. Then you submit your new filesystem adapted to use those features as they where (then already) merged. Failing to go through this process results in failure to be accepted for inclusion in mainline. Sure, this makes some useful stuff linger by the wayside (usually due to ego clashes), but it's the only way to maintain any sanity in a project as large and active as the Linux kernel and that's a good thing in the long run.
- DominikD 10y ago
  AFAIK they never tried submitting in bulk so I'm not sure why this (sensible, I think) process is used as an argument against them. I don't know if they tried to post piece by piece (and when/if they gave up). But there's a pretty well documented anti-grsecurity sentiment on the lists dating waaaay back. So if I were them I wouldn't be necessarily convinced that I should invest my time in slicing to begin with. ;)
- cyberpunk 10y ago
  You don't evolve proper security though a million tiny commits. You design it. This is why containers on Linux will probably never have the security that Solaris or the BSDs have, and this attitude is why. It will never get better until that is rethought, if you're right..
  - CrLf 10y ago
    That's what people used to say about OS kernels, but Linux proved them wrong by evolving the most successful kernel on the planet by almost all measures.
    - KingMob 10y ago
      True, but one measure Linux is not the best at is security. By many accounts, OpenBSD leads the pack there.
  - SwellJoe 10y ago
    I don't think I buy this argument; at least, I don't think grsecurity backs up this argument. grsecurity is a big collection of a bunch of different techniques. It isn't a total rethink of the Linux kernel, it is a bunch of patches spread over a large surface area. Some of the changes are far-reaching, yes, but even so, each of the techniques represented in grsecurity could be broken into independent patches. So, why haven't they been? (I honestly don't know why. It's not something I follow closely.)
    - zanny 10y ago
      > So, why haven't they been? Grsec upstream argues that only accepting part of the whole is worse than having their patches outside mainline in bulk, because it would give developers and users a false sense of security to only provide some of the hardening grsec does. See this[1] LWN article from 2009, which is a kernel developers post with a comment retort by some grsec developers. [1]: https://lwn.net/Articles/313621/ https://lwn.net/Articles/313621/ The frustration is that we are now seven years later and the situation has only gotten worse - plenty of kernel exploits have emerged, no distro is shipping grsec comprehensively, grsec itself had to stop providing stable releases, and it has become more religious than logical as to why many critical features, like the internal bug prevention functionality, never will be mainlined and will be re-implemented with much less thoroughly tested surrogates.
- cm3 10y ago
  VFS is a good example of something that's hairy and easy to introduce regressions affecting everyone. That's why Al Viro dissects stuff in detail. Most regressions are introduced in hardware drivers and those are the largest part of the kernel and the part that sees most churn. It's great that we don't see more regressions, actually, though the Intel DRM drivers have gotten worse in quality over the last year.
- forgotpwtomain 10y ago
  > If there's something that defines the Linux kernel development process is that large/wide changes are never accepted in bulk. Changes must be broken up into independent pieces that make sense by themselves and can be well understood by the people doing the merges. And that's how you end up with clearly superior subsystem implementations (BFQ) that linger outside the kernel as GP mentions. Because a professor doing free work should somehow find a way to hack a cohesive working subsystem into the mess which is the old codebase so as to satisfy the maintainers (who are paid to work on this stuff) need for a gradual transition (which will eventually rip out all of the old code anyways).
  - CrLf 10y ago
    That's just it, isn't it? What makes it clearly superior? (BTW, I'm not arguing it isn't.) Linux supports quite a lot of different workloads on quite a lot of different hardware. Looking for the best possible performance while ensuring reliability is a very big task and a very big responsibility not to be taken lightly by stereotyping kernel maintainers as some villains looking to make victims out or poor, well-intentioned, contributors. Also, arguing that contributors don't have the means to go through a public review process intended to result in better code doesn't really add much credibility to the original code, does it? (Not saying that's the case with BFQ either.) There have been many instances in the past where "clearly superior" patches ended up having pathological behavior for many workloads. Something that would only be discovered after impacting a lot of users if it weren't for the review processes in place. Some of these were merged anyway (but not made default) while some others didn't prove to provide enough improvement (even in the cases they optimize for) to justify the extra bloat.
    - forgotpwtomain 10y ago
      It's been a while since I've looked at the papers or read the discussion surrounding it's inclusion into the mainline (so I apologize if I get some details wrong), but my understanding regarding the first point is two fold: > That's just it, isn't it? What makes it clearly superior? (BTW, I'm not arguing it isn't.) A) There are is a range of benchmarks which support that conclusion across various hardware and workloads. [0][1] B) It is my understanding that the current day CFQ is a complicated code-base with various engineering improvements that don't have formal specs or theoretical guarantees, the improvements BFQ brings have theoretical underpinnings [2][3], at the least a spec other than "read the code to understand how and why". > Looking for the best possible performance while ensuring reliability is a very big task and a very big responsibility not to be taken lightly by stereotyping kernel maintainers as some villains looking to make victims out or poor, well-intentioned, contributors. It's definitely not my intention to villainize kernel developers (for example Tejun Heo appears to be highly supportive in [4]) - and I don't think Paolo is a victim here; but I do think that both casual and professional users are hurt by the end result. > Also, arguing that contributors don't have the means to go through a public review process intended to result in better code doesn't really add much credibility to the original code, does it? (Not saying that's the case with BFQ either.) I mean to the extent that you are talking about ripping out a sub-system for a new one, it's nearly impossible to really talk about having the means. The later is I presume a non-trivial task even for someone with a long track-record of good engineering and maintenance. Paolo is if anything also a very good engineer - but that's not even the case for a lot of academics. I guess from my perspective if an Academic provides you with formalized improvements + benchmarks + working code that's already really, really great, there is very significant reason to have engineers with a long track record of reliability who can review the specs, implement/verify/improve the code to get it up to standards, and to deal the with requirements/politics for making these changes. Again my understanding can be totally off here, I'm not a kernel developer - I am just a user who patches with BFQ (and CK for my personal machine) since given the current process improvements do not seem to be forthcoming. [0] http://algo.ing.unimo.it/people/paolo/disk_sched/results.php http://algo.ing.unimo.it/people/paolo/disk_sched/results.php [1] http://algo.ing.unimo.it/people/paolo/disk_sched/comparison.php http://algo.ing.unimo.it/people/paolo/disk_sched/comparison.... [2] http://algo.ing.unimo.it/people/paolo/disk_sched/description.php http://algo.ing.unimo.it/people/paolo/disk_sched/description... [3] http://algo.ing.unimo.it/people/paolo/disk_sched/bfq-techreport.pdf http://algo.ing.unimo.it/people/paolo/disk_sched/bfq-techrep... [4] https://lkml.org/lkml/2014/5/30/412 https://lkml.org/lkml/2014/5/30/412

by zanny 10y ago

CrLf 10y ago

If there's something that defines the Linux kernel development process is that large/wide changes are never accepted in bulk. Changes must be broken up into independent pieces that make sense by themselves and can be well understood by the people doing the merges. For example, if you want to introduce a new filesystem that requires VFS features not yet present, you submit those features first and make a case for their usefulness beyond your own use case. Then you submit your new filesystem adapted to use those features as they where (then already) merged. Failing to go through this process results in failure to be accepted for inclusion in mainline. Sure, this makes some useful stuff linger by the wayside (usually due to ego clashes), but it's the only way to maintain any sanity in a project as large and active as the Linux kernel and that's a good thing in the long run.

DominikD 10y ago
AFAIK they never tried submitting in bulk so I'm not sure why this (sensible, I think) process is used as an argument against them. I don't know if they tried to post piece by piece (and when/if they gave up). But there's a pretty well documented anti-grsecurity sentiment on the lists dating waaaay back. So if I were them I wouldn't be necessarily convinced that I should invest my time in slicing to begin with. ;)
cyberpunk 10y ago
You don't evolve proper security though a million tiny commits. You design it. This is why containers on Linux will probably never have the security that Solaris or the BSDs have, and this attitude is why. It will never get better until that is rethought, if you're right..
- CrLf 10y ago
  That's what people used to say about OS kernels, but Linux proved them wrong by evolving the most successful kernel on the planet by almost all measures.
  - KingMob 10y ago
    True, but one measure Linux is not the best at is security. By many accounts, OpenBSD leads the pack there.
- SwellJoe 10y ago
  I don't think I buy this argument; at least, I don't think grsecurity backs up this argument. grsecurity is a big collection of a bunch of different techniques. It isn't a total rethink of the Linux kernel, it is a bunch of patches spread over a large surface area. Some of the changes are far-reaching, yes, but even so, each of the techniques represented in grsecurity could be broken into independent patches. So, why haven't they been? (I honestly don't know why. It's not something I follow closely.)
  - zanny 10y ago
    > So, why haven't they been? Grsec upstream argues that only accepting part of the whole is worse than having their patches outside mainline in bulk, because it would give developers and users a false sense of security to only provide some of the hardening grsec does. See this[1] LWN article from 2009, which is a kernel developers post with a comment retort by some grsec developers. [1]: https://lwn.net/Articles/313621/ https://lwn.net/Articles/313621/ The frustration is that we are now seven years later and the situation has only gotten worse - plenty of kernel exploits have emerged, no distro is shipping grsec comprehensively, grsec itself had to stop providing stable releases, and it has become more religious than logical as to why many critical features, like the internal bug prevention functionality, never will be mainlined and will be re-implemented with much less thoroughly tested surrogates.
cm3 10y ago
VFS is a good example of something that's hairy and easy to introduce regressions affecting everyone. That's why Al Viro dissects stuff in detail. Most regressions are introduced in hardware drivers and those are the largest part of the kernel and the part that sees most churn. It's great that we don't see more regressions, actually, though the Intel DRM drivers have gotten worse in quality over the last year.
forgotpwtomain 10y ago
> If there's something that defines the Linux kernel development process is that large/wide changes are never accepted in bulk. Changes must be broken up into independent pieces that make sense by themselves and can be well understood by the people doing the merges. And that's how you end up with clearly superior subsystem implementations (BFQ) that linger outside the kernel as GP mentions. Because a professor doing free work should somehow find a way to hack a cohesive working subsystem into the mess which is the old codebase so as to satisfy the maintainers (who are paid to work on this stuff) need for a gradual transition (which will eventually rip out all of the old code anyways).
- CrLf 10y ago
  That's just it, isn't it? What makes it clearly superior? (BTW, I'm not arguing it isn't.) Linux supports quite a lot of different workloads on quite a lot of different hardware. Looking for the best possible performance while ensuring reliability is a very big task and a very big responsibility not to be taken lightly by stereotyping kernel maintainers as some villains looking to make victims out or poor, well-intentioned, contributors. Also, arguing that contributors don't have the means to go through a public review process intended to result in better code doesn't really add much credibility to the original code, does it? (Not saying that's the case with BFQ either.) There have been many instances in the past where "clearly superior" patches ended up having pathological behavior for many workloads. Something that would only be discovered after impacting a lot of users if it weren't for the review processes in place. Some of these were merged anyway (but not made default) while some others didn't prove to provide enough improvement (even in the cases they optimize for) to justify the extra bloat.
  - forgotpwtomain 10y ago
    It's been a while since I've looked at the papers or read the discussion surrounding it's inclusion into the mainline (so I apologize if I get some details wrong), but my understanding regarding the first point is two fold: > That's just it, isn't it? What makes it clearly superior? (BTW, I'm not arguing it isn't.) A) There are is a range of benchmarks which support that conclusion across various hardware and workloads. [0][1] B) It is my understanding that the current day CFQ is a complicated code-base with various engineering improvements that don't have formal specs or theoretical guarantees, the improvements BFQ brings have theoretical underpinnings [2][3], at the least a spec other than "read the code to understand how and why". > Looking for the best possible performance while ensuring reliability is a very big task and a very big responsibility not to be taken lightly by stereotyping kernel maintainers as some villains looking to make victims out or poor, well-intentioned, contributors. It's definitely not my intention to villainize kernel developers (for example Tejun Heo appears to be highly supportive in [4]) - and I don't think Paolo is a victim here; but I do think that both casual and professional users are hurt by the end result. > Also, arguing that contributors don't have the means to go through a public review process intended to result in better code doesn't really add much credibility to the original code, does it? (Not saying that's the case with BFQ either.) I mean to the extent that you are talking about ripping out a sub-system for a new one, it's nearly impossible to really talk about having the means. The later is I presume a non-trivial task even for someone with a long track-record of good engineering and maintenance. Paolo is if anything also a very good engineer - but that's not even the case for a lot of academics. I guess from my perspective if an Academic provides you with formalized improvements + benchmarks + working code that's already really, really great, there is very significant reason to have engineers with a long track record of reliability who can review the specs, implement/verify/improve the code to get it up to standards, and to deal the with requirements/politics for making these changes. Again my understanding can be totally off here, I'm not a kernel developer - I am just a user who patches with BFQ (and CK for my personal machine) since given the current process improvements do not seem to be forthcoming. [0] http://algo.ing.unimo.it/people/paolo/disk_sched/results.php http://algo.ing.unimo.it/people/paolo/disk_sched/results.php [1] http://algo.ing.unimo.it/people/paolo/disk_sched/comparison.php http://algo.ing.unimo.it/people/paolo/disk_sched/comparison.... [2] http://algo.ing.unimo.it/people/paolo/disk_sched/description.php http://algo.ing.unimo.it/people/paolo/disk_sched/description... [3] http://algo.ing.unimo.it/people/paolo/disk_sched/bfq-techreport.pdf http://algo.ing.unimo.it/people/paolo/disk_sched/bfq-techrep... [4] https://lkml.org/lkml/2014/5/30/412 https://lkml.org/lkml/2014/5/30/412