25 ms·

“Fast Kernel Headers” Tree -v1: Eliminate the Linux Kernel's “Dependency Hell”

549 points by nnx 5y ago

formerly_proven 5y ago
From: Ingo Molnar Subject: [PATCH 0000/2297] Eliminate the Linux kernel's "Dependency Hell" ... 25,288 files changed, 178,024 insertions(+), 74,720 deletions(-)
- londons_explore 5y ago
  You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. Think how many lines are now duplicate (and therefore need to be updated in twice as many places, and bugs introduced when a copy is missed). Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. If the 100k lines were in their own subdirectory and added a major new feature, it would be worth it. But spread across the whole codebase, and introducing more duplication and chances for bugs I think outweighs a 'neater' header file system. The 'dependency hell' is only an issue for people who only want to build part of the kernel anyway. If you're building the whole lot, you might as well just include everything and it'll work great.
  - tedunangst 5y ago
    Maybe the people developing the project have some insights into which build improvements they would benefit from. Having to recompile the entire tree every time you change a seemingly unrelated header gets old fast.
  - derefr 5y ago
    > [This patch series] decouples much of the high level headers from others, uninlining of unnecessary functions, a decoupling of the type and API headers, automated dependency handling of header files, and a variety of other changes That's a lot more than "shaving the build time." As far as I understand it, the goal of this patchset isn't to improve the build time; that's just a nice consequence. The goal was to refactor the header-file hierarchy to make it more maintainable and less "brittle." Sometimes, increasing maintainability requires more code. (Almost always, if the current version is a terse mess of mixed concerns.) Think of it this way: take an IOCCC entry, and de-obfuscate it. You're "increasing the size of the codebase." You might even be "duplicating" some things (e.g. magic constants that were forcefully squashed together because they happened to share a value, which are now separate constants per semantic meaning.) But doing this obviously increases the maintainability of the code.
    - XorNot 5y ago
      I'd say cycle time is also important anyway: we spend a lot of time building the Linux kernel in various forms. The savings across the board in manhours waiting for compilation, or for git bisect, are not insubstantial. I wonder what the memory savings for compilation look like? Because that's also potentially more workers in automated testing farms for the same cost.
      - derefr 5y ago
        I've never tried it myself, but presuming you build the kernel in modular rather than monolithic mode, wouldn't the incremental compiles during git-bisect et al already be pretty quick? You'd only be rebuilding the modules whose source files changed, and those (presumably) wouldn't be pulling in the entire header tree. (Or would they, with that being "the problem"?)
        tedunangst 5y ago
        That is the problem.
  - aaomidi 5y ago
  - bsder 5y ago
    > You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. In Ingo's post, he points out that the main speedup is coming from the fact that the expansion after the C preprocessor step is a LOT smaller. That's a lot of decoupling. As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. Having those data structures in files actually mentioned by the file you're working on would be a huge cognitive load improvement as well as make tool assistance much more plausible. Even if this particular patch doesn't land, it lights the path. What types of changes need to be made are now clear. How many changes are required before you see the improvement is now clear. With those, the changes required can be driven down into the maintainers and rolled out incrementally, if desired.
    - yjftsjthsd-h 5y ago
      > As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. As someone who tried poking around: Oh good; I assumed I was just missing something. Alternatively, oh no; I had assumed I was missing something and there was a more elegant tool out there.
      - alexforencich 5y ago
        https://elixir.bootlin.com/linux/latest/source https://elixir.bootlin.com/linux/latest/source can be far more helpful than grep for this sort of thing
    - wott 5y ago
      > As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. It has always amazed me how finding something which originally seemed a trivial little thing, usually meant going through a chain of #defines and typedefs across many header files. It's the same with GLibc, by the way. It' a bit like when you hike to a summit by following a crest path: you always think the next hump in sight is the right one, your destination, the promise land; and when you reach it, dammit, it wasn't, your goal is actually the next one. Or perhaps the next after the next. Or...
  - Supermancho 5y ago
    > You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. When you're talking a project of half a million lines, sure. The Linux kernel has around 27.8 million lines of code. An increase of .35% > Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. Why add features at all? Code has a purpose. Sometimes bringing code into a static context is a net good. It was going to be generated at runtime anyway. > If you're building the whole lot, you might as well just include everything and it'll work great. That's not strictly true, but it's true for these features, which is a stated reasoning.
    - kangalioo 5y ago
      Good golly, that 0.35% figure really puts the patch set into perspective A library I'm working on is 7000 LOC which seems pretty sizable, but 0.35% of that is 25 LOC. The Linux patch set is actually tiny
    - theon144 5y ago
      >The Linux kernel has around 27.8 million lines of code. An increase of .35% This is horribly misleading; most of these lines of code are drivers, which this patchset doesn't even concern. It's still a massive change that only a handful of developers will ever be able to review in entirety - a fact to which the size of the project is completely irrelevant - if anything, actually, it urges even more caution, given the implied complexity. Which I believe was (at least in part) parent comment's point - given the importance and ubiquity of the Linux kernel, this may be concerning. That said, I am very confident in the structures put in place by the kernel devs, their competence and the necessity for such a change - but trivializing a 100k LoC patchset because the project it's intended to land in is even more colossally complex isn't how I'd choose my approach.
      - Denvercoder9 5y ago
        > This is horribly misleading; most of these lines of code are drivers, which this patchset doesn't even concern. That's not true at all, a big part of those added lines are added includes in drivers. E.g. this commit I picked at random adds 1500 lines, of which just a few procent are in the core kernel: https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git/commit/?id=390dec865b2afb6219914d08b4b5c870d1f052a6 https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.gi...
  - stjohnswarts 5y ago
    You need to read the article. It explains all this.
  - fargle 5y ago
    > You have to have an awfully good reason to add 100k lines of code to a project yes. and this is very, very good reason. As another poster said, you at 0.35% lines to make it compile almost twice as fast? And you're not happy about that? > Think how many lines are now duplicate (and therefore need to be updated in twice ... OK, how many? None! That's how many. Adding proper header dependencies to the .c module doesn't duplicate anything. Unless you think adding #include <stdio.h> in every module somehow creates unmaintainable duplication. > Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. OK. Hrm... I think a lot, lot less is how much. That's the whole point of a major cleanup like this. Proper decoupling. Headers that you use are obvious where they belong not being brought in with some action-at-a-distance accident.
- kangalioo 5y ago
  Where did the additional 100kLOC come from?
  - crishoj 5y ago
    I suspect it’s this: “Automated dependency addition to .h and .c files. This is about 790 commits, which are strictly limited to trivial addition of header dependencies. These patches make indirect dependencies explicit, and thus prepare the tree for aggressive removal of cross-dependencies from headers.”
  - jontro 5y ago
    Did a brief scan of the repo. Looks like a lot of it comes from adding new headers. This commit adds +#include <linux/wait_api.h> to 1627 files for instance. https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git/commit/?id=4a3fb303583079cdf40f2bbb2d2eaedb664b3558 https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.gi...
- charcircuit 5y ago
  >2298 emails When will Linux move to a site like github / gitlab / something similar self hosted that supports people proposing changes without having to send every single person interested in the development of Linux thousands of emails.
  - CRConrad 5y ago
    It's a fucking shame how many people nowadays can't grok the difference between git and GitHub. Your question is fundamentally wrong; totally bass-ackwards. The correct question is: When will all the zillion other projects move off of GitHub / GitLab / all the other usurper sites?
  - ReactiveJelly 5y ago
    I think having everyone see all the emails is part of the point. Even with self-hosted Gitea, someone has to _host_ it. And there isn't a trivial export button, and if the host goes rogue they can make it inaccessible and make everyone lose efficiency while they swap back to the old method. And if you need email to authenticate on a website, why not just use email anyway? But this is an interesting idea, and I'm on the look-out for ideas to replace email. I'll keep "LKML" in the back of my head as a use case.
    - charcircuit 5y ago
      >And there isn't a trivial export button It does support making backups of repositories at least. >and if the host goes rogue they can make it inaccessible and make everyone lose efficiency while they swap back to the old method. Isn't this also a problem if the person managing the email list goes rogue? You have to trust someone to host the infrastructure. >And if you need email to authenticate on a website, why not just use email anyway? Because Email may not be the optimal user interface for handling issues, pull requests, code review, etc.
      - xyzzy_plugh 5y ago
        > Isn't this also a problem if the person managing the email list goes rogue? You have to trust someone to host the infrastructure. It depends on what kind of threat you are worried about. The beautiful thing is even if LKML was abused and ruined tomorrow, nothing is lost or damaged. The emails have already been sent, and moving to a new list is more of a nuisance at this point. If you're worried about authenticating a patch series, committers can sign their commits a number of ways. Lastly, a large chunk of Linux development happens off-list, with subsystem maintainers building branches for Linus or Greg to pull from.
bcwarner 5y ago
What's the process like for approving and implementing a series of patches this huge in quantity? Seems like it would almost require stalling on other pending issues for a significant amount of time while this is worked in.
- taeric 5y ago
  If it does as advertised, probably should not affect too much. Specifically, moving implicit dependencies to explicit should not impact anyone using the dependency. Might trip up code that is adding a new implicit?
  - AnssiH 5y ago
    Making implicit deps explicit is only one part of the patchset. Moving all that header code around between files will conflict with all other code merges that touch those same lines.
    - taeric 5y ago
      Fair. I'm assuming the patches to update includes covers a wider set than changing the headers themselves. Though, really, I was just saying why it may not be as bad as it sounds. Can still be bad, of course.
      - hnaccount_rng 5y ago
        FTA: > - As to maintenance overhead: it was surprisingly low overhead to keep dependencies at a minimum across upstream kernel releases - there were tyically just around ~5 dependency additions that need to be addressed. This makes me hopeful that an optimal 'fast' state of header dependencies can be maintained going forward - once the initial set of fixes are in of course.
- colonwqbang 5y ago
  It can probably be merged piece by piece, so it won't be a problem. The reason he wrote the full patch series out like this, is because he wanted to measure the speed gains. Without significant speed gains it would be difficult to convince the kernel developers to make a big uncomfortable change like this.
yrral 5y ago
Kudos to getting this out. I can't imagine working on a series of patches for 2 years for build cleanups/speedups on a separate branch, having to periodically rebase against master and throwing away my work 3 times in order to figure out how to do it correctly. At any company I have worked for, I would have been fired for being unproductive/afraid of making changes.
- dv_dt 5y ago
  Why not work on it and release in smaller batches though?
  - Bayart 5y ago
    He says himself he didn't see major improvements before 1500 commits.
  - detaro 5y ago
    As the RFC email clearly explains, they were experimenting, restarted multiple times and to go quite deep into it until they got to a point where this was worth announcing as something that might be worthwhile to merge.
    - vlovich123 5y ago
      Yeah. I’ve done this in the past. The first few attempts are about learning where the pain points are and to get ideas about what a good way to structure the patch set might look like. Edit: obviously not at this scale. 2300 patches at once in a fork is a lot.
    - dv_dt 5y ago
      Well at this point i think it’s good work but now that the change is identified as good and fairly concrete - it’s still likely a benefit to break it up into smaller more reviewable chunks to take in over time. Maybe by major subsystem so their expert maintainers can take closer looks at it.
      - 5y ago
stefan_ 5y ago
This kids is exactly why nobody maintains out-of-tree kernel drivers.
- unmole 5y ago
  Meh. There are loads of reasons why maintaining out of tree drivers is a bad idea. But this patch set is unlikly to cause any problems.
- phkahler 5y ago
  >> This kids is exactly why nobody maintains out-of-tree kernel drivers. After this it should be easier to maintain out of tree stuff since it eliminates "dependency hell".
  - Arch-TK 5y ago
    The dependency hell is not the kind of dependency hell where it's difficult to work on the code because including something might break something else. The dependency hell described in this email is to do with headers including headers including headers, eventually including a large quantity of text (which now has to be parsed) most of which is unnecessary. The only problem this "dependency hell" causes for out-of-tree modules is that builds take a bit longer. The fixes to this dependency hell are not going to make out-of-tree modules work better, they're likely going to break them in a lot of ways due to the many implicitly included headers they happen to have.
dopidopHN 5y ago
what the process to get that type of change request merged? I suspect it's complex to find reviewers for such a broad change set.
- bombcar 5y ago
  Each maintainer will check their subsystem/driver, etc, standard tests will be run, and someone like Linus will go over the methodology is likely what will happen. Ingo has likely done a bunch of tests himself, of course. As someone pointed out, it’s only like 0.13% of the kernel.
- oneplane 5y ago
  I imagine all the individual owners, or Linus on a treadmill would be the way to go.
- hamburglar 5y ago
  I think step 1 is “be Ingo or someone comparably legendary” :)
- prirun 5y ago
  Disclaimer: I know nothing about the Linux kernel! It sounds to me like this work is re-organizing and moving around existing code in the header files to reduce the amount of code the C compiler has to wade through during a Linux build. So one way to verify the changes is to compare the binary files (or the resulting executable) built the old way vs the new way. In theory, they should be identical. (Or I could be misunderstanding the whole thing and you should stop reading now!) I did something like this for a Prime minicomputer emulator project: https://github.com/prirun/p50em https://github.com/prirun/p50em It was initially coded for a PowerPC chip because that is big-endian like the minicomputer being emulated, and to be honest, I wasn't sure I could even do the emulation, let alone do it and be mentally swapping bytes all the time. After about 10 years of it working well, I spent a week or two reworking it to add byte-swapping for Intel CPUs. The method I used was to first add macros to all memory and register references. This was done on the PowerPC build, and after each set of incremental changes, I'd verify that the executable created with macros was identical to the original executable without macros. If it wasn't, I goofed somewhere, so backed up and added the changes in smaller sections until I found the problem. I didn't have to do any testing of the resulting executable to verify thousands of code changes. Practically every line of code in the 10K LOC project was touched because of the new macros, but each macro should have been a no-op on the PowerPC build with no byte swapping. Next I built it on an Intel CPU where the byte-swap macros were enabled and it basically worked, maybe with a tweak here or there. As a further test, I ran the PowerPC emulator through a bunch of existing tests with tracing enabled and the clock rate fixed. This records the machine state after each instruction, and with a fixed clock, yields a reproduceable instruction trace. Then ran the same test with the Intel version and compared the instruction traces. If there were any differences, it was because I forgot to add a swap macro somewhere. After a few months of running it 24x7 myself w/o problems, a customer did find a bug in the serial device I/O subsystem where I had forgotten to add a macro. I hadn't done any testing of this subsystem (terminal I/O on real serial RS232 devices like printers.) If something similar is being done with these Linux changes, verifying them may not be as hard as it seems initially.
cerved 5y ago
The hero we need but don't deserve
bigdict 5y ago
Can't imagine the amount of whack-a-mole and backtracking involved in refactoring this many headers in a large project. I've done this for a small project and it's a migraine inducing type of work. Yet so valuable. Kudos!
- Gibbon1 5y ago
  I have a firmware project written in C that I maintain. Probably 200k lines of code. I feel like I'm constantly fretting how to keep the headers and dependencies from becoming a hot mess. At the same time not exploding the number of headers either.
gravypod 5y ago
Hopefully the Linux kernel will eventually get automated presubmits for IWYU and other things.
RustyRussell 5y ago
Ingo Molnar was always one of my favourite scary kernel devs. This kind of massive rewrite was always a key strength of OG devs, who were prepared to tackle maintenance tasks beyond the limits of mortal sanity. The result of such ongoing struggles is why the Linux kernel still works after decades of active development, despite radical changing requirements.
- yjftsjthsd-h 5y ago
  A question that I've been mulling over: How do we create more people like this? Better documentation? A... "brief guide to the entire kernel" blog series? Aggressively pushing GSoC project mentoring?
  - mtrycz2 5y ago
    Motivation.
  - stathibus 5y ago
    The same way you build talent in any other organization - give people money, tell them this is their job, and put them in the same "room" as people who also have this job but have been doing it longer. Or are you asking how we create more people who will do this in their free time?
    - voakbasda 5y ago
      These people already exist. They do not need to be manufactured. I think many get turned away from the Linux community for non-technical reasons. For some, the community takes a bigger personal toll than the technical challenges. I personally gave up over that aspect and stopped contributing. I did not feel welcome or needed.
      - eyegor 5y ago
        I have tried to contribute to the Linux kernel development in the past, but I found it to be a strong inner circle of seasoned devs. The fact that they were still using mailing lists instead of github/gitlab/etc to keep track of development was painful enough, but they were also hostile to asking for documentation on inner modules. I'm not sure how it will progress when they retire or die off.
theHIDninja 5y ago
Forgive my ignorance as I'm not a kernel dev. It seems like a change this massive would have the possibility of introducing a bunch of security holes, would it not?
- trebligdivad 5y ago
  All changes can have screwups; but a lot of these are moving things around and fairly mechanical infrastructure changes; so it shouldn't be that dangerous.
- teruakohatu 5y ago
  It seems to be rejigging header files so largely not adding new functionality or modifying code, but giving the compiler a lot less work to do.
- SEJeff 5y ago
  It is not a code change per-se as much as a change to how the files are built. Think of it like changing the order around of how you cook a fine meal and putting a few tricks all senior chefs know into the recipe explicitly. No ingredients were added or removed, do poison doesn’t get accidentally added.
dstroot 5y ago
"Don't forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away." — Ingo Molnar http://www.ussg.iu.edu/hypermail/linux/kernel/9906.0/0746.html http://www.ussg.iu.edu/hypermail/linux/kernel/9906.0/0746.ht... to the Linux Kernel mailing list in 1999.
- inter_netuser 5y ago
  link broken?
  - sp332 5y ago
    It's cited from https://quotepark.com/quotes/1742017/history/ https://quotepark.com/quotes/1742017/history/ but probably rotted some time in the last 23 years. Edit: here's a live mirror. http://lkml.iu.edu/hypermail/linux/kernel/9906.0/0746.html http://lkml.iu.edu/hypermail/linux/kernel/9906.0/0746.html
- fit2rule 5y ago
  Hey now, the full quote is a little cheekier: > platitudes hardly provide compelling evidence for ignoring 20 years > of networking/OS research and architecting unnecessary copies into the. > core of an OS that wants to be taken seriously... dont forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away. -- mingo<endquote> .. don't you go ignoring the fact he was talking about zero-copy in the kernel!
- numbsafari 5y ago
  This is how I feel about the Go programming language. So often you see people ranting about the Go team either being ignorant or scornful of PL research. I always wonder if those same people are Agile Manifesto zealots at work.
  - dodobirdlord 5y ago
    Well, those critiques are not unfounded. The designers of Go are certainly not ignorant of the last few decades of PL research, but they did make a deliberate decision to ignore it and create (they admit!) an inferior language for skilled practitioners, in the interest of being maximally accessible to newer programmers who are only familiar with imperative languages with C-like syntax. > "The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt. – Rob Pike" > "It must be familiar, roughly C-like. Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical. – Rob Pike" https://news.ycombinator.com/item?id=16143918 https://news.ycombinator.com/item?id=16143918 If you're interested in a critique that goes more deeply into Go's actual design flaws, as opposed to its deliberate design tradeoffs, I enjoy https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-....
gitgud 5y ago
Here's the [1] patch cover letter sent to Linus Torvalds (similar to a Pull Request I suppose) [1] https://lore.kernel.org/lkml/YdIfz+LMewetSaEB@gmail.com/T/#u https://lore.kernel.org/lkml/YdIfz+LMewetSaEB@gmail.com/T/#u
- srcreigh 5y ago
  The submitted link should be changed to this, imo.
  - fouc 5y ago
    It's directly linked in the article anyways
- aero-glide2 5y ago
  >similiar to a pull request. Someone should make a tool which parses mailing lists and presents it in a Github-like frontend
  - yhorawu8 5y ago
    Aerc is built around git+email flow: https://aerc-mail.org/ https://aerc-mail.org/
  - db48x 5y ago
    Why? Github and similar front–ends top out at a few dozen commits or comments on a merge request before they start whining that the change is too big.
    - aero-glide2 5y ago
      Could be read-only. Mailing lists aren't very accessible.
      - MayeulC 5y ago
        It depends on your definition of accessible. This is the tool kernel devs use, and that patch is intended for them. It's mostly a matter of getting used to it. Github's user interface was scary at first for me, much more than a plain-text mail. So was GitLab. Some people I know have complained to me that I send plain-text mails and that nobody wants to read a wall of text. I think that's probably a user issue, and it can in turn help filter low-effort comments. If talking about a11y, plaintext mail ought to be one of the most accessible formats: high-contrast, can be processed by multiple tools including screen readers. ASCII art trough screenreaders is probably an exception. ----- Now, about your specific question: there's patchwork[1], though I don't think that instance covers the general LKML. [1] https://patchwork.kernel.org/ https://patchwork.kernel.org/
deleted 5y ago
[deleted]
throwaway81523 5y ago
GCC needs something like that too. Building on a big multicore box turns out to not get that much parallelism, because of the dependency ordering.
aks2161989 5y ago
Sime6
kevinastone 5y ago
Duplicate of https://news.ycombinator.com/item?id=29775298 https://news.ycombinator.com/item?id=29775298
- cerved 5y ago
  not exactly, this is the mailing list. IMHO, much more interesting that the original article
- dang 5y ago
  Thanks! It's true that that was posted earlier, but it is a blog post reporting on this email, and the site guidelines call for original sources: "Please submit the original source. If a post reports on something found on another site, submit the latter." So I think we'll merge that thread into this one.
jonahbenton 5y ago
Very curious to see the reaction to this. Such a massive restructuring of the trees, header dependencies, and some of the machinery- these facts with which so many people are so familiar. There are few "who moved my cheese" events like moving code around. For the other humans whose brains work on the kernel- the abrupt shock of moving to a vaguely familiar but in the end quite different house. Yes, the old one was suboptimal in various ways but this new one... Few people's brains work like Ingo's. And he has had 2 years to work and iterate through the mental model. Field day here for the Science, Technology, and Society crowd, watching how old dogs deal with new tricks.
sydthrowaway 5y ago
Yeah, this is not going to get merged.
- RustyRussell 5y ago
  I disagree, but it's really up to Linus. I'd expect it'll go in in pieces, after a bit of bikeshedding and a few iterations of archs fixing up compilation (struct task changes mainly).
- kzrdude 5y ago
  It's definitely going to be merged. Of course not in the RFC version, but Molnar didn't work on this for a year in three revisions for it to be thrown away.
  - wmu 5y ago
    It was already said by the author, that approx ~70% of patches can be applied independently in various part of the source tree. The rest has to be synchronised.
  - tecleandor 5y ago
    Just the sheer amount of money and time that could be saved on computing resources in CI processes (i.e. Android or Linux builds) would mean that some companies would be keen to give support to his huge change.
remexre 5y ago
Does the kernel currently use precompiled headers? Anyone know how much they ought to help with this sorta scenario? (Like, presumably their gains would be lessened by this patchset, but could they have gotten halfway there on performance on their own?)
- edflsafoiewq 5y ago
  > As to other approaches attempted: I also tried enabling the kernel to use pre-compiled headers (with GCC), which didn't bring a performance improvement beyond a 1-2% for repeat builds. Plus during development most of the precompiled headers are cache-cold in any case and need to be generated by the compiler again and again.
- gnulinux 5y ago
  Precompiling C headers is almost never useful. It gives essentially no benefit (since compiling most C headers is fast) with more binaries to worry about. I only bother to do it for especially slow libraries in C++.
da-x 5y ago
> For the 'reference' subsystem of the scheduler, I also improved build speed by consolidating .c files into roughly equal size build units. Instead of 20+ separate .o's, there's now just 4 .o's being built. Obviously this approach does not scale to the over 30,000 .c files in the kernel, but I wanted to demonstrate it because optimizing at that level brings the next level of build performance, and it might be feasible for a handful of other core kernel subsystems. This is important, realizing that compilation units rarely collide on private namespace usage. Called 'compilation unit grouping', or 'CU grouping', I implemented this once at a customer's and reduced build time of a large C++ project by 2x.
- rep_movsd 5y ago
  I remember doing this - one C file with about 100 lines from #include "file001.c" to #include "file299.c" These were files from a library that used a large number of tiny files, we prefered to embed this in our project rather than link in. Massive build time reduction. There is some technical name for this other than "compilation unit grouping" - I forget what
  - tambre 5y ago
    Unity builds, but also unified and jumbo builds.
- kzrdude 5y ago
  Merging CU's should preferably be done by the build tool and not have to be written into the .c source files themselves.
  - da-x 5y ago
    Exactly that - the implementation of it was done transparently under scons's SConstruct logic.
- jcelerier 5y ago
  If you use CMake it's just a matter of setting -DCMAKE_UNITY_BUILD=1
fMwhy 5y ago
> # Performance counter stats for 'make -j96 vmlinux' -j96 ...you flexing bruh ???
- RandomBK 5y ago
  Shouldn't be too surprising that one of the core devs at the heart of modern computing infra has a Threadripper...
  - bombcar 5y ago
    More importantly it shows the gains when already massively parallel (and perhaps on a single or dual core you don’t get as much gains as it doesn’t matter that the CPU has to go in order when you only have a few).
- stjohnswarts 5y ago
  j96 really isn't that big of a deal these days if you have access to server hardware.
mfrw 5y ago
I tried to take it for a spin using `make LLVM=1 -j$(nproc)`, but, it seems it only compiles for GCC. Has anyone compiled it using `LLVM=1` ?
- karlding 5y ago
  It looks like there's some discussion on the patch series which suggest various diffs that need to be applied in order to compile with LLVM=1 [0]. [0] https://lwn.net/ml/linux-kernel/YdM4Z5a+SWV53yol@archlinux-ax161/ https://lwn.net/ml/linux-kernel/YdM4Z5a+SWV53yol@archlinux-a...
  - mfrw 5y ago
    Thank you :) I was starting to wonder if I am missing anything trivial.
mrlonglong 5y ago
I like it, this will certainly help speed up kernel builds on slow computing platforms such as the raspberry pi which can take hours if you can't cross compile.
vrodic 5y ago
Sounds like headers in general are not necessarily a good idea. They provide flexibility but at a cost. I enjoyed Borland Pascal compilation speeds and clean dependencies before switching to Linux. I miss it, together with Delphi.
- rep_movsd 5y ago
  C++ has been trying modules, and Gabriel Dos Reis from Microsoft has gotten a pretty decent standard and implementation done. It's likely it will be added to the C++23 standard. The biggest issue seems to be "What cross compiler format do we use for the compiled modules" (the equivalent of DCUs). The other problem is that macros have to still work, and code that uses them will not benefit as much from modules. The gains in compilation speed (including linking) and dependency information tracking is phenomenal though. It would certainly bring C++ in line with languages like Java, Rust etc. The CPPCon 2021 talk about this is enlightening. If C++ gets modules, there is no reason why C can't use the same model - since the compilers will have that stuff anyway.
  - dextercd 5y ago
    Modules are part of C++20
    - vitus 5y ago
      And it's telling that we're in 2022 and still the only compiler with full C++ module support according to cppreference [0] is MSVC. [0] https://en.cppreference.com/w/cpp/compiler_support https://en.cppreference.com/w/cpp/compiler_support https://clang.llvm.org/cxx_status.html https://clang.llvm.org/cxx_status.html https://gcc.gnu.org/projects/cxx-status.html https://gcc.gnu.org/projects/cxx-status.html
- phendrenad2 5y ago
  The c compilation model is tuned for ancient hardware. On modern hardware, incremental compilation isn't all that useful, so the entire import/header model makes less sense.
rwmj 5y ago
He writes: As to maintenance overhead: it was surprisingly low overhead to keep dependencies at a minimum across upstream kernel releases - there were typically just around ~5 dependency additions that need to be addressed. This makes me hopeful that an optimal 'fast' state of header dependencies can be maintained going forward - once the initial set of fixes are in of course. It seems like this could regress quickly just by a developer adding a "bad" dependency to some core header which happened to #include a lot of code by accident. Don't we need tools to ensure this doesn't regress? Like something which warns "this git commit increases the amount of compiled code by 20%".
- phkahler 5y ago
  Not sure, but isn't a large part of it keeping #include out of header files? That's easy to spot, but can be hard to untangle once its done.
egberts 5y ago
I’ve embarked down this same path with GCC 2.7.4 from a … while … back. Ingo is on the right track. I’ve always thought that this stuff should have been dealt with by the GCC (in form of pre-cached header files). So sad that it is a thorny issue to this day (for large scale projects); so glad that this issue is now alive so it can be dealt with once and for all. I envision some type of *.ho filetype to hold these quasi-compiled headers for the next build iteration.
- folmar 5y ago
  Do you mean .pch? I think it works quite well generally, but for the kernel it might be too fragile as it is not so widely used.
- pas 5y ago
  "... dealt with once and for all." Is at least one GCC maintainer looking at this? Otherwise it'll likely not be dealt with "for all", right? On the kernel side every new dependency has to be added with keeping in mind this problem as long as there's no automation to help with this it'll be a problem.
egberts 5y ago
And Ingo probably had the greatest heads up over the rest of us by having his new kernel perform the fastest iterative kernel rebuilds, probably using all permutation of kernel option settings for rebuildings.
commandlinefan 5y ago
> I got to over 500 commits in this tree and had to throw away my second attempt But wait, why didn't you just break the task down into sub-tasks that could be estimated in four-hour increments in the first place? Every single "scrum master" I've ever labored under has insisted that every software development task can be broken down this way.
- Arch-TK 5y ago
  Clearly Ingo is just an inexperienced developer who struggles to do this simple task. Maybe he needs more training?
- klysm 5y ago
  I hadn’t thought about it, but this is a great counter example to the (bad) assumptions of scrum.
astrostl 5y ago
I find it fascinating to consider the potential impact on energy consumption, human time, hardware and hosting costs, etc. of just one patch to just one thing.
- mtnygard 5y ago
  I was wondering the same thing. Something like the Linux kernel gets built and rebuilt constantly by build systems and users all over the world. Even marginal improvements in build cost become big when there's such a large multiplier.
rektide 5y ago
Linux 6!! Do it!
geophertz 5y ago
From what I understand, this was mostly done manually. Would it have been possible to represent dependencies as a graph to obtain an optimal result?
phendrenad2 5y ago
Don't driver files just include whatever headers they want? Does this mean that out-of-mainline drivers will need to be rewritten?
- pas 5y ago
  > Does this mean that out-of-mainline drivers will need to be rewritten? Aren't they always anyway? Plus if the driver can be built separately then it can include whatever it wants.
habanany 5y ago
999900000999 5y ago
How do you even QA this.

549 points by nnx 5y ago

From: Ingo Molnar Subject: [PATCH 0000/2297] Eliminate the Linux kernel's "Dependency Hell" ... 25,288 files changed, 178,024 insertions(+), 74,720 deletions(-)

londons_explore 5y ago
You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. Think how many lines are now duplicate (and therefore need to be updated in twice as many places, and bugs introduced when a copy is missed). Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. If the 100k lines were in their own subdirectory and added a major new feature, it would be worth it. But spread across the whole codebase, and introducing more duplication and chances for bugs I think outweighs a 'neater' header file system. The 'dependency hell' is only an issue for people who only want to build part of the kernel anyway. If you're building the whole lot, you might as well just include everything and it'll work great.
- tedunangst 5y ago
  Maybe the people developing the project have some insights into which build improvements they would benefit from. Having to recompile the entire tree every time you change a seemingly unrelated header gets old fast.
- derefr 5y ago
  > [This patch series] decouples much of the high level headers from others, uninlining of unnecessary functions, a decoupling of the type and API headers, automated dependency handling of header files, and a variety of other changes That's a lot more than "shaving the build time." As far as I understand it, the goal of this patchset isn't to improve the build time; that's just a nice consequence. The goal was to refactor the header-file hierarchy to make it more maintainable and less "brittle." Sometimes, increasing maintainability requires more code. (Almost always, if the current version is a terse mess of mixed concerns.) Think of it this way: take an IOCCC entry, and de-obfuscate it. You're "increasing the size of the codebase." You might even be "duplicating" some things (e.g. magic constants that were forcefully squashed together because they happened to share a value, which are now separate constants per semantic meaning.) But doing this obviously increases the maintainability of the code.
  - XorNot 5y ago
    I'd say cycle time is also important anyway: we spend a lot of time building the Linux kernel in various forms. The savings across the board in manhours waiting for compilation, or for git bisect, are not insubstantial. I wonder what the memory savings for compilation look like? Because that's also potentially more workers in automated testing farms for the same cost.
    - derefr 5y ago
      I've never tried it myself, but presuming you build the kernel in modular rather than monolithic mode, wouldn't the incremental compiles during git-bisect et al already be pretty quick? You'd only be rebuilding the modules whose source files changed, and those (presumably) wouldn't be pulling in the entire header tree. (Or would they, with that being "the problem"?)
      - tedunangst 5y ago
        That is the problem.
- aaomidi 5y ago
- bsder 5y ago
  > You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. In Ingo's post, he points out that the main speedup is coming from the fact that the expansion after the C preprocessor step is a LOT smaller. That's a lot of decoupling. As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. Having those data structures in files actually mentioned by the file you're working on would be a huge cognitive load improvement as well as make tool assistance much more plausible. Even if this particular patch doesn't land, it lights the path. What types of changes need to be made are now clear. How many changes are required before you see the improvement is now clear. With those, the changes required can be driven down into the maintainers and rolled out incrementally, if desired.
  - yjftsjthsd-h 5y ago
    > As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. As someone who tried poking around: Oh good; I assumed I was just missing something. Alternatively, oh no; I had assumed I was missing something and there was a more elegant tool out there.
    - alexforencich 5y ago
      https://elixir.bootlin.com/linux/latest/source https://elixir.bootlin.com/linux/latest/source can be far more helpful than grep for this sort of thing
  - wott 5y ago
    > As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. It has always amazed me how finding something which originally seemed a trivial little thing, usually meant going through a chain of #defines and typedefs across many header files. It's the same with GLibc, by the way. It' a bit like when you hike to a summit by following a crest path: you always think the next hump in sight is the right one, your destination, the promise land; and when you reach it, dammit, it wasn't, your goal is actually the next one. Or perhaps the next after the next. Or...
- Supermancho 5y ago
  > You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. When you're talking a project of half a million lines, sure. The Linux kernel has around 27.8 million lines of code. An increase of .35% > Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. Why add features at all? Code has a purpose. Sometimes bringing code into a static context is a net good. It was going to be generated at runtime anyway. > If you're building the whole lot, you might as well just include everything and it'll work great. That's not strictly true, but it's true for these features, which is a stated reasoning.
  - kangalioo 5y ago
    Good golly, that 0.35% figure really puts the patch set into perspective A library I'm working on is 7000 LOC which seems pretty sizable, but 0.35% of that is 25 LOC. The Linux patch set is actually tiny
  - theon144 5y ago
    >The Linux kernel has around 27.8 million lines of code. An increase of .35% This is horribly misleading; most of these lines of code are drivers, which this patchset doesn't even concern. It's still a massive change that only a handful of developers will ever be able to review in entirety - a fact to which the size of the project is completely irrelevant - if anything, actually, it urges even more caution, given the implied complexity. Which I believe was (at least in part) parent comment's point - given the importance and ubiquity of the Linux kernel, this may be concerning. That said, I am very confident in the structures put in place by the kernel devs, their competence and the necessity for such a change - but trivializing a 100k LoC patchset because the project it's intended to land in is even more colossally complex isn't how I'd choose my approach.
    - Denvercoder9 5y ago
      > This is horribly misleading; most of these lines of code are drivers, which this patchset doesn't even concern. That's not true at all, a big part of those added lines are added includes in drivers. E.g. this commit I picked at random adds 1500 lines, of which just a few procent are in the core kernel: https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git/commit/?id=390dec865b2afb6219914d08b4b5c870d1f052a6 https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.gi...
- stjohnswarts 5y ago
  You need to read the article. It explains all this.
- fargle 5y ago
  > You have to have an awfully good reason to add 100k lines of code to a project yes. and this is very, very good reason. As another poster said, you at 0.35% lines to make it compile almost twice as fast? And you're not happy about that? > Think how many lines are now duplicate (and therefore need to be updated in twice ... OK, how many? None! That's how many. Adding proper header dependencies to the .c module doesn't duplicate anything. Unless you think adding #include <stdio.h> in every module somehow creates unmaintainable duplication. > Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. OK. Hrm... I think a lot, lot less is how much. That's the whole point of a major cleanup like this. Proper decoupling. Headers that you use are obvious where they belong not being brought in with some action-at-a-distance accident.
kangalioo 5y ago
Where did the additional 100kLOC come from?
- crishoj 5y ago
  I suspect it’s this: “Automated dependency addition to .h and .c files. This is about 790 commits, which are strictly limited to trivial addition of header dependencies. These patches make indirect dependencies explicit, and thus prepare the tree for aggressive removal of cross-dependencies from headers.”
- jontro 5y ago
  Did a brief scan of the repo. Looks like a lot of it comes from adding new headers. This commit adds +#include <linux/wait_api.h> to 1627 files for instance. https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git/commit/?id=4a3fb303583079cdf40f2bbb2d2eaedb664b3558 https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.gi...
charcircuit 5y ago
>2298 emails When will Linux move to a site like github / gitlab / something similar self hosted that supports people proposing changes without having to send every single person interested in the development of Linux thousands of emails.
- CRConrad 5y ago
  It's a fucking shame how many people nowadays can't grok the difference between git and GitHub. Your question is fundamentally wrong; totally bass-ackwards. The correct question is: When will all the zillion other projects move off of GitHub / GitLab / all the other usurper sites?
- ReactiveJelly 5y ago
  I think having everyone see all the emails is part of the point. Even with self-hosted Gitea, someone has to _host_ it. And there isn't a trivial export button, and if the host goes rogue they can make it inaccessible and make everyone lose efficiency while they swap back to the old method. And if you need email to authenticate on a website, why not just use email anyway? But this is an interesting idea, and I'm on the look-out for ideas to replace email. I'll keep "LKML" in the back of my head as a use case.
  - charcircuit 5y ago
    >And there isn't a trivial export button It does support making backups of repositories at least. >and if the host goes rogue they can make it inaccessible and make everyone lose efficiency while they swap back to the old method. Isn't this also a problem if the person managing the email list goes rogue? You have to trust someone to host the infrastructure. >And if you need email to authenticate on a website, why not just use email anyway? Because Email may not be the optimal user interface for handling issues, pull requests, code review, etc.
    - xyzzy_plugh 5y ago
      > Isn't this also a problem if the person managing the email list goes rogue? You have to trust someone to host the infrastructure. It depends on what kind of threat you are worried about. The beautiful thing is even if LKML was abused and ruined tomorrow, nothing is lost or damaged. The emails have already been sent, and moving to a new list is more of a nuisance at this point. If you're worried about authenticating a patch series, committers can sign their commits a number of ways. Lastly, a large chunk of Linux development happens off-list, with subsystem maintainers building branches for Linus or Greg to pull from.

bcwarner 5y ago

What's the process like for approving and implementing a series of patches this huge in quantity? Seems like it would almost require stalling on other pending issues for a significant amount of time while this is worked in.

taeric 5y ago
If it does as advertised, probably should not affect too much. Specifically, moving implicit dependencies to explicit should not impact anyone using the dependency. Might trip up code that is adding a new implicit?
- AnssiH 5y ago
  Making implicit deps explicit is only one part of the patchset. Moving all that header code around between files will conflict with all other code merges that touch those same lines.
  - taeric 5y ago
    Fair. I'm assuming the patches to update includes covers a wider set than changing the headers themselves. Though, really, I was just saying why it may not be as bad as it sounds. Can still be bad, of course.
    - hnaccount_rng 5y ago
      FTA: > - As to maintenance overhead: it was surprisingly low overhead to keep dependencies at a minimum across upstream kernel releases - there were tyically just around ~5 dependency additions that need to be addressed. This makes me hopeful that an optimal 'fast' state of header dependencies can be maintained going forward - once the initial set of fixes are in of course.
colonwqbang 5y ago
It can probably be merged piece by piece, so it won't be a problem. The reason he wrote the full patch series out like this, is because he wanted to measure the speed gains. Without significant speed gains it would be difficult to convince the kernel developers to make a big uncomfortable change like this.

yrral 5y ago

Kudos to getting this out. I can't imagine working on a series of patches for 2 years for build cleanups/speedups on a separate branch, having to periodically rebase against master and throwing away my work 3 times in order to figure out how to do it correctly. At any company I have worked for, I would have been fired for being unproductive/afraid of making changes.

dv_dt 5y ago
Why not work on it and release in smaller batches though?
- Bayart 5y ago
  He says himself he didn't see major improvements before 1500 commits.
- detaro 5y ago
  As the RFC email clearly explains, they were experimenting, restarted multiple times and to go quite deep into it until they got to a point where this was worth announcing as something that might be worthwhile to merge.
  - vlovich123 5y ago
    Yeah. I’ve done this in the past. The first few attempts are about learning where the pain points are and to get ideas about what a good way to structure the patch set might look like. Edit: obviously not at this scale. 2300 patches at once in a fork is a lot.
  - dv_dt 5y ago
    Well at this point i think it’s good work but now that the change is identified as good and fairly concrete - it’s still likely a benefit to break it up into smaller more reviewable chunks to take in over time. Maybe by major subsystem so their expert maintainers can take closer looks at it.
    - 5y ago

stefan_ 5y ago

This kids is exactly why nobody maintains out-of-tree kernel drivers.

unmole 5y ago
Meh. There are loads of reasons why maintaining out of tree drivers is a bad idea. But this patch set is unlikly to cause any problems.
phkahler 5y ago
>> This kids is exactly why nobody maintains out-of-tree kernel drivers. After this it should be easier to maintain out of tree stuff since it eliminates "dependency hell".
- Arch-TK 5y ago
  The dependency hell is not the kind of dependency hell where it's difficult to work on the code because including something might break something else. The dependency hell described in this email is to do with headers including headers including headers, eventually including a large quantity of text (which now has to be parsed) most of which is unnecessary. The only problem this "dependency hell" causes for out-of-tree modules is that builds take a bit longer. The fixes to this dependency hell are not going to make out-of-tree modules work better, they're likely going to break them in a lot of ways due to the many implicitly included headers they happen to have.

dopidopHN 5y ago

what the process to get that type of change request merged? I suspect it's complex to find reviewers for such a broad change set.

bombcar 5y ago
Each maintainer will check their subsystem/driver, etc, standard tests will be run, and someone like Linus will go over the methodology is likely what will happen. Ingo has likely done a bunch of tests himself, of course. As someone pointed out, it’s only like 0.13% of the kernel.
oneplane 5y ago
I imagine all the individual owners, or Linus on a treadmill would be the way to go.
hamburglar 5y ago
I think step 1 is “be Ingo or someone comparably legendary” :)
prirun 5y ago
Disclaimer: I know nothing about the Linux kernel! It sounds to me like this work is re-organizing and moving around existing code in the header files to reduce the amount of code the C compiler has to wade through during a Linux build. So one way to verify the changes is to compare the binary files (or the resulting executable) built the old way vs the new way. In theory, they should be identical. (Or I could be misunderstanding the whole thing and you should stop reading now!) I did something like this for a Prime minicomputer emulator project: https://github.com/prirun/p50em https://github.com/prirun/p50em It was initially coded for a PowerPC chip because that is big-endian like the minicomputer being emulated, and to be honest, I wasn't sure I could even do the emulation, let alone do it and be mentally swapping bytes all the time. After about 10 years of it working well, I spent a week or two reworking it to add byte-swapping for Intel CPUs. The method I used was to first add macros to all memory and register references. This was done on the PowerPC build, and after each set of incremental changes, I'd verify that the executable created with macros was identical to the original executable without macros. If it wasn't, I goofed somewhere, so backed up and added the changes in smaller sections until I found the problem. I didn't have to do any testing of the resulting executable to verify thousands of code changes. Practically every line of code in the 10K LOC project was touched because of the new macros, but each macro should have been a no-op on the PowerPC build with no byte swapping. Next I built it on an Intel CPU where the byte-swap macros were enabled and it basically worked, maybe with a tweak here or there. As a further test, I ran the PowerPC emulator through a bunch of existing tests with tracing enabled and the clock rate fixed. This records the machine state after each instruction, and with a fixed clock, yields a reproduceable instruction trace. Then ran the same test with the Intel version and compared the instruction traces. If there were any differences, it was because I forgot to add a swap macro somewhere. After a few months of running it 24x7 myself w/o problems, a customer did find a bug in the serial device I/O subsystem where I had forgotten to add a macro. I hadn't done any testing of this subsystem (terminal I/O on real serial RS232 devices like printers.) If something similar is being done with these Linux changes, verifying them may not be as hard as it seems initially.

cerved 5y ago

The hero we need but don't deserve

bigdict 5y ago

Can't imagine the amount of whack-a-mole and backtracking involved in refactoring this many headers in a large project. I've done this for a small project and it's a migraine inducing type of work. Yet so valuable. Kudos!

Gibbon1 5y ago
I have a firmware project written in C that I maintain. Probably 200k lines of code. I feel like I'm constantly fretting how to keep the headers and dependencies from becoming a hot mess. At the same time not exploding the number of headers either.

gravypod 5y ago

Hopefully the Linux kernel will eventually get automated presubmits for IWYU and other things.

RustyRussell 5y ago

Ingo Molnar was always one of my favourite scary kernel devs. This kind of massive rewrite was always a key strength of OG devs, who were prepared to tackle maintenance tasks beyond the limits of mortal sanity. The result of such ongoing struggles is why the Linux kernel still works after decades of active development, despite radical changing requirements.

yjftsjthsd-h 5y ago
A question that I've been mulling over: How do we create more people like this? Better documentation? A... "brief guide to the entire kernel" blog series? Aggressively pushing GSoC project mentoring?
- mtrycz2 5y ago
  Motivation.
- stathibus 5y ago
  The same way you build talent in any other organization - give people money, tell them this is their job, and put them in the same "room" as people who also have this job but have been doing it longer. Or are you asking how we create more people who will do this in their free time?
  - voakbasda 5y ago
    These people already exist. They do not need to be manufactured. I think many get turned away from the Linux community for non-technical reasons. For some, the community takes a bigger personal toll than the technical challenges. I personally gave up over that aspect and stopped contributing. I did not feel welcome or needed.
    - eyegor 5y ago
      I have tried to contribute to the Linux kernel development in the past, but I found it to be a strong inner circle of seasoned devs. The fact that they were still using mailing lists instead of github/gitlab/etc to keep track of development was painful enough, but they were also hostile to asking for documentation on inner modules. I'm not sure how it will progress when they retire or die off.

theHIDninja 5y ago

Forgive my ignorance as I'm not a kernel dev. It seems like a change this massive would have the possibility of introducing a bunch of security holes, would it not?

trebligdivad 5y ago
All changes can have screwups; but a lot of these are moving things around and fairly mechanical infrastructure changes; so it shouldn't be that dangerous.
teruakohatu 5y ago
It seems to be rejigging header files so largely not adding new functionality or modifying code, but giving the compiler a lot less work to do.
SEJeff 5y ago
It is not a code change per-se as much as a change to how the files are built. Think of it like changing the order around of how you cook a fine meal and putting a few tricks all senior chefs know into the recipe explicitly. No ingredients were added or removed, do poison doesn’t get accidentally added.

dstroot 5y ago

"Don't forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away." — Ingo Molnar http://www.ussg.iu.edu/hypermail/linux/kernel/9906.0/0746.html http://www.ussg.iu.edu/hypermail/linux/kernel/9906.0/0746.ht... to the Linux Kernel mailing list in 1999.

inter_netuser 5y ago
link broken?
- sp332 5y ago
  It's cited from https://quotepark.com/quotes/1742017/history/ https://quotepark.com/quotes/1742017/history/ but probably rotted some time in the last 23 years. Edit: here's a live mirror. http://lkml.iu.edu/hypermail/linux/kernel/9906.0/0746.html http://lkml.iu.edu/hypermail/linux/kernel/9906.0/0746.html
fit2rule 5y ago
Hey now, the full quote is a little cheekier: > platitudes hardly provide compelling evidence for ignoring 20 years > of networking/OS research and architecting unnecessary copies into the. > core of an OS that wants to be taken seriously... dont forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away. -- mingo<endquote> .. don't you go ignoring the fact he was talking about zero-copy in the kernel!
numbsafari 5y ago
This is how I feel about the Go programming language. So often you see people ranting about the Go team either being ignorant or scornful of PL research. I always wonder if those same people are Agile Manifesto zealots at work.
- dodobirdlord 5y ago
  Well, those critiques are not unfounded. The designers of Go are certainly not ignorant of the last few decades of PL research, but they did make a deliberate decision to ignore it and create (they admit!) an inferior language for skilled practitioners, in the interest of being maximally accessible to newer programmers who are only familiar with imperative languages with C-like syntax. > "The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt. – Rob Pike" > "It must be familiar, roughly C-like. Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical. – Rob Pike" https://news.ycombinator.com/item?id=16143918 https://news.ycombinator.com/item?id=16143918 If you're interested in a critique that goes more deeply into Go's actual design flaws, as opposed to its deliberate design tradeoffs, I enjoy https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-....

gitgud 5y ago

Here's the [1] patch cover letter sent to Linus Torvalds (similar to a Pull Request I suppose) [1] https://lore.kernel.org/lkml/YdIfz+LMewetSaEB@gmail.com/T/#u https://lore.kernel.org/lkml/YdIfz+LMewetSaEB@gmail.com/T/#u

srcreigh 5y ago
The submitted link should be changed to this, imo.
- fouc 5y ago
  It's directly linked in the article anyways
aero-glide2 5y ago
>similiar to a pull request. Someone should make a tool which parses mailing lists and presents it in a Github-like frontend
- yhorawu8 5y ago
  Aerc is built around git+email flow: https://aerc-mail.org/ https://aerc-mail.org/
- db48x 5y ago
  Why? Github and similar front–ends top out at a few dozen commits or comments on a merge request before they start whining that the change is too big.
  - aero-glide2 5y ago
    Could be read-only. Mailing lists aren't very accessible.
    - MayeulC 5y ago
      It depends on your definition of accessible. This is the tool kernel devs use, and that patch is intended for them. It's mostly a matter of getting used to it. Github's user interface was scary at first for me, much more than a plain-text mail. So was GitLab. Some people I know have complained to me that I send plain-text mails and that nobody wants to read a wall of text. I think that's probably a user issue, and it can in turn help filter low-effort comments. If talking about a11y, plaintext mail ought to be one of the most accessible formats: high-contrast, can be processed by multiple tools including screen readers. ASCII art trough screenreaders is probably an exception. ----- Now, about your specific question: there's patchwork[1], though I don't think that instance covers the general LKML. [1] https://patchwork.kernel.org/ https://patchwork.kernel.org/

deleted 5y ago

[deleted]

throwaway81523 5y ago

GCC needs something like that too. Building on a big multicore box turns out to not get that much parallelism, because of the dependency ordering.

aks2161989 5y ago

Sime6

kevinastone 5y ago

Duplicate of https://news.ycombinator.com/item?id=29775298 https://news.ycombinator.com/item?id=29775298

cerved 5y ago
not exactly, this is the mailing list. IMHO, much more interesting that the original article
dang 5y ago
Thanks! It's true that that was posted earlier, but it is a blog post reporting on this email, and the site guidelines call for original sources: "Please submit the original source. If a post reports on something found on another site, submit the latter." So I think we'll merge that thread into this one.

jonahbenton 5y ago

Very curious to see the reaction to this. Such a massive restructuring of the trees, header dependencies, and some of the machinery- these facts with which so many people are so familiar. There are few "who moved my cheese" events like moving code around. For the other humans whose brains work on the kernel- the abrupt shock of moving to a vaguely familiar but in the end quite different house. Yes, the old one was suboptimal in various ways but this new one... Few people's brains work like Ingo's. And he has had 2 years to work and iterate through the mental model. Field day here for the Science, Technology, and Society crowd, watching how old dogs deal with new tricks.

sydthrowaway 5y ago

Yeah, this is not going to get merged.

RustyRussell 5y ago
I disagree, but it's really up to Linus. I'd expect it'll go in in pieces, after a bit of bikeshedding and a few iterations of archs fixing up compilation (struct task changes mainly).
kzrdude 5y ago
It's definitely going to be merged. Of course not in the RFC version, but Molnar didn't work on this for a year in three revisions for it to be thrown away.
- wmu 5y ago
  It was already said by the author, that approx ~70% of patches can be applied independently in various part of the source tree. The rest has to be synchronised.
- tecleandor 5y ago
  Just the sheer amount of money and time that could be saved on computing resources in CI processes (i.e. Android or Linux builds) would mean that some companies would be keen to give support to his huge change.

remexre 5y ago

Does the kernel currently use precompiled headers? Anyone know how much they ought to help with this sorta scenario? (Like, presumably their gains would be lessened by this patchset, but could they have gotten halfway there on performance on their own?)

edflsafoiewq 5y ago
> As to other approaches attempted: I also tried enabling the kernel to use pre-compiled headers (with GCC), which didn't bring a performance improvement beyond a 1-2% for repeat builds. Plus during development most of the precompiled headers are cache-cold in any case and need to be generated by the compiler again and again.
gnulinux 5y ago
Precompiling C headers is almost never useful. It gives essentially no benefit (since compiling most C headers is fast) with more binaries to worry about. I only bother to do it for especially slow libraries in C++.

da-x 5y ago

> For the 'reference' subsystem of the scheduler, I also improved build speed by consolidating .c files into roughly equal size build units. Instead of 20+ separate .o's, there's now just 4 .o's being built. Obviously this approach does not scale to the over 30,000 .c files in the kernel, but I wanted to demonstrate it because optimizing at that level brings the next level of build performance, and it might be feasible for a handful of other core kernel subsystems. This is important, realizing that compilation units rarely collide on private namespace usage. Called 'compilation unit grouping', or 'CU grouping', I implemented this once at a customer's and reduced build time of a large C++ project by 2x.

rep_movsd 5y ago
I remember doing this - one C file with about 100 lines from #include "file001.c" to #include "file299.c" These were files from a library that used a large number of tiny files, we prefered to embed this in our project rather than link in. Massive build time reduction. There is some technical name for this other than "compilation unit grouping" - I forget what
- tambre 5y ago
  Unity builds, but also unified and jumbo builds.
kzrdude 5y ago
Merging CU's should preferably be done by the build tool and not have to be written into the .c source files themselves.
- da-x 5y ago
  Exactly that - the implementation of it was done transparently under scons's SConstruct logic.
jcelerier 5y ago
If you use CMake it's just a matter of setting -DCMAKE_UNITY_BUILD=1

fMwhy 5y ago

> # Performance counter stats for 'make -j96 vmlinux' -j96 ...you flexing bruh ???

RandomBK 5y ago
Shouldn't be too surprising that one of the core devs at the heart of modern computing infra has a Threadripper...
- bombcar 5y ago
  More importantly it shows the gains when already massively parallel (and perhaps on a single or dual core you don’t get as much gains as it doesn’t matter that the CPU has to go in order when you only have a few).
stjohnswarts 5y ago
j96 really isn't that big of a deal these days if you have access to server hardware.

mfrw 5y ago

I tried to take it for a spin using `make LLVM=1 -j$(nproc)`, but, it seems it only compiles for GCC. Has anyone compiled it using `LLVM=1` ?

karlding 5y ago
It looks like there's some discussion on the patch series which suggest various diffs that need to be applied in order to compile with LLVM=1 [0]. [0] https://lwn.net/ml/linux-kernel/YdM4Z5a+SWV53yol@archlinux-ax161/ https://lwn.net/ml/linux-kernel/YdM4Z5a+SWV53yol@archlinux-a...
- mfrw 5y ago
  Thank you :) I was starting to wonder if I am missing anything trivial.

mrlonglong 5y ago

I like it, this will certainly help speed up kernel builds on slow computing platforms such as the raspberry pi which can take hours if you can't cross compile.

vrodic 5y ago

Sounds like headers in general are not necessarily a good idea. They provide flexibility but at a cost. I enjoyed Borland Pascal compilation speeds and clean dependencies before switching to Linux. I miss it, together with Delphi.

rep_movsd 5y ago
C++ has been trying modules, and Gabriel Dos Reis from Microsoft has gotten a pretty decent standard and implementation done. It's likely it will be added to the C++23 standard. The biggest issue seems to be "What cross compiler format do we use for the compiled modules" (the equivalent of DCUs). The other problem is that macros have to still work, and code that uses them will not benefit as much from modules. The gains in compilation speed (including linking) and dependency information tracking is phenomenal though. It would certainly bring C++ in line with languages like Java, Rust etc. The CPPCon 2021 talk about this is enlightening. If C++ gets modules, there is no reason why C can't use the same model - since the compilers will have that stuff anyway.
- dextercd 5y ago
  Modules are part of C++20
  - vitus 5y ago
    And it's telling that we're in 2022 and still the only compiler with full C++ module support according to cppreference [0] is MSVC. [0] https://en.cppreference.com/w/cpp/compiler_support https://en.cppreference.com/w/cpp/compiler_support https://clang.llvm.org/cxx_status.html https://clang.llvm.org/cxx_status.html https://gcc.gnu.org/projects/cxx-status.html https://gcc.gnu.org/projects/cxx-status.html
phendrenad2 5y ago
The c compilation model is tuned for ancient hardware. On modern hardware, incremental compilation isn't all that useful, so the entire import/header model makes less sense.

rwmj 5y ago

He writes: As to maintenance overhead: it was surprisingly low overhead to keep dependencies at a minimum across upstream kernel releases - there were typically just around ~5 dependency additions that need to be addressed. This makes me hopeful that an optimal 'fast' state of header dependencies can be maintained going forward - once the initial set of fixes are in of course. It seems like this could regress quickly just by a developer adding a "bad" dependency to some core header which happened to #include a lot of code by accident. Don't we need tools to ensure this doesn't regress? Like something which warns "this git commit increases the amount of compiled code by 20%".

phkahler 5y ago
Not sure, but isn't a large part of it keeping #include out of header files? That's easy to spot, but can be hard to untangle once its done.

egberts 5y ago

I’ve embarked down this same path with GCC 2.7.4 from a … while … back. Ingo is on the right track. I’ve always thought that this stuff should have been dealt with by the GCC (in form of pre-cached header files). So sad that it is a thorny issue to this day (for large scale projects); so glad that this issue is now alive so it can be dealt with once and for all. I envision some type of *.ho filetype to hold these quasi-compiled headers for the next build iteration.

folmar 5y ago
Do you mean .pch? I think it works quite well generally, but for the kernel it might be too fragile as it is not so widely used.
pas 5y ago
"... dealt with once and for all." Is at least one GCC maintainer looking at this? Otherwise it'll likely not be dealt with "for all", right? On the kernel side every new dependency has to be added with keeping in mind this problem as long as there's no automation to help with this it'll be a problem.

egberts 5y ago

And Ingo probably had the greatest heads up over the rest of us by having his new kernel perform the fastest iterative kernel rebuilds, probably using all permutation of kernel option settings for rebuildings.

commandlinefan 5y ago

> I got to over 500 commits in this tree and had to throw away my second attempt But wait, why didn't you just break the task down into sub-tasks that could be estimated in four-hour increments in the first place? Every single "scrum master" I've ever labored under has insisted that every software development task can be broken down this way.

Arch-TK 5y ago
Clearly Ingo is just an inexperienced developer who struggles to do this simple task. Maybe he needs more training?
klysm 5y ago
I hadn’t thought about it, but this is a great counter example to the (bad) assumptions of scrum.

astrostl 5y ago

I find it fascinating to consider the potential impact on energy consumption, human time, hardware and hosting costs, etc. of just one patch to just one thing.

mtnygard 5y ago
I was wondering the same thing. Something like the Linux kernel gets built and rebuilt constantly by build systems and users all over the world. Even marginal improvements in build cost become big when there's such a large multiplier.

rektide 5y ago

Linux 6!! Do it!

geophertz 5y ago

From what I understand, this was mostly done manually. Would it have been possible to represent dependencies as a graph to obtain an optimal result?

phendrenad2 5y ago

Don't driver files just include whatever headers they want? Does this mean that out-of-mainline drivers will need to be rewritten?

pas 5y ago
> Does this mean that out-of-mainline drivers will need to be rewritten? Aren't they always anyway? Plus if the driver can be built separately then it can include whatever it wants.

habanany 5y ago

999900000999 5y ago

How do you even QA this.