7 ms·

From: Ingo Molnar Subject: [PATCH 0000/2297] Eliminate the Linux kernel's "Dependency Hell" ... 25,288 files changed, 178,024 insertions(+), 74

From: Ingo Molnar Subject: [PATCH 0000/2297] Eliminate the Linux kernel's "Dependency Hell" ... 25,288 files changed, 178,024 insertions(+), 74,720 deletions(-)

londons_explore 5y ago
You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. Think how many lines are now duplicate (and therefore need to be updated in twice as many places, and bugs introduced when a copy is missed). Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. If the 100k lines were in their own subdirectory and added a major new feature, it would be worth it. But spread across the whole codebase, and introducing more duplication and chances for bugs I think outweighs a 'neater' header file system. The 'dependency hell' is only an issue for people who only want to build part of the kernel anyway. If you're building the whole lot, you might as well just include everything and it'll work great.
- tedunangst 5y ago
  Maybe the people developing the project have some insights into which build improvements they would benefit from. Having to recompile the entire tree every time you change a seemingly unrelated header gets old fast.
- derefr 5y ago
  > [This patch series] decouples much of the high level headers from others, uninlining of unnecessary functions, a decoupling of the type and API headers, automated dependency handling of header files, and a variety of other changes That's a lot more than "shaving the build time." As far as I understand it, the goal of this patchset isn't to improve the build time; that's just a nice consequence. The goal was to refactor the header-file hierarchy to make it more maintainable and less "brittle." Sometimes, increasing maintainability requires more code. (Almost always, if the current version is a terse mess of mixed concerns.) Think of it this way: take an IOCCC entry, and de-obfuscate it. You're "increasing the size of the codebase." You might even be "duplicating" some things (e.g. magic constants that were forcefully squashed together because they happened to share a value, which are now separate constants per semantic meaning.) But doing this obviously increases the maintainability of the code.
  - XorNot 5y ago
    I'd say cycle time is also important anyway: we spend a lot of time building the Linux kernel in various forms. The savings across the board in manhours waiting for compilation, or for git bisect, are not insubstantial. I wonder what the memory savings for compilation look like? Because that's also potentially more workers in automated testing farms for the same cost.
    - derefr 5y ago
      I've never tried it myself, but presuming you build the kernel in modular rather than monolithic mode, wouldn't the incremental compiles during git-bisect et al already be pretty quick? You'd only be rebuilding the modules whose source files changed, and those (presumably) wouldn't be pulling in the entire header tree. (Or would they, with that being "the problem"?)
      - tedunangst 5y ago
        That is the problem.
- aaomidi 5y ago
- bsder 5y ago
  > You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. In Ingo's post, he points out that the main speedup is coming from the fact that the expansion after the C preprocessor step is a LOT smaller. That's a lot of decoupling. As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. Having those data structures in files actually mentioned by the file you're working on would be a huge cognitive load improvement as well as make tool assistance much more plausible. Even if this particular patch doesn't land, it lights the path. What types of changes need to be made are now clear. How many changes are required before you see the improvement is now clear. With those, the changes required can be driven down into the maintainers and rolled out incrementally, if desired.
  - yjftsjthsd-h 5y ago
    > As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. As someone who tried poking around: Oh good; I assumed I was just missing something. Alternatively, oh no; I had assumed I was missing something and there was a more elegant tool out there.
    - alexforencich 5y ago
      https://elixir.bootlin.com/linux/latest/source https://elixir.bootlin.com/linux/latest/source can be far more helpful than grep for this sort of thing
  - wott 5y ago
    > As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. It has always amazed me how finding something which originally seemed a trivial little thing, usually meant going through a chain of #defines and typedefs across many header files. It's the same with GLibc, by the way. It' a bit like when you hike to a summit by following a crest path: you always think the next hump in sight is the right one, your destination, the promise land; and when you reach it, dammit, it wasn't, your goal is actually the next one. Or perhaps the next after the next. Or...
- Supermancho 5y ago
  > You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. When you're talking a project of half a million lines, sure. The Linux kernel has around 27.8 million lines of code. An increase of .35% > Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. Why add features at all? Code has a purpose. Sometimes bringing code into a static context is a net good. It was going to be generated at runtime anyway. > If you're building the whole lot, you might as well just include everything and it'll work great. That's not strictly true, but it's true for these features, which is a stated reasoning.
  - kangalioo 5y ago
    Good golly, that 0.35% figure really puts the patch set into perspective A library I'm working on is 7000 LOC which seems pretty sizable, but 0.35% of that is 25 LOC. The Linux patch set is actually tiny
  - theon144 5y ago
    >The Linux kernel has around 27.8 million lines of code. An increase of .35% This is horribly misleading; most of these lines of code are drivers, which this patchset doesn't even concern. It's still a massive change that only a handful of developers will ever be able to review in entirety - a fact to which the size of the project is completely irrelevant - if anything, actually, it urges even more caution, given the implied complexity. Which I believe was (at least in part) parent comment's point - given the importance and ubiquity of the Linux kernel, this may be concerning. That said, I am very confident in the structures put in place by the kernel devs, their competence and the necessity for such a change - but trivializing a 100k LoC patchset because the project it's intended to land in is even more colossally complex isn't how I'd choose my approach.
    - Denvercoder9 5y ago
      > This is horribly misleading; most of these lines of code are drivers, which this patchset doesn't even concern. That's not true at all, a big part of those added lines are added includes in drivers. E.g. this commit I picked at random adds 1500 lines, of which just a few procent are in the core kernel: https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git/commit/?id=390dec865b2afb6219914d08b4b5c870d1f052a6 https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.gi...
- stjohnswarts 5y ago
  You need to read the article. It explains all this.
- fargle 5y ago
  > You have to have an awfully good reason to add 100k lines of code to a project yes. and this is very, very good reason. As another poster said, you at 0.35% lines to make it compile almost twice as fast? And you're not happy about that? > Think how many lines are now duplicate (and therefore need to be updated in twice ... OK, how many? None! That's how many. Adding proper header dependencies to the .c module doesn't duplicate anything. Unless you think adding #include <stdio.h> in every module somehow creates unmaintainable duplication. > Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. OK. Hrm... I think a lot, lot less is how much. That's the whole point of a major cleanup like this. Proper decoupling. Headers that you use are obvious where they belong not being brought in with some action-at-a-distance accident.
kangalioo 5y ago
Where did the additional 100kLOC come from?
- crishoj 5y ago
  I suspect it’s this: “Automated dependency addition to .h and .c files. This is about 790 commits, which are strictly limited to trivial addition of header dependencies. These patches make indirect dependencies explicit, and thus prepare the tree for aggressive removal of cross-dependencies from headers.”
- jontro 5y ago
  Did a brief scan of the repo. Looks like a lot of it comes from adding new headers. This commit adds +#include <linux/wait_api.h> to 1627 files for instance. https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git/commit/?id=4a3fb303583079cdf40f2bbb2d2eaedb664b3558 https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.gi...
charcircuit 5y ago
>2298 emails When will Linux move to a site like github / gitlab / something similar self hosted that supports people proposing changes without having to send every single person interested in the development of Linux thousands of emails.
- CRConrad 5y ago
  It's a fucking shame how many people nowadays can't grok the difference between git and GitHub. Your question is fundamentally wrong; totally bass-ackwards. The correct question is: When will all the zillion other projects move off of GitHub / GitLab / all the other usurper sites?
- ReactiveJelly 5y ago
  I think having everyone see all the emails is part of the point. Even with self-hosted Gitea, someone has to _host_ it. And there isn't a trivial export button, and if the host goes rogue they can make it inaccessible and make everyone lose efficiency while they swap back to the old method. And if you need email to authenticate on a website, why not just use email anyway? But this is an interesting idea, and I'm on the look-out for ideas to replace email. I'll keep "LKML" in the back of my head as a use case.
  - charcircuit 5y ago
    >And there isn't a trivial export button It does support making backups of repositories at least. >and if the host goes rogue they can make it inaccessible and make everyone lose efficiency while they swap back to the old method. Isn't this also a problem if the person managing the email list goes rogue? You have to trust someone to host the infrastructure. >And if you need email to authenticate on a website, why not just use email anyway? Because Email may not be the optimal user interface for handling issues, pull requests, code review, etc.
    - xyzzy_plugh 5y ago
      > Isn't this also a problem if the person managing the email list goes rogue? You have to trust someone to host the infrastructure. It depends on what kind of threat you are worried about. The beautiful thing is even if LKML was abused and ruined tomorrow, nothing is lost or damaged. The emails have already been sent, and moving to a new list is more of a nuisance at this point. If you're worried about authenticating a patch series, committers can sign their commits a number of ways. Lastly, a large chunk of Linux development happens off-list, with subsystem maintainers building branches for Linus or Greg to pull from.

From: Ingo Molnar Subject: [PATCH 0000/2297] Eliminate the Linux kernel's "Dependency Hell" ... 25,288 files changed, 178,024 insertions(+), 74,720 deletions(-)

londons_explore 5y ago

You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. Think how many lines are now duplicate (and therefore need to be updated in twice as many places, and bugs introduced when a copy is missed). Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. If the 100k lines were in their own subdirectory and added a major new feature, it would be worth it. But spread across the whole codebase, and introducing more duplication and chances for bugs I think outweighs a 'neater' header file system. The 'dependency hell' is only an issue for people who only want to build part of the kernel anyway. If you're building the whole lot, you might as well just include everything and it'll work great.

tedunangst 5y ago
Maybe the people developing the project have some insights into which build improvements they would benefit from. Having to recompile the entire tree every time you change a seemingly unrelated header gets old fast.
derefr 5y ago
> [This patch series] decouples much of the high level headers from others, uninlining of unnecessary functions, a decoupling of the type and API headers, automated dependency handling of header files, and a variety of other changes That's a lot more than "shaving the build time." As far as I understand it, the goal of this patchset isn't to improve the build time; that's just a nice consequence. The goal was to refactor the header-file hierarchy to make it more maintainable and less "brittle." Sometimes, increasing maintainability requires more code. (Almost always, if the current version is a terse mess of mixed concerns.) Think of it this way: take an IOCCC entry, and de-obfuscate it. You're "increasing the size of the codebase." You might even be "duplicating" some things (e.g. magic constants that were forcefully squashed together because they happened to share a value, which are now separate constants per semantic meaning.) But doing this obviously increases the maintainability of the code.
- XorNot 5y ago
  I'd say cycle time is also important anyway: we spend a lot of time building the Linux kernel in various forms. The savings across the board in manhours waiting for compilation, or for git bisect, are not insubstantial. I wonder what the memory savings for compilation look like? Because that's also potentially more workers in automated testing farms for the same cost.
  - derefr 5y ago
    I've never tried it myself, but presuming you build the kernel in modular rather than monolithic mode, wouldn't the incremental compiles during git-bisect et al already be pretty quick? You'd only be rebuilding the modules whose source files changed, and those (presumably) wouldn't be pulling in the entire header tree. (Or would they, with that being "the problem"?)
    - tedunangst 5y ago
      That is the problem.
aaomidi 5y ago
bsder 5y ago
> You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. In Ingo's post, he points out that the main speedup is coming from the fact that the expansion after the C preprocessor step is a LOT smaller. That's a lot of decoupling. As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. Having those data structures in files actually mentioned by the file you're working on would be a huge cognitive load improvement as well as make tool assistance much more plausible. Even if this particular patch doesn't land, it lights the path. What types of changes need to be made are now clear. How many changes are required before you see the improvement is now clear. With those, the changes required can be driven down into the maintainers and rolled out incrementally, if desired.
- yjftsjthsd-h 5y ago
  > As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. As someone who tried poking around: Oh good; I assumed I was just missing something. Alternatively, oh no; I had assumed I was missing something and there was a more elegant tool out there.
  - alexforencich 5y ago
    https://elixir.bootlin.com/linux/latest/source https://elixir.bootlin.com/linux/latest/source can be far more helpful than grep for this sort of thing
- wott 5y ago
  > As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. It has always amazed me how finding something which originally seemed a trivial little thing, usually meant going through a chain of #defines and typedefs across many header files. It's the same with GLibc, by the way. It' a bit like when you hike to a summit by following a crest path: you always think the next hump in sight is the right one, your destination, the promise land; and when you reach it, dammit, it wasn't, your goal is actually the next one. Or perhaps the next after the next. Or...
Supermancho 5y ago
> You have to have an awfully good reason to add 100k lines of code to a project... And I don't think merely shaving 50% off the build time is enough. When you're talking a project of half a million lines, sure. The Linux kernel has around 27.8 million lines of code. An increase of .35% > Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. Why add features at all? Code has a purpose. Sometimes bringing code into a static context is a net good. It was going to be generated at runtime anyway. > If you're building the whole lot, you might as well just include everything and it'll work great. That's not strictly true, but it's true for these features, which is a stated reasoning.
- kangalioo 5y ago
  Good golly, that 0.35% figure really puts the patch set into perspective A library I'm working on is 7000 LOC which seems pretty sizable, but 0.35% of that is 25 LOC. The Linux patch set is actually tiny
- theon144 5y ago
  >The Linux kernel has around 27.8 million lines of code. An increase of .35% This is horribly misleading; most of these lines of code are drivers, which this patchset doesn't even concern. It's still a massive change that only a handful of developers will ever be able to review in entirety - a fact to which the size of the project is completely irrelevant - if anything, actually, it urges even more caution, given the implied complexity. Which I believe was (at least in part) parent comment's point - given the importance and ubiquity of the Linux kernel, this may be concerning. That said, I am very confident in the structures put in place by the kernel devs, their competence and the necessity for such a change - but trivializing a 100k LoC patchset because the project it's intended to land in is even more colossally complex isn't how I'd choose my approach.
  - Denvercoder9 5y ago
    > This is horribly misleading; most of these lines of code are drivers, which this patchset doesn't even concern. That's not true at all, a big part of those added lines are added includes in drivers. E.g. this commit I picked at random adds 1500 lines, of which just a few procent are in the core kernel: https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git/commit/?id=390dec865b2afb6219914d08b4b5c870d1f052a6 https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.gi...
stjohnswarts 5y ago
You need to read the article. It explains all this.
fargle 5y ago
> You have to have an awfully good reason to add 100k lines of code to a project yes. and this is very, very good reason. As another poster said, you at 0.35% lines to make it compile almost twice as fast? And you're not happy about that? > Think how many lines are now duplicate (and therefore need to be updated in twice ... OK, how many? None! That's how many. Adding proper header dependencies to the .c module doesn't duplicate anything. Unless you think adding #include <stdio.h> in every module somehow creates unmaintainable duplication. > Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file. OK. Hrm... I think a lot, lot less is how much. That's the whole point of a major cleanup like this. Proper decoupling. Headers that you use are obvious where they belong not being brought in with some action-at-a-distance accident.

kangalioo 5y ago

Where did the additional 100kLOC come from?

crishoj 5y ago
I suspect it’s this: “Automated dependency addition to .h and .c files. This is about 790 commits, which are strictly limited to trivial addition of header dependencies. These patches make indirect dependencies explicit, and thus prepare the tree for aggressive removal of cross-dependencies from headers.”
jontro 5y ago
Did a brief scan of the repo. Looks like a lot of it comes from adding new headers. This commit adds +#include <linux/wait_api.h> to 1627 files for instance. https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git/commit/?id=4a3fb303583079cdf40f2bbb2d2eaedb664b3558 https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.gi...

charcircuit 5y ago

>2298 emails When will Linux move to a site like github / gitlab / something similar self hosted that supports people proposing changes without having to send every single person interested in the development of Linux thousands of emails.

CRConrad 5y ago
It's a fucking shame how many people nowadays can't grok the difference between git and GitHub. Your question is fundamentally wrong; totally bass-ackwards. The correct question is: When will all the zillion other projects move off of GitHub / GitLab / all the other usurper sites?
ReactiveJelly 5y ago
I think having everyone see all the emails is part of the point. Even with self-hosted Gitea, someone has to _host_ it. And there isn't a trivial export button, and if the host goes rogue they can make it inaccessible and make everyone lose efficiency while they swap back to the old method. And if you need email to authenticate on a website, why not just use email anyway? But this is an interesting idea, and I'm on the look-out for ideas to replace email. I'll keep "LKML" in the back of my head as a use case.
- charcircuit 5y ago
  >And there isn't a trivial export button It does support making backups of repositories at least. >and if the host goes rogue they can make it inaccessible and make everyone lose efficiency while they swap back to the old method. Isn't this also a problem if the person managing the email list goes rogue? You have to trust someone to host the infrastructure. >And if you need email to authenticate on a website, why not just use email anyway? Because Email may not be the optimal user interface for handling issues, pull requests, code review, etc.
  - xyzzy_plugh 5y ago
    > Isn't this also a problem if the person managing the email list goes rogue? You have to trust someone to host the infrastructure. It depends on what kind of threat you are worried about. The beautiful thing is even if LKML was abused and ruined tomorrow, nothing is lost or damaged. The emails have already been sent, and moving to a new list is more of a nuisance at this point. If you're worried about authenticating a patch series, committers can sign their commits a number of ways. Lastly, a large chunk of Linux development happens off-list, with subsystem maintainers building branches for Linus or Greg to pull from.