6 ms·

I think the question is more how to find/create more people who would spend a good chunk of their productivity on what sounds like a massively boring rewrite th

I think the question is more how to find/create more people who would spend a good chunk of their productivity on what sounds like a massively boring rewrite that has marginal returns on investment (Moore’s law says that 50-80% gains will be made up in less than a year by hardware getting faster, and even if it has slowed down it isn’t completely out of the picture). That does take a special kind of person to see this as an ambitious project and not a boring one.

fouc 5y ago
I find it hard to see how any developer would describe it as "massively boring". At a basic level, every time a coder solves a problem, there's a dopamine hit. The greater the familiarity with the material, the easier it is to solve problems. Anyone that spends a sufficient amount of time in a niche will develop that familiarity.
- denton-scratch 5y ago
  Refactoring can be satisfying; but spending two years refactoring kernel headers for a gain in build performance but not in runtime performance, is objectively "massively boring". I've attempted something similar myself - a refactoring of a very dirty Java project that was orders of magnitude smaller than the Linux kernel. I didn't have good refactoring tools, nor even a good SCCS; my month-long effort had to be backed-out. I didn't enjoy it - it was boring, but needed doing, because the accumulating cruft was impacting my ability to work on the codebase (and presumably everyone else's). The project stayed dirty, and my reputation suffered. Credit to Ingo for his perseverance. That is massive commitment.
  - whizzter 5y ago
    Big turds often leaves refactors almost equal to Joels "V2 rewrites", continuous runnable general cleanups is often the first step (and yields surprising benefits in general) then still runnable prep refactors that makes the bigger leaps smaller (temporarily increasing complexity so you really want them in just before embarking on the bigger stuff so it doesn't increase complexity in itself). That said, in your case it felt a bit like an organizational failure also. As you could read in Ingo's mail he had to start over after plenty of work put in, for him there wasn't any reputation damage because he seems to have had free reign to start this as he wished.
  - fouc 5y ago
    > but spending two years refactoring kernel headers for a gain in build performance but not in runtime performance, is objectively "massively boring". It makes a big difference for the thousands of kernel developers [0]. There's also more than a million linux users globally[2], assuming only 1% might compile their own kernel, that still leaves tens of thousands of linux users benefiting. Not to forget about the side benefits of a refactor. There could be runtime improvements coming out of this too down the road. [1] https://en.wikipedia.org/wiki/Linux_kernel https://en.wikipedia.org/wiki/Linux_kernel [2] https://www.zdnet.com/article/how-many-linux-users-are-there-anyway/ https://www.zdnet.com/article/how-many-linux-users-are-there...
hermitdev 5y ago
Betting on Moore's law to speed up your software hasn't held for more than a decade. Software keeps getting worse and hardware has largely plateau'd in what you can expect to get in single threaded performance. And besides, running a CPU for 2-5x as long to accomplish the same task is wasteful in both time and energy, both of which are scarce resources. Especially for something that is done as many times as rebuilding the Linux kernel.
- foxhop 5y ago
  Not all software is getting worse. : )
deleted 5y ago
[deleted]
Dylan16807 5y ago
If you could stay 50% ahead of Moore's law in general, that would be worth tens of billions of dollars per year, increasing as Moore's law slows. Your returns don't become marginal until an iteration is so fast that it might as well be instant, and there's no benefit in doing more iterations per second. The kernel isn't in that range.
- kaliszad 5y ago
  We would stay ahead of the curve on maintenance, if the kernel and other software was easier to live-patch and it was the default & very stable in pretty much all relevant distributions. No more clusters just for the sake of being able to update during business hours. Also, as a Clojure dev, you cannot recompile the running kernel while using it at the same time using something akin to REPL driven development. That would be the other side of the live-patch productivity boost in so many areas. If I am not mistaken, the CPU and other devices don't care, if everything is compiled ahead of time and a few functions tacked on later, as they are changed by e.g. patches, or if the kernel is completely recompiled. Obviously, I am simplifying a lot but it is a very serious question. We really should strive to build maintainable systems that don't have to reboot just to apply updates.
  - pabs3 5y ago
    The Linux kernel already supports live-patching, but as I understand it, the patches themselves are hard to produce or can't be produced automatically, so to be able to produce them you need a team that understands the process, which limits the live patching to commercial distros that can afford a team for this.
    - kaliszad 5y ago
      I know, and that is exactly my point. It isn't like "send to REPL" in pretty much any half-decent LISP. And it hurts development comfort, speed and maintenance is way more burdensome than what it could be. Here, you would need to involve a compiler of course and do more due diligence and then run ftrace or something to exchange the code. Also, you would probably in some cases need to make sure, the ordering of code is right as to not hurt performance. (See Emery Berger's talk https://www.youtube.com/watch?v=7g1Acy5eGbE https://www.youtube.com/watch?v=7g1Acy5eGbE which probably applies here as well.) Of course, with great power comes great responsibility. You could add some optional checking facility, that would be able to roll back the changes to code if some conditions were met.
wbl 5y ago
Denard scaling has been dead my entire career. Moore's law was never about speed.
- Dylan16807 5y ago
  This is compilation of thousands of files. It doesn't really depend on Dennard scaling.
Ericson2314 5y ago
As a bit of an extremely refactorer myself, there are some of us who like adding features, but others who just like "kneading" the code.
- jonathanlydall 5y ago
  Was going to say this as it somewhat applies to me, or as my business partner says I’m a bit of a “sweeper”. I seem to get satisfaction from things that a lot of others might find tedious/boring. People like us though just need to watch ourselves that what we’re doing is actually useful opposed to just scratching an itch. Which of course is not exclusive to us as developers often find many other ways of scratching itches without actually adding value.
  - Ericson2314 5y ago
    It is healthy to be self aware about outcomes vs just indulging, but I am not longer so worried. On my best projects, I've done the first 90% so that a great many other can collective do the 2nd, 3rd, and 4th 90%s :). Without being "goal oriented" you can feel good about unlocking that part that would be too annoying for others, and they in turn can do the finishing work that wouldn't be fun for you. Don't feel like you're not doing important work just because others are slotting in the keystones.
- noobermin 5y ago
  People need to be taught to read code and not only write it. I feel like a lot of problems start with people seeing more than 100 lines then saying "fuck it" then start clamoring to remove technical debt by adopting something new or even an writing an entire new code base. Reading code is fun and it's a skill that took me time to develop. People need to told that's a skill you need and yes (shock) not emphasize creativity as much as we do. Creativity is great but it has enough encouragement in society already. We need people willing to read what already exists as well.
  - voakbasda 5y ago
    The problem is that most code is utter shit. If you just read shit code and do nothing, you have made a choice to preserve the status quo. At that point, you have become part of the problem.
marcan_42 5y ago
Kernel builds have been getting slower, not faster. You aren't going to make up the gains, because other things will slow you down again. Therefore, this improvement will be at least a linear improvement in kernel build times for the foreseeable future, if not possibly supra-linear depending on how things work out.
phs318u 5y ago
I think there's also an element of "ownership" to the kernel for many of the OG maintainers. It's their "baby". And as every parent knows, you would do a lot more for your babies than for anyone else. It may well be that same sense of ownership creates an "inner circle" effect that is a barrier to entry for creating more such developers.
- regularfry 5y ago
  The inner circle effect is absolutely real. Some projects get rid of it with very loose commit-bit policies, but even getting to that point can be painful. There are software projects I'd like to contribute to and have (I think) reasonable ideas about, but no way in hell am I running the gauntlet of the communities involved.
colechristensen 5y ago
Some people like this kind of thing and just seem to be predisposed to wanting to do this kind of work. Good team builders will find a variety of different employee talents and interests and let people do what they like and develop those skills.
floatingatoll 5y ago
I would love to do more work like this at work, but since I can’t guarantee the results up front, no one is interested and code rot sets in. “Minimum impact, equivalency-first rewrites” are the gold standard of optimization and DevOps badly underestimates their value.

fouc 5y ago

I find it hard to see how any developer would describe it as "massively boring". At a basic level, every time a coder solves a problem, there's a dopamine hit. The greater the familiarity with the material, the easier it is to solve problems. Anyone that spends a sufficient amount of time in a niche will develop that familiarity.

denton-scratch 5y ago
Refactoring can be satisfying; but spending two years refactoring kernel headers for a gain in build performance but not in runtime performance, is objectively "massively boring". I've attempted something similar myself - a refactoring of a very dirty Java project that was orders of magnitude smaller than the Linux kernel. I didn't have good refactoring tools, nor even a good SCCS; my month-long effort had to be backed-out. I didn't enjoy it - it was boring, but needed doing, because the accumulating cruft was impacting my ability to work on the codebase (and presumably everyone else's). The project stayed dirty, and my reputation suffered. Credit to Ingo for his perseverance. That is massive commitment.
- whizzter 5y ago
  Big turds often leaves refactors almost equal to Joels "V2 rewrites", continuous runnable general cleanups is often the first step (and yields surprising benefits in general) then still runnable prep refactors that makes the bigger leaps smaller (temporarily increasing complexity so you really want them in just before embarking on the bigger stuff so it doesn't increase complexity in itself). That said, in your case it felt a bit like an organizational failure also. As you could read in Ingo's mail he had to start over after plenty of work put in, for him there wasn't any reputation damage because he seems to have had free reign to start this as he wished.
- fouc 5y ago
  > but spending two years refactoring kernel headers for a gain in build performance but not in runtime performance, is objectively "massively boring". It makes a big difference for the thousands of kernel developers [0]. There's also more than a million linux users globally[2], assuming only 1% might compile their own kernel, that still leaves tens of thousands of linux users benefiting. Not to forget about the side benefits of a refactor. There could be runtime improvements coming out of this too down the road. [1] https://en.wikipedia.org/wiki/Linux_kernel https://en.wikipedia.org/wiki/Linux_kernel [2] https://www.zdnet.com/article/how-many-linux-users-are-there-anyway/ https://www.zdnet.com/article/how-many-linux-users-are-there...

hermitdev 5y ago

Betting on Moore's law to speed up your software hasn't held for more than a decade. Software keeps getting worse and hardware has largely plateau'd in what you can expect to get in single threaded performance. And besides, running a CPU for 2-5x as long to accomplish the same task is wasteful in both time and energy, both of which are scarce resources. Especially for something that is done as many times as rebuilding the Linux kernel.

foxhop 5y ago
Not all software is getting worse. : )

deleted 5y ago

[deleted]

Dylan16807 5y ago

If you could stay 50% ahead of Moore's law in general, that would be worth tens of billions of dollars per year, increasing as Moore's law slows. Your returns don't become marginal until an iteration is so fast that it might as well be instant, and there's no benefit in doing more iterations per second. The kernel isn't in that range.

kaliszad 5y ago
We would stay ahead of the curve on maintenance, if the kernel and other software was easier to live-patch and it was the default & very stable in pretty much all relevant distributions. No more clusters just for the sake of being able to update during business hours. Also, as a Clojure dev, you cannot recompile the running kernel while using it at the same time using something akin to REPL driven development. That would be the other side of the live-patch productivity boost in so many areas. If I am not mistaken, the CPU and other devices don't care, if everything is compiled ahead of time and a few functions tacked on later, as they are changed by e.g. patches, or if the kernel is completely recompiled. Obviously, I am simplifying a lot but it is a very serious question. We really should strive to build maintainable systems that don't have to reboot just to apply updates.
- pabs3 5y ago
  The Linux kernel already supports live-patching, but as I understand it, the patches themselves are hard to produce or can't be produced automatically, so to be able to produce them you need a team that understands the process, which limits the live patching to commercial distros that can afford a team for this.
  - kaliszad 5y ago
    I know, and that is exactly my point. It isn't like "send to REPL" in pretty much any half-decent LISP. And it hurts development comfort, speed and maintenance is way more burdensome than what it could be. Here, you would need to involve a compiler of course and do more due diligence and then run ftrace or something to exchange the code. Also, you would probably in some cases need to make sure, the ordering of code is right as to not hurt performance. (See Emery Berger's talk https://www.youtube.com/watch?v=7g1Acy5eGbE https://www.youtube.com/watch?v=7g1Acy5eGbE which probably applies here as well.) Of course, with great power comes great responsibility. You could add some optional checking facility, that would be able to roll back the changes to code if some conditions were met.

wbl 5y ago

Denard scaling has been dead my entire career. Moore's law was never about speed.

Dylan16807 5y ago
This is compilation of thousands of files. It doesn't really depend on Dennard scaling.

Ericson2314 5y ago

As a bit of an extremely refactorer myself, there are some of us who like adding features, but others who just like "kneading" the code.

jonathanlydall 5y ago
Was going to say this as it somewhat applies to me, or as my business partner says I’m a bit of a “sweeper”. I seem to get satisfaction from things that a lot of others might find tedious/boring. People like us though just need to watch ourselves that what we’re doing is actually useful opposed to just scratching an itch. Which of course is not exclusive to us as developers often find many other ways of scratching itches without actually adding value.
- Ericson2314 5y ago
  It is healthy to be self aware about outcomes vs just indulging, but I am not longer so worried. On my best projects, I've done the first 90% so that a great many other can collective do the 2nd, 3rd, and 4th 90%s :). Without being "goal oriented" you can feel good about unlocking that part that would be too annoying for others, and they in turn can do the finishing work that wouldn't be fun for you. Don't feel like you're not doing important work just because others are slotting in the keystones.
noobermin 5y ago
People need to be taught to read code and not only write it. I feel like a lot of problems start with people seeing more than 100 lines then saying "fuck it" then start clamoring to remove technical debt by adopting something new or even an writing an entire new code base. Reading code is fun and it's a skill that took me time to develop. People need to told that's a skill you need and yes (shock) not emphasize creativity as much as we do. Creativity is great but it has enough encouragement in society already. We need people willing to read what already exists as well.
- voakbasda 5y ago
  The problem is that most code is utter shit. If you just read shit code and do nothing, you have made a choice to preserve the status quo. At that point, you have become part of the problem.

marcan_42 5y ago

Kernel builds have been getting slower, not faster. You aren't going to make up the gains, because other things will slow you down again. Therefore, this improvement will be at least a linear improvement in kernel build times for the foreseeable future, if not possibly supra-linear depending on how things work out.

phs318u 5y ago

I think there's also an element of "ownership" to the kernel for many of the OG maintainers. It's their "baby". And as every parent knows, you would do a lot more for your babies than for anyone else. It may well be that same sense of ownership creates an "inner circle" effect that is a barrier to entry for creating more such developers.

regularfry 5y ago
The inner circle effect is absolutely real. Some projects get rid of it with very loose commit-bit policies, but even getting to that point can be painful. There are software projects I'd like to contribute to and have (I think) reasonable ideas about, but no way in hell am I running the gauntlet of the communities involved.

colechristensen 5y ago

Some people like this kind of thing and just seem to be predisposed to wanting to do this kind of work. Good team builders will find a variety of different employee talents and interests and let people do what they like and develop those skills.

floatingatoll 5y ago

I would love to do more work like this at work, but since I can’t guarantee the results up front, no one is interested and code rot sets in. “Minimum impact, equivalency-first rewrites” are the gold standard of optimization and DevOps badly underestimates their value.