21 ms·

Ingo Molnar was always one of my favourite scary kernel devs. This kind of massive rewrite was always a key strength of OG devs, who were prepared to tackle mai

• yjftsjthsd-h 5y ago
  A question that I've been mulling over: How do we create more people like this? Better documentation? A... "brief guide to the entire kernel" blog series? Aggressively pushing GSoC project mentoring?
  • mtrycz2 5y ago
    Motivation.
  • stathibus 5y ago
    The same way you build talent in any other organization - give people money, tell them this is their job, and put them in the same "room" as people who also have this job but have been doing it longer. Or are you asking how we create more people who will do this in their free time?
    • voakbasda 5y ago
      These people already exist. They do not need to be manufactured. I think many get turned away from the Linux community for non-technical reasons. For some, the community takes a bigger personal toll than the technical challenges. I personally gave up over that aspect and stopped contributing. I did not feel welcome or needed.
      • eyegor 5y ago
        I have tried to contribute to the Linux kernel development in the past, but I found it to be a strong inner circle of seasoned devs. The fact that they were still using mailing lists instead of github/gitlab/etc to keep track of development was painful enough, but they were also hostile to asking for documentation on inner modules. I'm not sure how it will progress when they retire or die off.
        • lproven 5y ago
          I don't have anything to add to your overall point – I'm sure you're right. I just wanted to address two smaller ones: > The fact that they were still using mailing lists In my personal experience, going back some 30 years or so online with my own personal paid email account – which is still live and still works – I find that most people I've worked or interacted with who dislike using email for workflow or other important comms do not know how to use email effectively. Strange as it may seem, this now applies to the developers of most email clients and online email services. Email and mailing lists are remarkably powerful and capable tools, if used properly. Most people do not use it or them properly. I have yet to see anything else by anyone that is an improvement in every way on email. > I'm not sure how it will progress when they retire or die off. TBPH, as a Linux user and professional, I rather hope it does not. Linux is an amazingly useful tool, but OTOH it's now, in and of itself, a vast hairball of technical debt. The traditional UNIX model itself is. It is long past time that we should have moved past it on to better things. There have been many attempts but none have ever achieved critical mass. At the rate things are going it looks quite likely that our technological civilization will collapse due to out-of-control global warming and the ongoing and accelerating mass extinction event. As the parent of a 2YO I very much hope I am wrong. But if I am and it doesn't, the world needs better tech, and a ½ century old UNIX clone already does not really cut it today. If Linux and all the other monolithic Unixes die when their dev teams die, we'll have to move on to newer, smaller, simpler systems that a new generation of programmers can actually read from top to bottom, understand, and do useful work on. If monolithic Unices become the mid-21st-century COBOL, just kept around for a few old systems and only critical bugs patched, that will be a big win for us all.
          • 5y ago
    • pclmulqdq 5y ago
      You create a lot of unimpressive people by doing that too. You need someone to have passion for a project like this to do a huge rewrite. It's pretty clear that passion can't come from money.
      • stathibus 5y ago
        This is obviously false, the world is full of people who are passionate about what they get paid to do.
        • Spivak 5y ago
          Yeah, and for “professionals” it’s pretty rare to find someone that doesn’t at least like the field they chose, spent 4+ years of university on, and applied for jobs in. It’s vanishingly rare to find someone exceptionally talented in a field they hate because of the practice it takes to get to that level.
          • nik282000 5y ago
            >It’s vanishingly rare to find someone exceptionally talented in a field they hate because of the practice it takes to get to that level. So what do I win?
            • saurik 5y ago
              I mean, the prize is really bad :(.
        • pclmulqdq 5y ago
          The world is also full of people who are not passionate about what they get paid to do, and they do it because it is high-paying and offers security.
        • numpad0 5y ago
          I would say the world is full of people who are passionate about getting paid, which makes best course of action to be not doing a thing while convincing others you do, which IMO is harmful.
    • IgorPartola 5y ago
      I think the question is more how to find/create more people who would spend a good chunk of their productivity on what sounds like a massively boring rewrite that has marginal returns on investment (Moore’s law says that 50-80% gains will be made up in less than a year by hardware getting faster, and even if it has slowed down it isn’t completely out of the picture). That does take a special kind of person to see this as an ambitious project and not a boring one.
      • fouc 5y ago
        I find it hard to see how any developer would describe it as "massively boring". At a basic level, every time a coder solves a problem, there's a dopamine hit. The greater the familiarity with the material, the easier it is to solve problems. Anyone that spends a sufficient amount of time in a niche will develop that familiarity.
        • denton-scratch 5y ago
          Refactoring can be satisfying; but spending two years refactoring kernel headers for a gain in build performance but not in runtime performance, is objectively "massively boring". I've attempted something similar myself - a refactoring of a very dirty Java project that was orders of magnitude smaller than the Linux kernel. I didn't have good refactoring tools, nor even a good SCCS; my month-long effort had to be backed-out. I didn't enjoy it - it was boring, but needed doing, because the accumulating cruft was impacting my ability to work on the codebase (and presumably everyone else's). The project stayed dirty, and my reputation suffered. Credit to Ingo for his perseverance. That is massive commitment.
          • whizzter 5y ago
            Big turds often leaves refactors almost equal to Joels "V2 rewrites", continuous runnable general cleanups is often the first step (and yields surprising benefits in general) then still runnable prep refactors that makes the bigger leaps smaller (temporarily increasing complexity so you really want them in just before embarking on the bigger stuff so it doesn't increase complexity in itself). That said, in your case it felt a bit like an organizational failure also. As you could read in Ingo's mail he had to start over after plenty of work put in, for him there wasn't any reputation damage because he seems to have had free reign to start this as he wished.
    • baybal2 5y ago
      Ingo wasn't working at Linux full time at the time of his peak. He was indeed a Jedi master level C programmer even before he started working at Linux kernel seriously.
    • dijonman2 5y ago
      The people who are the most capable are typically the ones who do it for fun, not money.
      • sdevonoes 5y ago
        People who does stuff for fun usually have all their money problems already resolved. Health problems -> Money problems -> Fun problems. You get the privilege to work on "fun problems" once you don't have "money" or "health" problems (otherwise, the "fun" problems are the least of your "problems").
        • dijonman2 5y ago
          I got my start in software many years ago because I had passion. Now days the market is flooded with people who want to get paid. I had free time, and an interest. Often the same characteristics of people who contribute to FOSS.
        • nicoburns 5y ago
          I don’t think that’s true. A lot of the people who have had the greatest impact on the world were never financially rewarded for it. Many aren’t even recognised for their work.
          • xxpor 5y ago
            This thread seems to be going down the path of rediscoving Maslows hierarchy of needs.
          • cmeacham98 5y ago
            It's not that they are directly compensated for their work, it's that the money problem is solved, even if from an external source. Linus Torvalds worked on Linux for a long time before being directly compensated for it, but he probably would never have done so had he needed to worry about whether he could afford his next meal and rent that month.
    • phendrenad2 5y ago
      It's not about talent, it's about clout. Not just anyone could show up and do this.
  • R0b0t1 5y ago
    I think better docs is it, but from a high level systems view. I can dive into something (say usb gadgetfs) and get a decent view and start trying to solve my problem but when I need to start learning about general kernel mannerisms I hit some walls that slowed everything down.
  • pa7ch 5y ago
    In my experience, big and important rewrites only can happen after someone has been maintaining a code base a long time and they can spend many months or years mulling over simplifying architectural changes not just for change. Often for company software at least I was frustrated with the pace that they shuffle you around projects.
    • throwaway81523 5y ago
      It's not the rewrite per se--it's squashing all the bugs that the rewrite introduces. That is why big C programs are so scary. Because it's unusually easy to introduce such bugs.
  • chx 5y ago
    > How do we create more people like this? That'd be a challenge. He is obviously a very talented person who grew up behind the Iron Curtain and went to university just after that collapsed. The resource constraints bred creativity, a certain kind of hacker mentality. The lead developer of Crysis: Warhead and Crysis 2 (and indeed most of the former Crytek Budapest team) is another extremely talented programmer emerging from the same time. Palma sub pondere crescit!
  • dataflow 5y ago
    I can't speak for Linux specifically, but political capital is also an issue. You could have people willing and able to do such things, but that wouldn't be necessarily allowed to do so in practice if they haven't already established themselves for a while.
    • marcan_42 5y ago
      It's certainly part this, but also part experience. You get a feeling for who the people in charge of kernel subsystems are and how they like their changes after a while. That goes a long way towards acceptance; there's a lot more human interaction under the surface of kernel dev than people think. Obviously this is an extreme case, but it scales. Something like 8 years ago I noticed something was horribly wrong with the ARM32 boot wrapper, submitted a fix, and my approach was mostly shot down by the maintainer. Last week I submitted a 34-patch RFC series to finally add WiFi support for 5 years' worth of Macs (which required quite a bit of new scaffolding in that driver, as well as fixes and new features) and it's gotten positive reviews so far, modulo nits. I wouldn't have been able to pull that off 8 years ago. And it's not like I spent those 8 years doing kernel dev, but I've sent in a few fixes and watched how other kernel developers work. I started on a big kernel project a year ago and all that sitting and watching has been very helpful in putting out stuff that people like.
  • Starlevel001 5y ago
    There are plenty of people with the capability to this wasting their life building adware.
  • Jetrel 5y ago
    So, I've been a person like this on a smaller scale (taking a fully written program, and basically "completely rewriting it" in a new language. Biggest thing I can say is that if you have enough of a "crazy-eyed mofo" that's just insane enough to take a big job like that, 3 things are of utmost importance: 1] remove their technical blockers. Often a person like this doesn't have a universal skillset — they're usually good, but they can't do everything. If there's a 5% of the job that's hard-blocking them, make sure it gets done so their tires don't get stuck in the mud. 2] remove their bureaucratic blockers. Make absolutely sure the project WILL proceed, even in a skunkworks capacity (this is the primary value of skunkworks — it's the ability to proceed with something you know will work, in spite of authorities expressly forbidding it because they think it won't work and don't want it to happen for what's usually a petty reason like a "it's waste of resources"). 3] remove their emotional blockers. This is really the apex — THE primary value of a person like this is not technical skill, but rather, their work ethic. A person like this has a really profound, train-engine drive to just keep soldiering on. The danger here is the problem of the "weary crusader". This willpower is considerably above average ... but it's not infallible. It's not like a superhero that's always gonna come through no matter the odds. They can break down. They can lose heart. One of the things that will constantly break them down is if people are naysaying them, and calling into question the value of their work. If they're doing a giant refactor, and people are angrily opposing it simply out of fear of change, it will break them down. As silly as it sounds, you want to coddle them — they might be unusually emotionally tough, but they are the absolute last person you want breaking down. Treat them like a "snowflake". And yeah, I'm explicitly suggesting censorship. Don't permit negativity. Don't permit the usual cesspool OSS discussions where you have a bunch of people who contribute almost nothing bikeshedding a project to death and exercising a sort of "liberum veto" on any attempt to do new things. Just shut down those conversations — make it clear that only the people who do major heavy lifting (or really, who are discussing things "in good faith") have a voice. The whole OSS community has an unhealthy cultural fixation on "absolute free speech", and like ... I completely understand where that comes from, but it's absolutely toxic for motivation. I've observed over the years that a lot of the more aggressively successful groups at "creating things" basically just establish their own safe-spaces where people get emotionally reinforced instead of emotionally sabotaged. Occasionally their work will get exposed to the world and some toxicity from outside will leak in, but the bread-and-butter day to day experience of working on their projects has them surrounded by friendly people who believe in what they're trying to do, and encourage them to keep going. Partly because we know we won't be there forever to keep working on it. If we feel like we're part of a group that "has our back", and will support what we built, and grow it into something, it's a million times easier to keep working on something, compared to a situation in which someone actively holds our work in spite and is itching for the first chance to tear it down the moment we turn our back, or move on, or die, etc. This is all "emotional extrapolation", but these are the sort of depressive/anti-depressive thought cycles that go through your head during a mega project, and too much of the negative side can just break people.
  • tsss 5y ago
    A good start would be to stop actively discouraging this kind of development in the majority of work places.
    • strictfp 5y ago
      This. Management needs to understand the value of "tinkerers" and "massagers". They're immensely valuable, but since they don't have a direct production output, managers rarely understand the value they provide. Without them, quality degrades and you get a lot of busy work from working in inefficient systems.
      • foxfluff 5y ago
        I guess I have some of that in me. I have no shortage of things I'd do with a code base (test, fuzz, profile & improve performance, refactor, look for and fix bugs, audit for security issues, run various static & dynamic analyzers, and just generally try to improve architecture) to improve it. This kind of thing is a large part of what takes up time in my personal projects (and the very limited open source participation I've done). That's a ton of turd polishing but I think it pays in the end (and you see a lot of it going on if you follow projects with a reputation for good code, like OpenBSD or Linux). I did this at work too early on in my career too but it was quickly made clear to me that if there's no ticket that the customer has opened (or at least approved), they're not paying for it and I can't spend time on it. Tickets I opened myself were 99% ignored or only brought up when a problem I had anticipated actually manifested in product (told ya.. now this 18-month-old ticket is suddenly relevant?). I quickly learned not to give a crap about the code base (hard to give a shit if you're not really allowed to give a shit?). I guess that also slashed my motivation (and productivity with it). It's pretty frustrating to work this way. Creative freedom and autonomy are key. I'm sure there was no micromanager assigning Ingo Molnar the "make kernel builds faster (est. 80 hours of work)" ticket.
  • oneplane 5y ago
    I think some people aren't as easily "created" like that. Perhaps it is comparable to an artist; you can teach a lot of people to paint, write or sculpt, but that doesn't mean they all become artists. It's some sort of combination of affinity, drive, self-starting and creativity.
  • jbverschoor 5y ago
    They have to be inspired by some field. You can only inspire someone if you are able to transfer the purpose of something. Performance, UX, maintainability, scalability, product. Most highschool teachers are really bad at this. Also, you need to have gathered the skills to do what you wanna do. But you'll put in the effort yourself if you're inspired.
  • amineahd 5y ago
    I have and still struggling to get into Kernel dev. There seems to be a lack of documentation or a place where there are clear steps what to do in order to get active. I guess it gets easier if you work for a company when Kernel dev is a central place but otherwise it looks hard to find something interesting to do.
    • edoceo 5y ago
      Hibernate and touchpad still needs work.
      • wilsonjholmes 5y ago
        Yes, amazing work is being done, but preach it! We still need more quality of life changes on the desktop.
    • fit2rule 5y ago
  • mirekrusin 5y ago
    You need a man and a woman plus a bit of time.
  • phkahler 5y ago
    >> How do we create more people like this? Very few companies or other management structures would ever sanction this kind of work. They might recognize it as important, but it's too big and open ended to "allocate resources to". So for this to happen you need to pay people for more of a general "make stuff better for us" role. Even that's hard because you never know what you're going to get.
  • cogman10 5y ago
    Real talk, I don't think you can. Frankly everything I know about software dev is geared to discourage exactly this sort of behavior. I would not be happy to see a pull request touching the entirety of the code base for better compilation performance. Not because such a thing wouldn't pay dividends in developer performance or code readability, but because such a change is just inherently risky. You might be able to train more people with whole program understanding (Definitely a skill that is lacking, IMO). But I doubt you could train someone to be able to "touch every file in the kernel and get it merged".
• smcameron 5y ago
  Says Rusty Russell, who's pretty goddam scary himself. :)
  • RustyRussell 5y ago
    That's weird, because I don't see it that way. I look at Ingo, Dave Miller, Richard Henderson and Andrew Tridgell who are ~ my generation, all both better coders than I will ever be and really nice people to work with. Then I look at young coders like Olaoluwa Osuntokun and Bastien Teinturier who are also great to work with and who I can only keep up with because I have years of experience, and it completely keeps my massive ego in check!
  • hulitu 5y ago
    His name is Rusty "Trivial" Russel. :)
• IshKebab 5y ago
  To be fair nobody who wasn't in the inner circle would contemplate doing this amount of work when the likely outcome would be getting shut down by a Linus rant.