5 ms·

> The monolithic Linux kernel is just too big. What is it now, 20,000,000 lines? There's no hope of debugging that. It shows. The Linux kernel is about 20MSLOC

> The monolithic Linux kernel is just too big. What is it now, 20,000,000 lines? There's no hope of debugging that. It shows. The Linux kernel is about 20MSLOC. The Windows kernel is about 50MSLOC. IIRC, OS X used to be ~80KSLOC. Problems with debugging are endemic to any monolithic kernel. Neither Windows nor OS X is easier to debug technologically, but Microsoft and Apple both have many employees and lots of money invested compared to Linux. (Also, Apple solved the problem by making their OS specific to their computers, so they had the whole thing under their control.)

vezzy-fnord 11y ago
Kernel debuggers exist. Of course, we're all aware of Linus' opinion on the matter.
- serge2k 11y ago
  I'm not, does he not like kernel debuggers or something?
  - vezzy-fnord 11y ago
    "No" would be an understatement: https://lwn.net/2000/0914/a/lt-debugger.php3 https://lwn.net/2000/0914/a/lt-debugger.php3
    - 0xcde4c3db 11y ago
      He apparently softened somewhat on the issue and did eventually (2008 / 2.6.26) merge a debugger into mainline. I don't think he ever really had a good answer to Alan Cox pointing out that you can't always reason your way through hardware misbehavior.
      - carussell 11y ago
        His argument also necessarily presupposes that you're never going to use a debugger to root out latent bugs, that is, you're only using a debugger because you've just written some buggy code and and you're trying to figure out where the problem lies. Which is silly. There are bugs in the kernel right now, there were bugs when that was written (he even alludes to it, but seemingly fails to understand the implication of it), there will be bugs introduced in the future, and someone at some point might like to find those bugs. Some of the best "What is the worst bug you ever encountered?" war stories are cases where people end up narrowing things down to a bug in what they were previously treating as bedrock.
    - addicted 11y ago
      That's a really interesting post, and I'm glad I read it. I find myself agreeing with Linus quite a bit, and until now I did not realize that there is programming being done without step by step debugging. And I think Linus's claims that people would be more careful when first designing and writing code if they didn't have a debugger to help them going forward makes a lot of sense. Unfortunately I work with too many people whose approach to programming is "Read spec, code, debug why code isn't working to spec, fix that specific bug". Design, architecture, etc. are simply not part of the process. I can see how a lack of easy debugging may force some forethought into the development process. That being said, how valid would this be in a development environment (like the kernel) where a lot of the work you are doing is making changes designed and created by someone else? Linus says the solution is to make sure you were careful at the start. But what if you weren't even there at the start, and had to step in later?
      - vezzy-fnord 11y ago
        The point is his view about kernel debuggers is complete nonsense, and I'm baffled that you and so many others could ever take it seriously. Yes, people would be more careful without debuggers, the same way they'd be more careful with cooperative multitasking hanging the system when they forget to yield. That doesn't make it good.
        serge2k 11y ago
        Quite often I read posts by Linus and find myself unable to take him seriously. He's arrogant, abrasive, and some of his opinions (security, debuggers apparently) are ridiculous.
        xorblurb 11y ago
        Honestly Linus is a genius about the social side of big projects. He managed to scale Linux development very well, even writing Git in the middle. About the technical parts in their epoch context, it is more mixed: technically before 2.6 versions the kernel had far too many architectural flaw to be considered for serious workloads (even if I enjoyed using it even from the 2.0 series); and today I actually don't know of a single programmer around me that think he is right about the approach of security. On the other hand he was one of the first high profile programmer to call the modern interpretation of C undefined behaviors by modern compiler writers insane. But at least the beginning technical shortcomings could be explained because Linux kernel hackers of that time were unexperimented, and they actually several times silently learned from (some) NT designs and/or maybe big Unixes of that epoch (probably mainly Solaris) and adapted them not a too long time after trolling about why their previous simpler one was good enough (when it was clearly not). The security approach today coming from people having work that long in the industry is inexplicable. (maybe it is just to early and they will just silently convert to reasonable opinions in 2016? :p )
    - serge2k 11y ago
      > And quite frankly, I don't care. I don't think kernel development should be "easy". I do not condone single-stepping through code to find the bug. I do not think that extra visibility into the system is necessarily a good thing. That's insane.
- albinofrenchy 11y ago
  Not all debug tools are single step debuggers. There are memory verifiers, formal proof techniques and self testing frameworks that Linux uses that are very, very useful
  - xjia 11y ago
    What are those tools?
    - colin_mccabe 11y ago
      sparse -- adds annotations to kernel code which can be checked by the compiler. It is a little bit like a parallel type system which provides domain-specific knowledge like "this function takes lock A and then lock B" or "this function runs in interrupt context." See https://sparse.wiki.kernel.org/index.php/Main_Page https://sparse.wiki.kernel.org/index.php/Main_Page kmemcheck -- sort of like valgrind, but for the kernel. CONFIG_FAULT_INJECTION -- inject random faults at runtime (such as in memory allocation) to test infrequently encountered error paths. CONFIG_DEBUG_MUTEXES, CONFIG_DEBUG_SPINLOCK -- run expensive mutex validation checks at runtime. coccinelle -- a source code matching and transformation engine. You can use it in some of the same contexts as sed or awk. Unlike those tools, it is aware of the C language so it can do smarter things like add an extra final argument to all occurrences of a call to do_foo_bar_baz(). See http://coccinelle.lip6.fr/ http://coccinelle.lip6.fr/ checkpatch.pl -- Checks a patch to see if it conforms to the kernel style guide. Simple things like enforcing 80-column lines, but also more complicated things as well like variable naming, whitespace, etc. smatch, flawfinder -- static analysis tools that are similar in principle to Coverity. Like Coverity, they are unsound, but often helpful.
      - xjia 11y ago
        Thanks for the summary. However, what do we have on the formal proof side?
SEJeff 11y ago
Not so sure I really disagree. If I had to guess, I'd think there are a lot more people working on the Linux kernel in aggregate than on either the Windows kernel or the OS X kernel.
- sangnoir 11y ago
  Aren't a significant chunk of those people "working on the linux kernel" 1-commit wonders? My guess is if we rank by man-hours, Windows would top the list, possibly followed by OS X.
  - colin_mccabe 11y ago
    The desktop computer market is a pretty small part of the overall kernel developer market. Embedded systems is by far the biggest employer. There are just many more phones and other devices out there than PCs-- a trend that seems set to continue. Linux dominates embedded systems and the server market, so I would expect it to dominate the overall mindshare numbers as well. I would expect OS X to have the least number of kernel developers because there just aren't that many hardware configurations for iDevices, and everyone who is bringing up an OS X board works in Cupertino (which has good and bad aspects, of course).
    - sangnoir 11y ago
      I wasn't sure OS X would be ahead of Linux, that's why I hedged my statement. Also, while the embedded Linux ecosystem is rather large, what fraction is kernel code that gets upstreamed/mainlined? As an outsider, my guess that it's a minority (some work with is on non-kernel code, and not all through embedded kernel code changes are/will be in the mainline kernel, so I would not count those as actual contributions). I say this as a Linux desktop user/layman.
      - colin_mccabe 11y ago
        There are a large number of people who do kernel-level integration work-- configuring things, downloading drivers, setting things up, and otherwise dealing with the hardware. This kind of work isn't the most glamorous, but it definitely counts as "actual contribution" to the community. Integrators find bugs, make suggestions on mailing lists, and help develop products that make investing in Linux worthwhile for everyone involved. Most integrators that I've known like to submit patches every now and then, just for fun. I would guess that there are more people whose primary job is integration than people whose primary job is kernel development. I don't really have any concrete numbers, though. On the driver development side, companies are starting to get smarter about getting their code upstream first-- look at NVidia and Intel's recent efforts in that area, for example.
archimedespi 11y ago
Oops, I meant ~80MSLOC for OS X.
jkldotio 11y ago
>Microsoft and Apple both have many employees and lots of money invested compared to Linux There are a lot of people who are paid to work on the kernel full time from Red Hat, Google, IBM and many others. If I had to guess I'd say there are probably more than the other two, it would be interesting to find out. But if you include people where it's not 100% of their job, but still an official part of their job, I'd say it's almost certainly more for Linux (not even counting unpaid contributions). Linux is the most popular platform for servers and HPC because most of the time it's the better kernel. It's so dominant that Apple has basically left the server area and the handful of showcase supercomputers built on Apple gear have long since faded from view. Linux went from having one supercomputer in the top 500, a fraction of a percent, in 1998 to 98.8% of the top 500 currently.[0] The other six seem to be IBM machines running something else. OS X's first release was 2001 and it and Microsoft offerings are simply not present in that top section of the HPC space. Linux is also the kernel on the most popular smartphone platform so it's not all computational either, when the hardware is tightly controlled it works fine. The problem isn't the number of developers on kernel or how hard it is to debug, it's that laptops and desktops aren't offered from a single company that can tie everything together. [0]http://www.top500.org/statistics/details/osfam/1 http://www.top500.org/statistics/details/osfam/1
- stonemetal 11y ago
  > Red Hat, Google, IBM and many others And they are all working on desktop\Laptop support right? Linux is great in the data center because it has big guns behind it in the data center. Linux runs well on cell phones because Google put in the effort. As soon as someone is willing and able to put in the effort on desktop Linux, it will be as good as it is in those other areas.
  - spoiler 11y ago
    I think everyone can agree that Linux's biggest shortcomings on desktops/laptops are the graphical and audio part. So, even though I agree with "As soon as someone is willing and able to put in the effort on desktop Linux, it will be as good as it is in those other areas." I would replace "someone" with "every vendor". I feel like most vendors (NVIDIA/ATI/Wacom/whatnot) concentrate much more of their effort into supporting Windows and even OSX becasuse thay's their audience. Also, I remember reading somewhere that NVIDIA/ATI work closely together with Microsoft because od Direct X[citation needed, though]. I had the opportunity to work with Direct X (the new API) and I found it much more pleasant than working with OpenGL (even though I ended up using OGL in the end; I used Windows and DX to simplify prototyping, because doing the same thing in OGL required much more dev time, at first atleast). EDIT: Also, let's not forget how most of the majority of linux developer community neglects GUI and the overall end-user friendliness, and how the environment is in most cases quite hostile towards UX/GUI designers in general. There are, of course, exceptions, but those are few.
aplorbust 11y ago
"...making their OS specific to their computers..." As a non-Linux open source project OS user, I am continually faced with driver deficiencies as a result of hardware specs being under NDA. A recurring idea I have which I am here sharing for the first time (apologies!) is: why not just pick a single item of hardware and build an open source, free OS project around it? Why? Hopefully, more control, to the extent possible (notwithstanding Intel ME, etc.). Coreboot, support for as many peripherals a possible, etc. Most importantly, the elimination of the issue of hardware support and the notion of a list of "supported hardware". Why not? Performance, latest advances, etc. Hasn't this been done? Maybe. OpenWRT, etc.? But my understanding is that the use of Linux on this router was initially the non-public work of a company, Linksys, and the open sourcing by Cisco was neither anticipated nor intentional. How is my idea different? The project would be free, open source, but intentionally focused on a _single_ target. Big tradeoff, but maybe some interesting gains. To be clear, I like the idea of hardware that is more or less "OS agnostic", e.g., RPi and booting from SD card. But I am tired of watching volunteers struggle to keep up with the latest hardware (many thanks to the OpenBSD and FreeBSD contributors who write drivers for networking, etc.), or having to settle for binary blobs. Maybe I am just dreaming but I could forsee such a project potentially growing into a symbiotic relationship with some manufacturer if the OS developed a sufficiently large, growing user base. And these users were all purchasing a very specific item(s) of hardware, known to be supported by this OS. If you comment, please remember I am not a Linux user. And hardware support is not quite the same under BSD. As such, it is something I often have to think about and cannot just take for granted.
- JupiterMoon 11y ago
  Linux mostly works on most laptops. However, my next laptop is definitely going to at the very least be on Ubuntu's supported list and probably going to be either a Dell or system 76 with pre-installed Ubuntu.

vezzy-fnord 11y ago

Kernel debuggers exist. Of course, we're all aware of Linus' opinion on the matter.

serge2k 11y ago
I'm not, does he not like kernel debuggers or something?
- vezzy-fnord 11y ago
  "No" would be an understatement: https://lwn.net/2000/0914/a/lt-debugger.php3 https://lwn.net/2000/0914/a/lt-debugger.php3
  - 0xcde4c3db 11y ago
    He apparently softened somewhat on the issue and did eventually (2008 / 2.6.26) merge a debugger into mainline. I don't think he ever really had a good answer to Alan Cox pointing out that you can't always reason your way through hardware misbehavior.
    - carussell 11y ago
      His argument also necessarily presupposes that you're never going to use a debugger to root out latent bugs, that is, you're only using a debugger because you've just written some buggy code and and you're trying to figure out where the problem lies. Which is silly. There are bugs in the kernel right now, there were bugs when that was written (he even alludes to it, but seemingly fails to understand the implication of it), there will be bugs introduced in the future, and someone at some point might like to find those bugs. Some of the best "What is the worst bug you ever encountered?" war stories are cases where people end up narrowing things down to a bug in what they were previously treating as bedrock.
  - addicted 11y ago
    That's a really interesting post, and I'm glad I read it. I find myself agreeing with Linus quite a bit, and until now I did not realize that there is programming being done without step by step debugging. And I think Linus's claims that people would be more careful when first designing and writing code if they didn't have a debugger to help them going forward makes a lot of sense. Unfortunately I work with too many people whose approach to programming is "Read spec, code, debug why code isn't working to spec, fix that specific bug". Design, architecture, etc. are simply not part of the process. I can see how a lack of easy debugging may force some forethought into the development process. That being said, how valid would this be in a development environment (like the kernel) where a lot of the work you are doing is making changes designed and created by someone else? Linus says the solution is to make sure you were careful at the start. But what if you weren't even there at the start, and had to step in later?
    - vezzy-fnord 11y ago
      The point is his view about kernel debuggers is complete nonsense, and I'm baffled that you and so many others could ever take it seriously. Yes, people would be more careful without debuggers, the same way they'd be more careful with cooperative multitasking hanging the system when they forget to yield. That doesn't make it good.
      - serge2k 11y ago
        Quite often I read posts by Linus and find myself unable to take him seriously. He's arrogant, abrasive, and some of his opinions (security, debuggers apparently) are ridiculous.
        xorblurb 11y ago
        Honestly Linus is a genius about the social side of big projects. He managed to scale Linux development very well, even writing Git in the middle. About the technical parts in their epoch context, it is more mixed: technically before 2.6 versions the kernel had far too many architectural flaw to be considered for serious workloads (even if I enjoyed using it even from the 2.0 series); and today I actually don't know of a single programmer around me that think he is right about the approach of security. On the other hand he was one of the first high profile programmer to call the modern interpretation of C undefined behaviors by modern compiler writers insane. But at least the beginning technical shortcomings could be explained because Linux kernel hackers of that time were unexperimented, and they actually several times silently learned from (some) NT designs and/or maybe big Unixes of that epoch (probably mainly Solaris) and adapted them not a too long time after trolling about why their previous simpler one was good enough (when it was clearly not). The security approach today coming from people having work that long in the industry is inexplicable. (maybe it is just to early and they will just silently convert to reasonable opinions in 2016? :p )
  - serge2k 11y ago
    > And quite frankly, I don't care. I don't think kernel development should be "easy". I do not condone single-stepping through code to find the bug. I do not think that extra visibility into the system is necessarily a good thing. That's insane.
albinofrenchy 11y ago
Not all debug tools are single step debuggers. There are memory verifiers, formal proof techniques and self testing frameworks that Linux uses that are very, very useful
- xjia 11y ago
  What are those tools?
  - colin_mccabe 11y ago
    sparse -- adds annotations to kernel code which can be checked by the compiler. It is a little bit like a parallel type system which provides domain-specific knowledge like "this function takes lock A and then lock B" or "this function runs in interrupt context." See https://sparse.wiki.kernel.org/index.php/Main_Page https://sparse.wiki.kernel.org/index.php/Main_Page kmemcheck -- sort of like valgrind, but for the kernel. CONFIG_FAULT_INJECTION -- inject random faults at runtime (such as in memory allocation) to test infrequently encountered error paths. CONFIG_DEBUG_MUTEXES, CONFIG_DEBUG_SPINLOCK -- run expensive mutex validation checks at runtime. coccinelle -- a source code matching and transformation engine. You can use it in some of the same contexts as sed or awk. Unlike those tools, it is aware of the C language so it can do smarter things like add an extra final argument to all occurrences of a call to do_foo_bar_baz(). See http://coccinelle.lip6.fr/ http://coccinelle.lip6.fr/ checkpatch.pl -- Checks a patch to see if it conforms to the kernel style guide. Simple things like enforcing 80-column lines, but also more complicated things as well like variable naming, whitespace, etc. smatch, flawfinder -- static analysis tools that are similar in principle to Coverity. Like Coverity, they are unsound, but often helpful.
    - xjia 11y ago
      Thanks for the summary. However, what do we have on the formal proof side?

SEJeff 11y ago

Not so sure I really disagree. If I had to guess, I'd think there are a lot more people working on the Linux kernel in aggregate than on either the Windows kernel or the OS X kernel.

sangnoir 11y ago
Aren't a significant chunk of those people "working on the linux kernel" 1-commit wonders? My guess is if we rank by man-hours, Windows would top the list, possibly followed by OS X.
- colin_mccabe 11y ago
  The desktop computer market is a pretty small part of the overall kernel developer market. Embedded systems is by far the biggest employer. There are just many more phones and other devices out there than PCs-- a trend that seems set to continue. Linux dominates embedded systems and the server market, so I would expect it to dominate the overall mindshare numbers as well. I would expect OS X to have the least number of kernel developers because there just aren't that many hardware configurations for iDevices, and everyone who is bringing up an OS X board works in Cupertino (which has good and bad aspects, of course).
  - sangnoir 11y ago
    I wasn't sure OS X would be ahead of Linux, that's why I hedged my statement. Also, while the embedded Linux ecosystem is rather large, what fraction is kernel code that gets upstreamed/mainlined? As an outsider, my guess that it's a minority (some work with is on non-kernel code, and not all through embedded kernel code changes are/will be in the mainline kernel, so I would not count those as actual contributions). I say this as a Linux desktop user/layman.
    - colin_mccabe 11y ago
      There are a large number of people who do kernel-level integration work-- configuring things, downloading drivers, setting things up, and otherwise dealing with the hardware. This kind of work isn't the most glamorous, but it definitely counts as "actual contribution" to the community. Integrators find bugs, make suggestions on mailing lists, and help develop products that make investing in Linux worthwhile for everyone involved. Most integrators that I've known like to submit patches every now and then, just for fun. I would guess that there are more people whose primary job is integration than people whose primary job is kernel development. I don't really have any concrete numbers, though. On the driver development side, companies are starting to get smarter about getting their code upstream first-- look at NVidia and Intel's recent efforts in that area, for example.

archimedespi 11y ago

Oops, I meant ~80MSLOC for OS X.

jkldotio 11y ago

>Microsoft and Apple both have many employees and lots of money invested compared to Linux There are a lot of people who are paid to work on the kernel full time from Red Hat, Google, IBM and many others. If I had to guess I'd say there are probably more than the other two, it would be interesting to find out. But if you include people where it's not 100% of their job, but still an official part of their job, I'd say it's almost certainly more for Linux (not even counting unpaid contributions). Linux is the most popular platform for servers and HPC because most of the time it's the better kernel. It's so dominant that Apple has basically left the server area and the handful of showcase supercomputers built on Apple gear have long since faded from view. Linux went from having one supercomputer in the top 500, a fraction of a percent, in 1998 to 98.8% of the top 500 currently.[0] The other six seem to be IBM machines running something else. OS X's first release was 2001 and it and Microsoft offerings are simply not present in that top section of the HPC space. Linux is also the kernel on the most popular smartphone platform so it's not all computational either, when the hardware is tightly controlled it works fine. The problem isn't the number of developers on kernel or how hard it is to debug, it's that laptops and desktops aren't offered from a single company that can tie everything together. [0]http://www.top500.org/statistics/details/osfam/1 http://www.top500.org/statistics/details/osfam/1

stonemetal 11y ago
> Red Hat, Google, IBM and many others And they are all working on desktop\Laptop support right? Linux is great in the data center because it has big guns behind it in the data center. Linux runs well on cell phones because Google put in the effort. As soon as someone is willing and able to put in the effort on desktop Linux, it will be as good as it is in those other areas.
- spoiler 11y ago
  I think everyone can agree that Linux's biggest shortcomings on desktops/laptops are the graphical and audio part. So, even though I agree with "As soon as someone is willing and able to put in the effort on desktop Linux, it will be as good as it is in those other areas." I would replace "someone" with "every vendor". I feel like most vendors (NVIDIA/ATI/Wacom/whatnot) concentrate much more of their effort into supporting Windows and even OSX becasuse thay's their audience. Also, I remember reading somewhere that NVIDIA/ATI work closely together with Microsoft because od Direct X[citation needed, though]. I had the opportunity to work with Direct X (the new API) and I found it much more pleasant than working with OpenGL (even though I ended up using OGL in the end; I used Windows and DX to simplify prototyping, because doing the same thing in OGL required much more dev time, at first atleast). EDIT: Also, let's not forget how most of the majority of linux developer community neglects GUI and the overall end-user friendliness, and how the environment is in most cases quite hostile towards UX/GUI designers in general. There are, of course, exceptions, but those are few.

aplorbust 11y ago

"...making their OS specific to their computers..." As a non-Linux open source project OS user, I am continually faced with driver deficiencies as a result of hardware specs being under NDA. A recurring idea I have which I am here sharing for the first time (apologies!) is: why not just pick a single item of hardware and build an open source, free OS project around it? Why? Hopefully, more control, to the extent possible (notwithstanding Intel ME, etc.). Coreboot, support for as many peripherals a possible, etc. Most importantly, the elimination of the issue of hardware support and the notion of a list of "supported hardware". Why not? Performance, latest advances, etc. Hasn't this been done? Maybe. OpenWRT, etc.? But my understanding is that the use of Linux on this router was initially the non-public work of a company, Linksys, and the open sourcing by Cisco was neither anticipated nor intentional. How is my idea different? The project would be free, open source, but intentionally focused on a _single_ target. Big tradeoff, but maybe some interesting gains. To be clear, I like the idea of hardware that is more or less "OS agnostic", e.g., RPi and booting from SD card. But I am tired of watching volunteers struggle to keep up with the latest hardware (many thanks to the OpenBSD and FreeBSD contributors who write drivers for networking, etc.), or having to settle for binary blobs. Maybe I am just dreaming but I could forsee such a project potentially growing into a symbiotic relationship with some manufacturer if the OS developed a sufficiently large, growing user base. And these users were all purchasing a very specific item(s) of hardware, known to be supported by this OS. If you comment, please remember I am not a Linux user. And hardware support is not quite the same under BSD. As such, it is something I often have to think about and cannot just take for granted.

JupiterMoon 11y ago
Linux mostly works on most laptops. However, my next laptop is definitely going to at the very least be on Ubuntu's supported list and probably going to be either a Dell or system 76 with pre-installed Ubuntu.