6 ms·

I used to be surprised when people came across these bugs. I would think: a kernel bug! They must have truly hit an edge case for something like that to skip in

I used to be surprised when people came across these bugs. I would think: a kernel bug! They must have truly hit an edge case for something like that to skip into production. But then I read Showstopper: The Race to Build Windows NT, specifically the part where that OS shipped with 65,000 known defect (and it was considered one of the most stable software releases of all time), and it became clear to me that these bugs are probably known by the team when the OS ships but just not considered high priority enough to fix. Same with the recent articles about iOS and engineers not getting time to fix P2 bugs. Long story short my entire perspective on kernel bugs (and partly software development) changed and I began to think of them more as known flaws that weren’t fixed because of schedules rather than unknown bugs that were uncovered in the wild.

mehrdadn 9y ago
Are data corruption defects treated the same way as other things? i.e., would kernels get shipped with known defects that cause data corruption under load? I could see kernels being shipped with known cases of hanging, crashes, out of memory, etc., but the same for data corruption under load would be a little hard to swallow.
- brucedawson 9y ago
  Bugs are prioritized. Data corruption would be higher than some other bugs, but it also depends on frequency
DoofusOfDeath 9y ago
This is one aspect of OS development in which I think open source wins: users are not at the mercy of the OS developers for bug tracking and/or fixes.
- chrisper 9y ago
  Are we not? If I find a bug, I sure hope they fix it soon. I do not have the time and energy to work myself into the kernel to fix the bug (unless it is a small bug that does not require a lot of knowledge about the kernel itself).
  - mseebach 9y ago
    Sure, as an individual, your direct powers are limited, but they're a lot less limited than if your kernel is closed. You can offer a cash bounty to anyone who wants to fix the bug (not just those employed by the kernel-vendor) and you can deploy their fix to your own systems independently of the kernel vendor releasing an update (or, accepting the fix at all) because you can build the kernel from source yourself (or, again, you can pay someone to do this for you). If you can't afford a big enough bounty to get anyone to bite, you can team up with others affected by the bug to pool your resources, and you can even use the kernel bug tracker to coordinate this effort.
    - chrisper 9y ago
      Good points. Thanks!
    - hiccuphippo 9y ago
      At the very least you can write a bug report. I tried writing one for Microsoft Office, not even found where.
      - oblio 9y ago
        I think they consider that a feature, with their 1 billion (not-really technical) users :)
      - Someone1234 9y ago
        First off, true, reporting bugs to Microsoft is hell. But on the other hand reporting feature suggestions to Microsoft is much more pleasant than anything open source has e.g. https://office.uservoice.com/forums/285186-general https://office.uservoice.com/forums/285186-general In Open Source world, because by the nature of it you're non-paying, asking for features either gets back "submit your pull request" or "go away." Which makes sense, those who contribute the most time get the most say, and that would be fine if those that contribute the most time had the same goals and taste as those that didn't (see the failure of Linux Desktop as an example).
- acdha 9y ago
  I think in reality it's more nuanced than that: the number of people who use computers who can fix a kernel bug is relatively small, and the process of getting a bug accepted, patched, and shipped is not trivial. I would not want to bet on a bug which Linus disagrees with getting patched faster than, say, a report on some company's enterprise support agreement. (disclaimer: I've been running Linux and other open-source OSes for over two decades. I like OSS but it's not perfect and staffing for projects is an under-solved problems.)
  - jjuhl 9y ago
    There are many small bugs in the kernel that are not hard to fix but are not priorities of many kernel developers, but the maintainers are usually happy to take your fixes if you create them. Proof; I've submitted a number of small fixes over the years that have been accepted: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?h=v4.16-rc3&qt=author&q=Jesper+Juhl https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
ashelmire 9y ago
Think about your code. Now imagine a codebase 1000 times as large written by thousands of engineers with deadlines. There are bugs or potential vulnerabilities in every codebase, but you’ll never encounter the vast majority of them.
pjc50 9y ago
I found my own one: Windows "structured exception handling" relies on code in the kernel that unwinds call stacks. This makes a whole bunch of assumptions about the layout of the code as well. On CE/ARM, we found that Visual Studio was sometimes generating code that used "bl [r3]" constructs which could not be unwound. The workaround was to use a different version of cl.exe that came with the CE dev kit and not the one that came with Visual Studio. Oh, and if you have more than about 512 bytes of local variables in a stack frame and try to use CE remote debugging, that stops working as well.
- slededit 9y ago
  Its the same on X64, although I think its better documented/understood on that platform.
  - trentnelson 9y ago
    Yeah that's one of the things Cutler played an important role in -- formalizing x64 prologue and epilogue constructs. The epilogue stuff is interesting, there's a small set of instructions that are allowed because the unwind code literally does opcode analysis to figure out if the current function is in its epilogue or not.
    - philiplu 9y ago
      Heh - that was partly me. The kernel team (i.e. mostly DaveC) mostly prescribed which instructions were allowed in prologues/epilogues. I was dev lead of VC's CRT team at the time (2000-ish), and got assigned to work on that and come up with an x64 version of the prologue/epilogue descriptors needed to unwind. I looked at what was done for Itanium, and simplified that waaaay down to what was needed for x64. I remember designing the ability to split prologues into multiple parts, to do something called shrink-wrapping, where you might not want to push all your saved registers all the time - there might be a subset of a function's code where a register needs to be pushed/popped. DaveC needed convincing that was a good idea.
ksk 9y ago
Yes, but do you know of any large software project that only ships when they hit zero bugs? Actually, this would probably extend to all products. Creating a product with no known defects is not possible AFAICT.
- wrs 9y ago
  If 250kLOC is “large”, then the Space Shuttle’s onboard software is one (the only?) example. https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff
deleted 9y ago
[deleted]
WalterGR 9y ago
that OS shipped with 65,000 known defect Issue tracking at Microsoft (and presumably all large tech companies) includes things like: * localization errors * typos in user-visible text * suggestions for rewording user-visible text * issues related to icons and other graphics * issues related to sounds that ship with the product * feature requests * feature enhancements * tracking tickets for work to be done or work in progress * tickets for unverified, incorrect, and unreproducable bugs So a. 65,000 tickets, not 65,000 defects. b. "65,000 tickets" is meaningless without knowing more about them.
zombieprocesses 9y ago
This is the case with all major software releases. The bug tracker is filled with bugs for any major release. It's just a matter of team lead, project manager and higher ups deciding what the priorities are. Not every feature makes it and not every bug gets fixed. The key is to hit the milestones and get the product working as much as possible. Enterprise Software Motto: "Sometimes good enough is good enough". Amen.

mehrdadn 9y ago

Are data corruption defects treated the same way as other things? i.e., would kernels get shipped with known defects that cause data corruption under load? I could see kernels being shipped with known cases of hanging, crashes, out of memory, etc., but the same for data corruption under load would be a little hard to swallow.

brucedawson 9y ago
Bugs are prioritized. Data corruption would be higher than some other bugs, but it also depends on frequency

DoofusOfDeath 9y ago

This is one aspect of OS development in which I think open source wins: users are not at the mercy of the OS developers for bug tracking and/or fixes.

chrisper 9y ago
Are we not? If I find a bug, I sure hope they fix it soon. I do not have the time and energy to work myself into the kernel to fix the bug (unless it is a small bug that does not require a lot of knowledge about the kernel itself).
- mseebach 9y ago
  Sure, as an individual, your direct powers are limited, but they're a lot less limited than if your kernel is closed. You can offer a cash bounty to anyone who wants to fix the bug (not just those employed by the kernel-vendor) and you can deploy their fix to your own systems independently of the kernel vendor releasing an update (or, accepting the fix at all) because you can build the kernel from source yourself (or, again, you can pay someone to do this for you). If you can't afford a big enough bounty to get anyone to bite, you can team up with others affected by the bug to pool your resources, and you can even use the kernel bug tracker to coordinate this effort.
  - chrisper 9y ago
    Good points. Thanks!
  - hiccuphippo 9y ago
    At the very least you can write a bug report. I tried writing one for Microsoft Office, not even found where.
    - oblio 9y ago
      I think they consider that a feature, with their 1 billion (not-really technical) users :)
    - Someone1234 9y ago
      First off, true, reporting bugs to Microsoft is hell. But on the other hand reporting feature suggestions to Microsoft is much more pleasant than anything open source has e.g. https://office.uservoice.com/forums/285186-general https://office.uservoice.com/forums/285186-general In Open Source world, because by the nature of it you're non-paying, asking for features either gets back "submit your pull request" or "go away." Which makes sense, those who contribute the most time get the most say, and that would be fine if those that contribute the most time had the same goals and taste as those that didn't (see the failure of Linux Desktop as an example).
acdha 9y ago
I think in reality it's more nuanced than that: the number of people who use computers who can fix a kernel bug is relatively small, and the process of getting a bug accepted, patched, and shipped is not trivial. I would not want to bet on a bug which Linus disagrees with getting patched faster than, say, a report on some company's enterprise support agreement. (disclaimer: I've been running Linux and other open-source OSes for over two decades. I like OSS but it's not perfect and staffing for projects is an under-solved problems.)
- jjuhl 9y ago
  There are many small bugs in the kernel that are not hard to fix but are not priorities of many kernel developers, but the maintainers are usually happy to take your fixes if you create them. Proof; I've submitted a number of small fixes over the years that have been accepted: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?h=v4.16-rc3&qt=author&q=Jesper+Juhl https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

ashelmire 9y ago

Think about your code. Now imagine a codebase 1000 times as large written by thousands of engineers with deadlines. There are bugs or potential vulnerabilities in every codebase, but you’ll never encounter the vast majority of them.

pjc50 9y ago

I found my own one: Windows "structured exception handling" relies on code in the kernel that unwinds call stacks. This makes a whole bunch of assumptions about the layout of the code as well. On CE/ARM, we found that Visual Studio was sometimes generating code that used "bl [r3]" constructs which could not be unwound. The workaround was to use a different version of cl.exe that came with the CE dev kit and not the one that came with Visual Studio. Oh, and if you have more than about 512 bytes of local variables in a stack frame and try to use CE remote debugging, that stops working as well.

slededit 9y ago
Its the same on X64, although I think its better documented/understood on that platform.
- trentnelson 9y ago
  Yeah that's one of the things Cutler played an important role in -- formalizing x64 prologue and epilogue constructs. The epilogue stuff is interesting, there's a small set of instructions that are allowed because the unwind code literally does opcode analysis to figure out if the current function is in its epilogue or not.
  - philiplu 9y ago
    Heh - that was partly me. The kernel team (i.e. mostly DaveC) mostly prescribed which instructions were allowed in prologues/epilogues. I was dev lead of VC's CRT team at the time (2000-ish), and got assigned to work on that and come up with an x64 version of the prologue/epilogue descriptors needed to unwind. I looked at what was done for Itanium, and simplified that waaaay down to what was needed for x64. I remember designing the ability to split prologues into multiple parts, to do something called shrink-wrapping, where you might not want to push all your saved registers all the time - there might be a subset of a function's code where a register needs to be pushed/popped. DaveC needed convincing that was a good idea.

ksk 9y ago

Yes, but do you know of any large software project that only ships when they hit zero bugs? Actually, this would probably extend to all products. Creating a product with no known defects is not possible AFAICT.

wrs 9y ago
If 250kLOC is “large”, then the Space Shuttle’s onboard software is one (the only?) example. https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff

deleted 9y ago

[deleted]

WalterGR 9y ago

that OS shipped with 65,000 known defect Issue tracking at Microsoft (and presumably all large tech companies) includes things like: * localization errors * typos in user-visible text * suggestions for rewording user-visible text * issues related to icons and other graphics * issues related to sounds that ship with the product * feature requests * feature enhancements * tracking tickets for work to be done or work in progress * tickets for unverified, incorrect, and unreproducable bugs So a. 65,000 tickets, not 65,000 defects. b. "65,000 tickets" is meaningless without knowing more about them.

zombieprocesses 9y ago

This is the case with all major software releases. The bug tracker is filled with bugs for any major release. It's just a matter of team lead, project manager and higher ups deciding what the priorities are. Not every feature makes it and not every bug gets fixed. The key is to hit the milestones and get the product working as much as possible. Enterprise Software Motto: "Sometimes good enough is good enough". Amen.