7 ms·

Why Linux developers do not fix reported issues or ignore bug reports

63 points by frankjr 2y ago

LegionMammal978 2y ago
Are there any good guides on compiling the Linux kernel from source and testing it on a VM? Many resources seem to assume that you want to install it next to your existing kernel, but I'd prefer not to risk my filesystem or hardware if something goes wrong.
- exe34 2y ago
  you could mount and chroot the vm image, and install into that.
- zamadatix 2y ago
  The easiest thing is just to follow said guide in a VM. Setting up a full system VM is the simpler path since it's such a common tasks that all the VM tooling is already built around the idea you'll want to do that. If you absolutely must compile outside the VM for whatever reason you can just transfer the resulting files over. It's a bit of a dead end if your kernel issue has to do with your hardware of course. I once had to patch a bug about specific NVMe drives I was working with so there was no choice but to do that with the actual hardware.
- rwmj 2y ago
  You can direct boot a local kernel with qemu: qemu-kvm -kernel /path/to/your/bzImage ...
  - ndesaulniers 2y ago
    Don't forget the bajillion arch specific command line flags for qemu!
- ndesaulniers 2y ago
  We have wrapper scripts around qemu that we use heavily for CI and development. https://github.com/ClangBuiltLinux/boot-utils https://github.com/ClangBuiltLinux/boot-utils https://docs.kernel.org/kbuild/llvm.html https://docs.kernel.org/kbuild/llvm.html
- commandersaki 2y ago
  Wireguard testsuite comes with a pretty good userland for testing kernel facilities (obviously wireguard itself). It's been integrated in the kernel tree, I can't remember where, but it's great to use for experimentation.
gwbas1c 2y ago
Not a "Linux" issue, but I once had to do a presentation to all developers / testers for a major product. It's quite important to set clear standards for what is, and isn't, in a bug report. Otherwise, far too much time is spent chasing down information that the reporter (unintentionally) neglected to provide. It's also important to realize that whoever is fixing a bug isn't responsible for onboarding the submitter in writing a good bug report. That also wastes time.
- mcguire 2y ago
  Both of these ideas also cut down on the number of bug reports you have to handle.
  - gwbas1c 2y ago
    I'm not sure if you're trying to be snarky or not. In our case, "the parties that be" realized they had to hold a higher bar at triage. It didn't cut down on the number of "bug reports," but instead cut down on the amount of back-and-forths on confusing bug reports.
    - bena 2y ago
      There's something I consider the three necessary components of a problem. You need: What you did, What you expected, and What you got. If you don't have those three, you have a complaint, not a problem. "Don't work" is not a problem, it's a complaint. "When I run the report with THESE PARAMETERS, I get X, but it should be Y" Sometimes, parts are implied, but they should be explicit when there's any chance of variability. Sometimes people would tell me, "this person isn't in the system". And I'd check the database, and they'd be there. So I would say "Yes, they are". Then, the back and forth to narrow down that what they meant was that this person wasn't showing up in a certain part of the application that they expected them to. Which is a different scenario all together. It's also, a little related to the "XY problem" as people like suggesting solutions rather than describing problems. Then there's also the case where the problem is the expectation itself rather than anything technical.
hinkley 2y ago
> The tone threw people off. That a group that contains Linus Torvalds is policing tone is hilariously tone deaf.
- moomin 2y ago
  Everyone cares about tone. Especially those so chippy they pretend not to.
- mnau 2y ago
  The positions of two parties are different, they are not symmetric. Person who is asking someone else to spend time on his problem should be aware of his tone. Unless bug reporter send report to RedHat or some other company he has business relationship with, it matters.
- arp242 2y ago
  Linus never shouted at random people reporting bugs, or anything like that. He occasionally shouted (or still shouts?) at people breaking stuff they really shouldn't have broken, and who really should have known better. What you think of that is up to you, but the suggestion that Linus is somehow a complete asshole and rude to everyone and everything is just not true, and has never been true. Besides, "hey, can you help me?" and "I gave you responsibility over this part and you broke stuff for users you numpty" are fundamentally different positions to start with.
Animats 2y ago
The last time I filed a Linux kernel bug report, it was to report that you could plug in a Firewire cable and directly read or write memory, which is a security bug. But only the first 4GB of memory, because the bug that allowed this was from the 32-bit days. There's are registers which set the upper and lower limits of the accessible area, and they were set for a range of 0..0xffffffff. Loud complaints from someone who had a debugger which used that exploit. Wasn't fixed.
- ykonstant 2y ago
  That sounds insane; can you share details or link to a report?
  - tjoff 2y ago
    Windows and OS X had the same/similar problems. On consuner hardware at that time you probably couldn't make firewire secure.
    - talldayo 2y ago
      Yep, same problem exists over Thunderbolt connections afaik. By emulating PCIe devices you can basically coerce the host into giving you whatever you want from memory, as long as it's configured to conveniently do-so by default: https://thunderspy.io/ https://thunderspy.io/
- mushufasa 2y ago
  my understanding is there are so many vulnerabilities when someone has physical device access that the security community essentially suggests you assume any device is compromised if you lost physical access
  - KerrAvon 2y ago
    That used to be the advice. But it doesn't fly anymore when your OS is running on an easily-stolen device like a phone.
jmclnx 2y ago
And I would like to add another reason: * Fixing the bug will not add value to the various Fortune 500 Companies that control Linux.
- brcmthrowaway 2y ago
  Tell us more
  - ykonstant 2y ago
    Not sure what more you need; it is no secret that kernel code is written primarily by and for the benefit of corporate entities, and naturally they have priorities. [I am not saying this as a bad thing; in fact, the GPL licenses are designed around this "fact of life" aiming to maximize the benefit for (all) the users and minimize control of a single powerful entity. I'd say the current state of affairs is amazing, considering the historical alternatives.]
  - iceburgcrm 2y ago
    Large companies hire people to work on open source. Some use their clout to influence standards. As a developer I like that companies hire for open source. As a user the quality increases with full time employees. What we don't notice is many projects morph to fit corporate needs because they become important stakeholders. Money and power corrupt by design.
  - dmurray 2y ago
    It seems like a reasonable complaint, but a subset of "Your bug is neither a regression nor a severe issue." Since the paid developers, just like literally everyone else who contributes, do fix the most serious bugs in their areas of expertise/responsibility, but triage some reports as not the most valuable use of their time, or, if applicable, their company's resources.
inopinatus 2y ago
The them-and-us mindset on display here is faulty. If you’re reporting a Linux kernel bug, then you are now a Linux kernel developer. Doesn’t matter that your name ain’t in MAINTAINERS or whatever, you still just joined the club. Please add your stone to the cauldron.
sobkas 2y ago
If you want to get something fixed whine a lot and bother everyone who can't run away from you. Also what a git I was... https://bugzilla.kernel.org/show_bug.cgi?id=101061 https://bugzilla.kernel.org/show_bug.cgi?id=101061
deleted 2y ago
[deleted]
OsrsNeedsf2P 2y ago
I will fix your bug if I find it interesting. I will fix your bug if you pay me. I will fix your bug if I get assigned it at work, or if I know you personally. Otherwise, fix it yourself. No one owes you anything, and if someone does, they can fix it for you.
- gamepsys 2y ago
  I like it when you create bug reports. However submitting a bug report to an open source project is not issuing a work order. If you have any urgency to your bug then you should pay to have it fixed. However this article is not about open source devs being treated as free labor. This article is all about how to file a good bug report. Having a bug report being politely submitted with all the correct details and in the appropriate channel is a skill many users are lacking. We should really be promoting this sort of document.
JohnFen 2y ago
What the article says is true. It's also a great list of why most people will never report bugs, and why many of the people who do come away from the experience unhappy about doing it.
- deleted 2y ago
  [deleted]
- shmerl 2y ago
  You mean many don't want to spend time bisecting and building the kernel with proposed fixes to test if they work? If the bug really bugs them, they'll probably overcome the reluctance to do the above. Not sure if it's many, but it's usually enough in case it's not some obscure component of the kernel but something more widely used like amdgpu for example. The more obscure the use case though, the worse it gets.
  - hinkley 2y ago
    One of the phenomena the industry identified back in the Windows days was subconscious bug avoidance. Some tech savvy people have a sense of what interactions with the OS are fraught and they steer themselves around them often without even realizing they are doing so. So you have one class of users who believe the system is much more stable than it really is, and an underclass who seem to bounce off every step as they tumble down the proverbial stairwell. So the bugs that cannot be avoided get fixed and the ones that are avoided by the “elites” may escape their attention if not in fact their knowledge.
    - robolange 2y ago
      I feel this. I recently ran into a problem with a kernel update breaking Bluetooth on resume from suspend. My Logitech keyboard and mouse that I use at my desk stopped working after each resume, and I had to go to the laptop keyboard and manually stop and start Bluetooth. On each resume. Look, I'd love to contribute to the kernel, but the amount of time I have to play around with these things is now measured in minutes per day. So no, sorry, I don't have time to rebuild my kernel from mainline, and then bisect, and then cherry pick patches, and then test out proposed updates, and then help shepherd it through my distribution's patching mechanism. So I just gave up on Bluetooth on Linux and plugged in Logitech's proprietary Bolt dongle, and re-paired my devices with it, and haven't had trouble since. I'd prefer to use standard protocols, but this isn't the first time a kernel update broke my Bluetooth setup, so I think I'll stick with what just works.

63 points by frankjr 2y ago

Are there any good guides on compiling the Linux kernel from source and testing it on a VM? Many resources seem to assume that you want to install it next to your existing kernel, but I'd prefer not to risk my filesystem or hardware if something goes wrong.

exe34 2y ago
you could mount and chroot the vm image, and install into that.
zamadatix 2y ago
The easiest thing is just to follow said guide in a VM. Setting up a full system VM is the simpler path since it's such a common tasks that all the VM tooling is already built around the idea you'll want to do that. If you absolutely must compile outside the VM for whatever reason you can just transfer the resulting files over. It's a bit of a dead end if your kernel issue has to do with your hardware of course. I once had to patch a bug about specific NVMe drives I was working with so there was no choice but to do that with the actual hardware.
rwmj 2y ago
You can direct boot a local kernel with qemu: qemu-kvm -kernel /path/to/your/bzImage ...
- ndesaulniers 2y ago
  Don't forget the bajillion arch specific command line flags for qemu!
ndesaulniers 2y ago
We have wrapper scripts around qemu that we use heavily for CI and development. https://github.com/ClangBuiltLinux/boot-utils https://github.com/ClangBuiltLinux/boot-utils https://docs.kernel.org/kbuild/llvm.html https://docs.kernel.org/kbuild/llvm.html
commandersaki 2y ago
Wireguard testsuite comes with a pretty good userland for testing kernel facilities (obviously wireguard itself). It's been integrated in the kernel tree, I can't remember where, but it's great to use for experimentation.

gwbas1c 2y ago

Not a "Linux" issue, but I once had to do a presentation to all developers / testers for a major product. It's quite important to set clear standards for what is, and isn't, in a bug report. Otherwise, far too much time is spent chasing down information that the reporter (unintentionally) neglected to provide. It's also important to realize that whoever is fixing a bug isn't responsible for onboarding the submitter in writing a good bug report. That also wastes time.

mcguire 2y ago
Both of these ideas also cut down on the number of bug reports you have to handle.
- gwbas1c 2y ago
  I'm not sure if you're trying to be snarky or not. In our case, "the parties that be" realized they had to hold a higher bar at triage. It didn't cut down on the number of "bug reports," but instead cut down on the amount of back-and-forths on confusing bug reports.
  - bena 2y ago
    There's something I consider the three necessary components of a problem. You need: What you did, What you expected, and What you got. If you don't have those three, you have a complaint, not a problem. "Don't work" is not a problem, it's a complaint. "When I run the report with THESE PARAMETERS, I get X, but it should be Y" Sometimes, parts are implied, but they should be explicit when there's any chance of variability. Sometimes people would tell me, "this person isn't in the system". And I'd check the database, and they'd be there. So I would say "Yes, they are". Then, the back and forth to narrow down that what they meant was that this person wasn't showing up in a certain part of the application that they expected them to. Which is a different scenario all together. It's also, a little related to the "XY problem" as people like suggesting solutions rather than describing problems. Then there's also the case where the problem is the expectation itself rather than anything technical.

hinkley 2y ago

> The tone threw people off. That a group that contains Linus Torvalds is policing tone is hilariously tone deaf.

moomin 2y ago
Everyone cares about tone. Especially those so chippy they pretend not to.
mnau 2y ago
The positions of two parties are different, they are not symmetric. Person who is asking someone else to spend time on his problem should be aware of his tone. Unless bug reporter send report to RedHat or some other company he has business relationship with, it matters.
arp242 2y ago
Linus never shouted at random people reporting bugs, or anything like that. He occasionally shouted (or still shouts?) at people breaking stuff they really shouldn't have broken, and who really should have known better. What you think of that is up to you, but the suggestion that Linus is somehow a complete asshole and rude to everyone and everything is just not true, and has never been true. Besides, "hey, can you help me?" and "I gave you responsibility over this part and you broke stuff for users you numpty" are fundamentally different positions to start with.

Animats 2y ago

The last time I filed a Linux kernel bug report, it was to report that you could plug in a Firewire cable and directly read or write memory, which is a security bug. But only the first 4GB of memory, because the bug that allowed this was from the 32-bit days. There's are registers which set the upper and lower limits of the accessible area, and they were set for a range of 0..0xffffffff. Loud complaints from someone who had a debugger which used that exploit. Wasn't fixed.

ykonstant 2y ago
That sounds insane; can you share details or link to a report?
- tjoff 2y ago
  Windows and OS X had the same/similar problems. On consuner hardware at that time you probably couldn't make firewire secure.
  - talldayo 2y ago
    Yep, same problem exists over Thunderbolt connections afaik. By emulating PCIe devices you can basically coerce the host into giving you whatever you want from memory, as long as it's configured to conveniently do-so by default: https://thunderspy.io/ https://thunderspy.io/
mushufasa 2y ago
my understanding is there are so many vulnerabilities when someone has physical device access that the security community essentially suggests you assume any device is compromised if you lost physical access
- KerrAvon 2y ago
  That used to be the advice. But it doesn't fly anymore when your OS is running on an easily-stolen device like a phone.

jmclnx 2y ago

And I would like to add another reason: * Fixing the bug will not add value to the various Fortune 500 Companies that control Linux.

brcmthrowaway 2y ago
Tell us more
- ykonstant 2y ago
  Not sure what more you need; it is no secret that kernel code is written primarily by and for the benefit of corporate entities, and naturally they have priorities. [I am not saying this as a bad thing; in fact, the GPL licenses are designed around this "fact of life" aiming to maximize the benefit for (all) the users and minimize control of a single powerful entity. I'd say the current state of affairs is amazing, considering the historical alternatives.]
- iceburgcrm 2y ago
  Large companies hire people to work on open source. Some use their clout to influence standards. As a developer I like that companies hire for open source. As a user the quality increases with full time employees. What we don't notice is many projects morph to fit corporate needs because they become important stakeholders. Money and power corrupt by design.
- dmurray 2y ago
  It seems like a reasonable complaint, but a subset of "Your bug is neither a regression nor a severe issue." Since the paid developers, just like literally everyone else who contributes, do fix the most serious bugs in their areas of expertise/responsibility, but triage some reports as not the most valuable use of their time, or, if applicable, their company's resources.

inopinatus 2y ago

The them-and-us mindset on display here is faulty. If you’re reporting a Linux kernel bug, then you are now a Linux kernel developer. Doesn’t matter that your name ain’t in MAINTAINERS or whatever, you still just joined the club. Please add your stone to the cauldron.

sobkas 2y ago

If you want to get something fixed whine a lot and bother everyone who can't run away from you. Also what a git I was... https://bugzilla.kernel.org/show_bug.cgi?id=101061 https://bugzilla.kernel.org/show_bug.cgi?id=101061

deleted 2y ago

[deleted]

OsrsNeedsf2P 2y ago

I will fix your bug if I find it interesting. I will fix your bug if you pay me. I will fix your bug if I get assigned it at work, or if I know you personally. Otherwise, fix it yourself. No one owes you anything, and if someone does, they can fix it for you.

gamepsys 2y ago
I like it when you create bug reports. However submitting a bug report to an open source project is not issuing a work order. If you have any urgency to your bug then you should pay to have it fixed. However this article is not about open source devs being treated as free labor. This article is all about how to file a good bug report. Having a bug report being politely submitted with all the correct details and in the appropriate channel is a skill many users are lacking. We should really be promoting this sort of document.

JohnFen 2y ago

What the article says is true. It's also a great list of why most people will never report bugs, and why many of the people who do come away from the experience unhappy about doing it.

deleted 2y ago
[deleted]
shmerl 2y ago
You mean many don't want to spend time bisecting and building the kernel with proposed fixes to test if they work? If the bug really bugs them, they'll probably overcome the reluctance to do the above. Not sure if it's many, but it's usually enough in case it's not some obscure component of the kernel but something more widely used like amdgpu for example. The more obscure the use case though, the worse it gets.
- hinkley 2y ago
  One of the phenomena the industry identified back in the Windows days was subconscious bug avoidance. Some tech savvy people have a sense of what interactions with the OS are fraught and they steer themselves around them often without even realizing they are doing so. So you have one class of users who believe the system is much more stable than it really is, and an underclass who seem to bounce off every step as they tumble down the proverbial stairwell. So the bugs that cannot be avoided get fixed and the ones that are avoided by the “elites” may escape their attention if not in fact their knowledge.
  - robolange 2y ago
    I feel this. I recently ran into a problem with a kernel update breaking Bluetooth on resume from suspend. My Logitech keyboard and mouse that I use at my desk stopped working after each resume, and I had to go to the laptop keyboard and manually stop and start Bluetooth. On each resume. Look, I'd love to contribute to the kernel, but the amount of time I have to play around with these things is now measured in minutes per day. So no, sorry, I don't have time to rebuild my kernel from mainline, and then bisect, and then cherry pick patches, and then test out proposed updates, and then help shepherd it through my distribution's patching mechanism. So I just gave up on Bluetooth on Linux and plugged in Logitech's proprietary Bolt dongle, and re-paired my devices with it, and haven't had trouble since. I'd prefer to use standard protocols, but this isn't the first time a kernel update broke my Bluetooth setup, so I think I'll stick with what just works.