4 ms·

Not a "Linux" issue, but I once had to do a presentation to all developers / testers for a major product. It's quite important to set clear standards for what

Not a "Linux" issue, but I once had to do a presentation to all developers / testers for a major product. It's quite important to set clear standards for what is, and isn't, in a bug report. Otherwise, far too much time is spent chasing down information that the reporter (unintentionally) neglected to provide. It's also important to realize that whoever is fixing a bug isn't responsible for onboarding the submitter in writing a good bug report. That also wastes time.

mcguire 2y ago
Both of these ideas also cut down on the number of bug reports you have to handle.
- gwbas1c 2y ago
  I'm not sure if you're trying to be snarky or not. In our case, "the parties that be" realized they had to hold a higher bar at triage. It didn't cut down on the number of "bug reports," but instead cut down on the amount of back-and-forths on confusing bug reports.
  - bena 2y ago
    There's something I consider the three necessary components of a problem. You need: What you did, What you expected, and What you got. If you don't have those three, you have a complaint, not a problem. "Don't work" is not a problem, it's a complaint. "When I run the report with THESE PARAMETERS, I get X, but it should be Y" Sometimes, parts are implied, but they should be explicit when there's any chance of variability. Sometimes people would tell me, "this person isn't in the system". And I'd check the database, and they'd be there. So I would say "Yes, they are". Then, the back and forth to narrow down that what they meant was that this person wasn't showing up in a certain part of the application that they expected them to. Which is a different scenario all together. It's also, a little related to the "XY problem" as people like suggesting solutions rather than describing problems. Then there's also the case where the problem is the expectation itself rather than anything technical.
    - Angostura 2y ago
      If I tell you what I did and tell you what I got, you have the information you need, I think. ‘I selected spell check and the application crashed’ is sufficient in many cases - and is a bug report.
      - marcosdumay 2y ago
        Yes, those 3 exist, and are important. But let's not pretend they are present in every single bug. Insisting on non-existent information is how you alienate bug reporters. Often enough "the application doesn't work" is a complete description of the problem. Even though many people would prefer the shibboleth of "the application has a fatal crash at startup".
      - gwbas1c 2y ago
        > ‘I selected spell check and the application crashed’ is sufficient in many cases - and is a bug report. Ooooh, that really depends on context. If that came from someone who isn't part of the team, who isn't knowledgeable in the information needed to diagnose the bug, ok. If that's actually how to reproduce, ok. What really happens is that the crash depends on running in an XYZ environment with a document that's exactly 200 letters long and the 3rd word needs to be red. In this case, ‘I selected spell check and the application crashed’ is quite unprofessional. QA is responsible for figuring out what combination of inputs creates the crash, or if that's not possible, documenting that it's really ‘I select spell check, the application crashes 15% of the time, and I don't know why’ Otherwise, someone providing that as a "bug report" isn't a team player, and it's up to management to train that person. (Or fire them if they refuse to do their part.)
        quectophoton 2y ago
        Agree. Details about the surrounding environment might not be too important most of the times, but they could be the key to solving nasty bugs. Example: > the application crashes 15% of the time, and I don't know why It's because the user was using a stylus instead of a mouse, and was accidentally dragging the button when trying to click it, triggering the "drag" event instead of the "click" event, and this control wasn't prepared to deal with that. The remaining 85% of times it doesn't happen is because the click is done properly. And the developers can't reproduce it no matter what they do because they all use a mouse so it's very difficult to accidentally drag the button. You only found the issue because the user told you it only happened when they had Photoshop open (because it's the only time they use a stylus instead of a mouse), and by chance someone from the design team offered themselves to give it a try because they heard you talking about it during lunch, and they could reproduce it. > Ooooh, that really depends on context. Another example. For me, my Steam Deck tends to freeze in two circumstances, requiring me to force-shutdown: 1. When I have a certain website open in Firefox for a few hours. The key information in this case is which tabs I usually have open when this happens. 2. When I'm playing Stellaris, and the Steam application decided to start eating all my RAM in background. The key information here is that I'm playing on Steam Deck, which is a very small part of the user base. There have been other bugs specific to Steam Deck as well.
  - mcguire 2y ago
    Oh, no, it's a well-known approach to handling bug reports. Years ago, I worked with several of IBM's AIX kernel team; they talked about a well-defined, 3-level triage process where the third level was the actual developers. Unfortunately, they were still getting too many bug reports which was impacting productivity. IBM added a fourth level between the first level, call-center tech support, and the second level (I can't remember their term for this, basically people who would try to reproduce the bug). I got to experience this system a little later, working as a sysadmin for a CS department. I made a clear bug report, including code to reproduce it, showing a denial of service attack against AIX 3.2.5 (and possibly earlier). The first layer of support read me the relevant manual page and pointed out I was using "undocumented behavior". I said, "Yes, but DOS attack." The bug report was closed at the second level as user error according to the documentation. I still feel bad that I didn't forward my code to the BUGTRAQ mailing list. Tl;dr: The more barriers you put into place, the fewer problems you have to actually handle.
    - gwbas1c 2y ago
      Which is why the problem arose: Far too many people believe anecdotes like yours, without understanding the difference between a barrier and common sense. (A four-step process to triage a bug only serves to protect fiefdoms.) We ultimately had the leads (managers and most experienced team members) triage in a small group 4x a week. It kept the BS (and barriers) to a minimum, and standards high. In your case, we generally didn't "close bugs in isolation" like you encountered. That being said, we did have a few "security" bugs raised by people who didn't understand the use case or deliberate tradeoffs. These were closed with a careful explanation of the tradeoffs or misunderstanding. In your case, I would have re-submitted the bug, and/or reopened it.
nicce 2y ago
Error or crash-based telemetry share is actually very efficient way to make valid bug reports. You would be able to catch many hard bugs automatically. But people don't like it for privacy reasons.
- xeeeeeeeeeeenu 2y ago
  There's a middle ground in the form of "report error" windows that pop up when the application crashes.
  - nicce 2y ago
    It depends on how many clicks it will take and whether it needs some additional manual input. If it is just one click, then most might actually do it. But I would not hold my breath.
    - baby_souffle 2y ago
      > If it is just one click, then most might actually do it. But I would not hold my breath. *exactly*. "You must tell us what you were doing at the time of the crash before submitting". Uhh, I woke the computer up, logged back in and was greeted by this prompt.
hinkley 2y ago
I think one of the most important responsibilities of a QA dept is to translate user complaints into an actionable bug report. The systematic erasure of QA from the “modern” development process is pure arrogance and I hope we grow the fuck up before I retire. I’ll take early retirement over continuing as we are.
- beachy 2y ago
  The QA team are chilling in the break room with the DBA team.
  - hinkley 2y ago
    I don’t like what you said but I agree anyway.
  - tomrod 2y ago
    Who knew adding indexes could be so simple but also so neglected?
    - rurp 2y ago
      I wonder if the neglect of basic efficiency practices is largely a function of cheap money and unlimited cloud budgets. Now that the new tech shine is fading from the cloud and companies are grappling with higher interest and operating costs, maybe we'll see more of an effort towards efficiency. On the other hand, who am I kidding. Engineering and business leaders are far more likely to push the focus towards AI solutions and other shiny trends, regardless of how effective they are.
- 1over137 2y ago
  >The systematic erasure of QA from the “modern” development process... Is that a thing? Which industry/country are you in?
  - mikestew 2y ago
    You’ve heard that Microsoft got rid of their software test teams, right? It’s not as if GP’s experience is an isolated incident in one small niche of the software world.
tgsovlerkhgsel 2y ago
There is a bit of a self-reinforcing effect going on: If I spend the 2-3 hours required to write a good bug report, and I see no indication that any human even looked at that work, I'll be less likely to put that effort in. After a few iterations, my bug reports, if I'm still filing them, are probably going to be quite bad, which isn't going to make it more likely that they'll be picked up...
- gwbas1c 2y ago
  > and I see no indication that any human even looked at that work You'll get plenty of indication that a human looked at the ticket when you see notifications in the ticketing system that the bug was triaged and then you're the one verifying that the bug was fixed.
  - tgsovlerkhgsel 2y ago
    Yes, I would consider that indication that a human looked at it. Unfortunately, for many bugs (not kernel related) I have never seen such an indication.

mcguire 2y ago

Both of these ideas also cut down on the number of bug reports you have to handle.

gwbas1c 2y ago
I'm not sure if you're trying to be snarky or not. In our case, "the parties that be" realized they had to hold a higher bar at triage. It didn't cut down on the number of "bug reports," but instead cut down on the amount of back-and-forths on confusing bug reports.
- bena 2y ago
  There's something I consider the three necessary components of a problem. You need: What you did, What you expected, and What you got. If you don't have those three, you have a complaint, not a problem. "Don't work" is not a problem, it's a complaint. "When I run the report with THESE PARAMETERS, I get X, but it should be Y" Sometimes, parts are implied, but they should be explicit when there's any chance of variability. Sometimes people would tell me, "this person isn't in the system". And I'd check the database, and they'd be there. So I would say "Yes, they are". Then, the back and forth to narrow down that what they meant was that this person wasn't showing up in a certain part of the application that they expected them to. Which is a different scenario all together. It's also, a little related to the "XY problem" as people like suggesting solutions rather than describing problems. Then there's also the case where the problem is the expectation itself rather than anything technical.
  - Angostura 2y ago
    If I tell you what I did and tell you what I got, you have the information you need, I think. ‘I selected spell check and the application crashed’ is sufficient in many cases - and is a bug report.
    - marcosdumay 2y ago
      Yes, those 3 exist, and are important. But let's not pretend they are present in every single bug. Insisting on non-existent information is how you alienate bug reporters. Often enough "the application doesn't work" is a complete description of the problem. Even though many people would prefer the shibboleth of "the application has a fatal crash at startup".
    - gwbas1c 2y ago
      > ‘I selected spell check and the application crashed’ is sufficient in many cases - and is a bug report. Ooooh, that really depends on context. If that came from someone who isn't part of the team, who isn't knowledgeable in the information needed to diagnose the bug, ok. If that's actually how to reproduce, ok. What really happens is that the crash depends on running in an XYZ environment with a document that's exactly 200 letters long and the 3rd word needs to be red. In this case, ‘I selected spell check and the application crashed’ is quite unprofessional. QA is responsible for figuring out what combination of inputs creates the crash, or if that's not possible, documenting that it's really ‘I select spell check, the application crashes 15% of the time, and I don't know why’ Otherwise, someone providing that as a "bug report" isn't a team player, and it's up to management to train that person. (Or fire them if they refuse to do their part.)
      - quectophoton 2y ago
        Agree. Details about the surrounding environment might not be too important most of the times, but they could be the key to solving nasty bugs. Example: > the application crashes 15% of the time, and I don't know why It's because the user was using a stylus instead of a mouse, and was accidentally dragging the button when trying to click it, triggering the "drag" event instead of the "click" event, and this control wasn't prepared to deal with that. The remaining 85% of times it doesn't happen is because the click is done properly. And the developers can't reproduce it no matter what they do because they all use a mouse so it's very difficult to accidentally drag the button. You only found the issue because the user told you it only happened when they had Photoshop open (because it's the only time they use a stylus instead of a mouse), and by chance someone from the design team offered themselves to give it a try because they heard you talking about it during lunch, and they could reproduce it. > Ooooh, that really depends on context. Another example. For me, my Steam Deck tends to freeze in two circumstances, requiring me to force-shutdown: 1. When I have a certain website open in Firefox for a few hours. The key information in this case is which tabs I usually have open when this happens. 2. When I'm playing Stellaris, and the Steam application decided to start eating all my RAM in background. The key information here is that I'm playing on Steam Deck, which is a very small part of the user base. There have been other bugs specific to Steam Deck as well.
- mcguire 2y ago
  Oh, no, it's a well-known approach to handling bug reports. Years ago, I worked with several of IBM's AIX kernel team; they talked about a well-defined, 3-level triage process where the third level was the actual developers. Unfortunately, they were still getting too many bug reports which was impacting productivity. IBM added a fourth level between the first level, call-center tech support, and the second level (I can't remember their term for this, basically people who would try to reproduce the bug). I got to experience this system a little later, working as a sysadmin for a CS department. I made a clear bug report, including code to reproduce it, showing a denial of service attack against AIX 3.2.5 (and possibly earlier). The first layer of support read me the relevant manual page and pointed out I was using "undocumented behavior". I said, "Yes, but DOS attack." The bug report was closed at the second level as user error according to the documentation. I still feel bad that I didn't forward my code to the BUGTRAQ mailing list. Tl;dr: The more barriers you put into place, the fewer problems you have to actually handle.
  - gwbas1c 2y ago
    Which is why the problem arose: Far too many people believe anecdotes like yours, without understanding the difference between a barrier and common sense. (A four-step process to triage a bug only serves to protect fiefdoms.) We ultimately had the leads (managers and most experienced team members) triage in a small group 4x a week. It kept the BS (and barriers) to a minimum, and standards high. In your case, we generally didn't "close bugs in isolation" like you encountered. That being said, we did have a few "security" bugs raised by people who didn't understand the use case or deliberate tradeoffs. These were closed with a careful explanation of the tradeoffs or misunderstanding. In your case, I would have re-submitted the bug, and/or reopened it.

nicce 2y ago

Error or crash-based telemetry share is actually very efficient way to make valid bug reports. You would be able to catch many hard bugs automatically. But people don't like it for privacy reasons.

xeeeeeeeeeeenu 2y ago
There's a middle ground in the form of "report error" windows that pop up when the application crashes.
- nicce 2y ago
  It depends on how many clicks it will take and whether it needs some additional manual input. If it is just one click, then most might actually do it. But I would not hold my breath.
  - baby_souffle 2y ago
    > If it is just one click, then most might actually do it. But I would not hold my breath. *exactly*. "You must tell us what you were doing at the time of the crash before submitting". Uhh, I woke the computer up, logged back in and was greeted by this prompt.

hinkley 2y ago

I think one of the most important responsibilities of a QA dept is to translate user complaints into an actionable bug report. The systematic erasure of QA from the “modern” development process is pure arrogance and I hope we grow the fuck up before I retire. I’ll take early retirement over continuing as we are.

beachy 2y ago
The QA team are chilling in the break room with the DBA team.
- hinkley 2y ago
  I don’t like what you said but I agree anyway.
- tomrod 2y ago
  Who knew adding indexes could be so simple but also so neglected?
  - rurp 2y ago
    I wonder if the neglect of basic efficiency practices is largely a function of cheap money and unlimited cloud budgets. Now that the new tech shine is fading from the cloud and companies are grappling with higher interest and operating costs, maybe we'll see more of an effort towards efficiency. On the other hand, who am I kidding. Engineering and business leaders are far more likely to push the focus towards AI solutions and other shiny trends, regardless of how effective they are.
1over137 2y ago
>The systematic erasure of QA from the “modern” development process... Is that a thing? Which industry/country are you in?
- mikestew 2y ago
  You’ve heard that Microsoft got rid of their software test teams, right? It’s not as if GP’s experience is an isolated incident in one small niche of the software world.

tgsovlerkhgsel 2y ago

There is a bit of a self-reinforcing effect going on: If I spend the 2-3 hours required to write a good bug report, and I see no indication that any human even looked at that work, I'll be less likely to put that effort in. After a few iterations, my bug reports, if I'm still filing them, are probably going to be quite bad, which isn't going to make it more likely that they'll be picked up...

gwbas1c 2y ago
> and I see no indication that any human even looked at that work You'll get plenty of indication that a human looked at the ticket when you see notifications in the ticketing system that the bug was triaged and then you're the one verifying that the bug was fixed.
- tgsovlerkhgsel 2y ago
  Yes, I would consider that indication that a human looked at it. Unfortunately, for many bugs (not kernel related) I have never seen such an indication.