4 ms·

Both of these ideas also cut down on the number of bug reports you have to handle.

gwbas1c 2y ago
I'm not sure if you're trying to be snarky or not. In our case, "the parties that be" realized they had to hold a higher bar at triage. It didn't cut down on the number of "bug reports," but instead cut down on the amount of back-and-forths on confusing bug reports.
- bena 2y ago
  There's something I consider the three necessary components of a problem. You need: What you did, What you expected, and What you got. If you don't have those three, you have a complaint, not a problem. "Don't work" is not a problem, it's a complaint. "When I run the report with THESE PARAMETERS, I get X, but it should be Y" Sometimes, parts are implied, but they should be explicit when there's any chance of variability. Sometimes people would tell me, "this person isn't in the system". And I'd check the database, and they'd be there. So I would say "Yes, they are". Then, the back and forth to narrow down that what they meant was that this person wasn't showing up in a certain part of the application that they expected them to. Which is a different scenario all together. It's also, a little related to the "XY problem" as people like suggesting solutions rather than describing problems. Then there's also the case where the problem is the expectation itself rather than anything technical.
  - Angostura 2y ago
    If I tell you what I did and tell you what I got, you have the information you need, I think. ‘I selected spell check and the application crashed’ is sufficient in many cases - and is a bug report.
    - marcosdumay 2y ago
      Yes, those 3 exist, and are important. But let's not pretend they are present in every single bug. Insisting on non-existent information is how you alienate bug reporters. Often enough "the application doesn't work" is a complete description of the problem. Even though many people would prefer the shibboleth of "the application has a fatal crash at startup".
    - gwbas1c 2y ago
      > ‘I selected spell check and the application crashed’ is sufficient in many cases - and is a bug report. Ooooh, that really depends on context. If that came from someone who isn't part of the team, who isn't knowledgeable in the information needed to diagnose the bug, ok. If that's actually how to reproduce, ok. What really happens is that the crash depends on running in an XYZ environment with a document that's exactly 200 letters long and the 3rd word needs to be red. In this case, ‘I selected spell check and the application crashed’ is quite unprofessional. QA is responsible for figuring out what combination of inputs creates the crash, or if that's not possible, documenting that it's really ‘I select spell check, the application crashes 15% of the time, and I don't know why’ Otherwise, someone providing that as a "bug report" isn't a team player, and it's up to management to train that person. (Or fire them if they refuse to do their part.)
      - quectophoton 2y ago
        Agree. Details about the surrounding environment might not be too important most of the times, but they could be the key to solving nasty bugs. Example: > the application crashes 15% of the time, and I don't know why It's because the user was using a stylus instead of a mouse, and was accidentally dragging the button when trying to click it, triggering the "drag" event instead of the "click" event, and this control wasn't prepared to deal with that. The remaining 85% of times it doesn't happen is because the click is done properly. And the developers can't reproduce it no matter what they do because they all use a mouse so it's very difficult to accidentally drag the button. You only found the issue because the user told you it only happened when they had Photoshop open (because it's the only time they use a stylus instead of a mouse), and by chance someone from the design team offered themselves to give it a try because they heard you talking about it during lunch, and they could reproduce it. > Ooooh, that really depends on context. Another example. For me, my Steam Deck tends to freeze in two circumstances, requiring me to force-shutdown: 1. When I have a certain website open in Firefox for a few hours. The key information in this case is which tabs I usually have open when this happens. 2. When I'm playing Stellaris, and the Steam application decided to start eating all my RAM in background. The key information here is that I'm playing on Steam Deck, which is a very small part of the user base. There have been other bugs specific to Steam Deck as well.
    - bena 2y ago
      In your case, “what you expected” is implied by telling what you did. And I did say some parts may be implied, but to be explicit where it could be ambiguous.
  - gwbas1c 2y ago
    > Then, the back and forth to narrow down that what they meant was that this person wasn't showing up in a certain part of the application In my case I said that developers shouldn't spend more than 90 minutes reproducing a problem. If it took longer than that, the submitter wasn't correctly communicating the problem. Everyone learned how to write a bug report after that.
- mcguire 2y ago
  Oh, no, it's a well-known approach to handling bug reports. Years ago, I worked with several of IBM's AIX kernel team; they talked about a well-defined, 3-level triage process where the third level was the actual developers. Unfortunately, they were still getting too many bug reports which was impacting productivity. IBM added a fourth level between the first level, call-center tech support, and the second level (I can't remember their term for this, basically people who would try to reproduce the bug). I got to experience this system a little later, working as a sysadmin for a CS department. I made a clear bug report, including code to reproduce it, showing a denial of service attack against AIX 3.2.5 (and possibly earlier). The first layer of support read me the relevant manual page and pointed out I was using "undocumented behavior". I said, "Yes, but DOS attack." The bug report was closed at the second level as user error according to the documentation. I still feel bad that I didn't forward my code to the BUGTRAQ mailing list. Tl;dr: The more barriers you put into place, the fewer problems you have to actually handle.
  - gwbas1c 2y ago
    Which is why the problem arose: Far too many people believe anecdotes like yours, without understanding the difference between a barrier and common sense. (A four-step process to triage a bug only serves to protect fiefdoms.) We ultimately had the leads (managers and most experienced team members) triage in a small group 4x a week. It kept the BS (and barriers) to a minimum, and standards high. In your case, we generally didn't "close bugs in isolation" like you encountered. That being said, we did have a few "security" bugs raised by people who didn't understand the use case or deliberate tradeoffs. These were closed with a careful explanation of the tradeoffs or misunderstanding. In your case, I would have re-submitted the bug, and/or reopened it.

I'm not sure if you're trying to be snarky or not. In our case, "the parties that be" realized they had to hold a higher bar at triage. It didn't cut down on the number of "bug reports," but instead cut down on the amount of back-and-forths on confusing bug reports.

bena 2y ago
There's something I consider the three necessary components of a problem. You need: What you did, What you expected, and What you got. If you don't have those three, you have a complaint, not a problem. "Don't work" is not a problem, it's a complaint. "When I run the report with THESE PARAMETERS, I get X, but it should be Y" Sometimes, parts are implied, but they should be explicit when there's any chance of variability. Sometimes people would tell me, "this person isn't in the system". And I'd check the database, and they'd be there. So I would say "Yes, they are". Then, the back and forth to narrow down that what they meant was that this person wasn't showing up in a certain part of the application that they expected them to. Which is a different scenario all together. It's also, a little related to the "XY problem" as people like suggesting solutions rather than describing problems. Then there's also the case where the problem is the expectation itself rather than anything technical.
- Angostura 2y ago
  If I tell you what I did and tell you what I got, you have the information you need, I think. ‘I selected spell check and the application crashed’ is sufficient in many cases - and is a bug report.
  - marcosdumay 2y ago
    Yes, those 3 exist, and are important. But let's not pretend they are present in every single bug. Insisting on non-existent information is how you alienate bug reporters. Often enough "the application doesn't work" is a complete description of the problem. Even though many people would prefer the shibboleth of "the application has a fatal crash at startup".
  - gwbas1c 2y ago
    > ‘I selected spell check and the application crashed’ is sufficient in many cases - and is a bug report. Ooooh, that really depends on context. If that came from someone who isn't part of the team, who isn't knowledgeable in the information needed to diagnose the bug, ok. If that's actually how to reproduce, ok. What really happens is that the crash depends on running in an XYZ environment with a document that's exactly 200 letters long and the 3rd word needs to be red. In this case, ‘I selected spell check and the application crashed’ is quite unprofessional. QA is responsible for figuring out what combination of inputs creates the crash, or if that's not possible, documenting that it's really ‘I select spell check, the application crashes 15% of the time, and I don't know why’ Otherwise, someone providing that as a "bug report" isn't a team player, and it's up to management to train that person. (Or fire them if they refuse to do their part.)
    - quectophoton 2y ago
      Agree. Details about the surrounding environment might not be too important most of the times, but they could be the key to solving nasty bugs. Example: > the application crashes 15% of the time, and I don't know why It's because the user was using a stylus instead of a mouse, and was accidentally dragging the button when trying to click it, triggering the "drag" event instead of the "click" event, and this control wasn't prepared to deal with that. The remaining 85% of times it doesn't happen is because the click is done properly. And the developers can't reproduce it no matter what they do because they all use a mouse so it's very difficult to accidentally drag the button. You only found the issue because the user told you it only happened when they had Photoshop open (because it's the only time they use a stylus instead of a mouse), and by chance someone from the design team offered themselves to give it a try because they heard you talking about it during lunch, and they could reproduce it. > Ooooh, that really depends on context. Another example. For me, my Steam Deck tends to freeze in two circumstances, requiring me to force-shutdown: 1. When I have a certain website open in Firefox for a few hours. The key information in this case is which tabs I usually have open when this happens. 2. When I'm playing Stellaris, and the Steam application decided to start eating all my RAM in background. The key information here is that I'm playing on Steam Deck, which is a very small part of the user base. There have been other bugs specific to Steam Deck as well.
  - bena 2y ago
    In your case, “what you expected” is implied by telling what you did. And I did say some parts may be implied, but to be explicit where it could be ambiguous.
- gwbas1c 2y ago
  > Then, the back and forth to narrow down that what they meant was that this person wasn't showing up in a certain part of the application In my case I said that developers shouldn't spend more than 90 minutes reproducing a problem. If it took longer than that, the submitter wasn't correctly communicating the problem. Everyone learned how to write a bug report after that.
mcguire 2y ago
Oh, no, it's a well-known approach to handling bug reports. Years ago, I worked with several of IBM's AIX kernel team; they talked about a well-defined, 3-level triage process where the third level was the actual developers. Unfortunately, they were still getting too many bug reports which was impacting productivity. IBM added a fourth level between the first level, call-center tech support, and the second level (I can't remember their term for this, basically people who would try to reproduce the bug). I got to experience this system a little later, working as a sysadmin for a CS department. I made a clear bug report, including code to reproduce it, showing a denial of service attack against AIX 3.2.5 (and possibly earlier). The first layer of support read me the relevant manual page and pointed out I was using "undocumented behavior". I said, "Yes, but DOS attack." The bug report was closed at the second level as user error according to the documentation. I still feel bad that I didn't forward my code to the BUGTRAQ mailing list. Tl;dr: The more barriers you put into place, the fewer problems you have to actually handle.
- gwbas1c 2y ago
  Which is why the problem arose: Far too many people believe anecdotes like yours, without understanding the difference between a barrier and common sense. (A four-step process to triage a bug only serves to protect fiefdoms.) We ultimately had the leads (managers and most experienced team members) triage in a small group 4x a week. It kept the BS (and barriers) to a minimum, and standards high. In your case, we generally didn't "close bugs in isolation" like you encountered. That being said, we did have a few "security" bugs raised by people who didn't understand the use case or deliberate tradeoffs. These were closed with a careful explanation of the tradeoffs or misunderstanding. In your case, I would have re-submitted the bug, and/or reopened it.