7 ms·

At the start of my career (late 70s), I worked at IBM (Hursley Park) in Product Assurance (their version of QA). We wrote code and built hardware to test the pr

by pjsg 3y ago

At the start of my career (late 70s), I worked at IBM (Hursley Park) in Product Assurance (their version of QA). We wrote code and built hardware to test the product that was our area of responsibility (it was a word processing system). We wrote test cases that our code would drive against the system under test. Any issues we would describe in general terms to the development team -- we didn't want them to figure out our testcases -- we wanted them to fix the bugs. Of course, this meant that we would find (say) three bugs in linewrapping of hyphenated words and the use of backspace to delete characters, and then the development team would fix four bugs in that area but only two of the actual bugs that we had found. This meant that you could use fancy statistics to estimate the actual number of bugs left. When I've worked for organizations without QA teams, I introduce the concept of "sniff tests". This is a short (typically 1 hour) test session where anybody in the company / department is encouraged to come and bash on the new feature. The feature is supposed to be complete, but it always turns out that the edge cases just don't work. I've been in these test session where we have generated 100 bug tickets in an hour (many are duplicates). I like putting "" into every field and pressing submit. I like trying to just use the keyboard to navigate the UI. I run my system with larger fonts by default. I sometime run my browser at 110% zoom. It used to be surprising how often these simple tests would lead to problems. I'm not surprised any more!

jxramos 3y ago
that's very interesting to hide the source of the automated tests from the developers as a strategy. I can see that shifting the focus to not just disabling the test or catering to the test etc. I'll have to think about this one, there's some rich thoughts to meditate on with this one.
- ansible 3y ago
  It is an interesting approach I hadn't heard of before. For complex systems though, often reproducing the bug reliably is a large part of the problem. So giving the developers the maximum information is necessary. Any time a "fix" is implemented, someone needs to be asking the right questions. Can this type of problem occur in other features / programs? What truly is the root cause, and how has that been addressed?
  - semireg 3y ago
    Wow, less IS more. Hear me out. How do we measure “more” information? Is it specificity or density? Because here, assuming they can reproduce, keeping the information fuzzy can make the problem space feel larger. This forces a larger mental-map.
pavel_lishin 3y ago
> When I've worked for organizations without QA teams, I introduce the concept of "sniff tests". This is a short (typically 1 hour) test session where anybody in the company / department is encouraged to come and bash on the new feature. We call those bug-bashes where we work, and they're also typically very productive in terms of defects discovered! It's especially useful since during development of small features, it's usually just us programmers testing stuff out, which may not actually reflect how the end users will use our software.
- steveBK123 3y ago
  A good QA person is basically a personification of all the edge cases of your actual production users. Our good QA person knew how human users used our app better than the dev or product team. It was generally a competition between QA & L2 support as to who actually understood the app best. The problem with devs testing their own & other devs code is that we test what we expect to work in the way we expect the user to use it. This completely misses all sorts of implementation error and edge cases. Of course the dev tests the happy path they coded.. that's what they thought users would do, and what they thought users wanted! Doesn't mean devs were right, and frequently they are not..
  - justinator 3y ago
    This dude gets it.
  - fatnoah 3y ago
    > It was generally a competition between QA & L2 support as to who actually understood the app best. So true!
    - steveBK123 3y ago
      And to clarify specifically because those who haven't experienced don't understand it... The uses of your app as intended by the authoring developers never matches the uses of your app out in the wild in the hands of human users. Over time, power users develop workflows that may go unknown by dev/product/management and are only well understood by QA / L2 support. The older the app, the more the divergence
wrs 3y ago
At Microsoft back in the day, we called those “bug bashes”, and my startup inherited the idea. We encouraged the whole company to take an afternoon off to participate, and gave out awards for highest impact bug, most interesting bug, etc.
- hornban 3y ago
  This is a bit of an aside, but I have a question that I'd like to ask the wider community here. How can you do a proper bug-bash when also dealing with Scrum metrics that result in a race for new features without any regard for quality? I've tried to do this with my teams several times, but ultimately we're always coming down to the end of the sprint with too much to do to implement features, and so anybody that "takes time off" to do bug bashing looks bad because ultimately they complete fewer story points than others that don't do it? Is the secret that it only works if the entire company does it, like you suggest? And yes, I completely realize that Scrum is terrible. I'm just trying to work within a system.
  - debatem1 3y ago
    The team with the lowest bug bash participation this week is the victim err host of next week's bug bash.
  - m4rtink 3y ago
    Seems like another data point stating sprints don't make sense in real world projects ?
  - bigbillheck 3y ago
    Only assign points based on a (n-1)-day sprint instead of a n-day one.
  - Shaanie 3y ago
    That's not a problem with Scrum, it's a problem with your team. If you're doing a bug bash every sprint, then your velocity is already including the time spent on bug bashes. If it's not in every sprint, you can reduce the forecast for sprints where you do them to account for it (similar to what you do when someone is off etc). If you're competing within the team to complete as many story points as possible that's pretty weird. Is someone using story points as a metric of anything other than forecasting?
jrockway 3y ago
I've always been impressed by hardware QA test teams I've worked with. On Google Fiber, they had an elaborate lab with every possible piece of consumer electronics equipment in there, and would evaluate every release against a (controlled) unfriendly RF environment. ("In version 1.2.3.4, the download from this MacBook Pro while the microwave was running was 123.4Mbps, but in version 1.2.4.5, it's 96.8Mbps." We actually had a lot of complexity beyond this that they tested, like bandsteering, roaming, etc.) I was always extremely impressed because they came up with test cases I wouldn't have thought of, and the feedback to the development team was always valuable to act on. If they're finding this issue, we get pages of charts and graphs and an invite to the lab. If a customer finds this issue, it just eats away at our customer satisfaction while we guess what could possibly have changed. Best to find the issue in QA or development. As for software engineers handling QA, I'm very much in favor of development teams doing as much as possible. I often see tests bolted on to the very end of projects, which isn't going to lead to good tests. I think that software engineers are missing good training on what to be suspicious of, and what best practices are. There are tons of books written on things like "how to write baby's first test", but honestly, as an industry, we're past that. We need resources on what you should look out for while reviewing designs, what you should look out for while reviewing code, what should trigger alarm bells in your head while you're writing code. I'm always surprised how I'll write some code that's weird, say to myself "this is weird", and then immediately write a test to watch it change from failing to passing. Like times when you're iterating over something where normally the exit condition is "i < max", but this one time, it's different, it actually has to be "i <= max". I get paranoid and write a lot of tests to check my work. Building that paranoia is key. > I like putting "" into every field and pressing submit. Going deeper into the training aspect, something I find very useful are fuzz tests. I have written a bunch of them and they have always found a few easy-to-fix but very-annoying-to-users bugs. I would never make a policy like "every PR must include a fuzz test", but I think it would be valuable to tell new hires how to write them, and why they might help find bugs. No need to have a human come up with weird inputs when your idle CI supercomputer can do it every night! (Of course, building that infrastructure is a pain. I run them on my workstation when I remember and it interests me. Great system.) At the end of the day, I'm somewhat disappointed in the standards that people set for software. To me, if I make something for you and it blows up in your hands... I feel really shitty. So I try to avoid that in the software world by trying to break things as I make them, and ensure that if you're going to spend time using something, you don't have a bad experience. I think it's rare, and it shouldn't be, it should be something the organization values from the top to the bottom. I suppose the market doesn't incentive quality as much as it should, and as a result, organizations don't value it as much as they should. But wouldn't it be nice to be the one software company that just makes good stuff that always works and doesn't require you to have 2 week calls with the support team? I'd buy it. And I like making it. But I'm just a weirdo, I guess.
JonChesterfield 3y ago
> This meant that you could use fancy statistics to estimate the actual number of bugs left. That's very clever. Precise test case in QA plus vague description given to dev. Haven't seen it before, thank you for sharing that insight.
- bluGill 3y ago
  If there is a precise testcase, automate it. There real value of manual tests is whene they explore to find variations you didn't think of. Your manual tests should be explore x
  - JonChesterfield 3y ago
    Yes, automated tests are better than manual. The subtle point I'd missed is to consider not giving the dev team your automated test to improve the odds they fix the general pattern as opposed to your point instance. The statistical side to it is really interesting too, still thinking about that.
- dredmorbius 3y ago
  Generally, "The German Tank Problem": <https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem> There are similar methods used in estimating wildlife populations, usually based on catch-release (with banding or tagging of birds or terrestrial wildlife) or repeat-observation (as with whales, whose fluke patterns are distinctive).
krisoft 3y ago
> I've been in these test session where we have generated 100 bug tickets in an hour. Is that like… usefull to anyone? Especially if they are duplicates. It feels to me that 10 different bugs is enough to demonstrate that the feature is really bad, after that you are just kinda bouncing the rubble?
- NegativeK 3y ago
  It's "free" QA. And presumably you'll do it again later until it's better.
- notpachet 3y ago
  As noted in the post, one characteristic of a healthy QA environment at your work is how effectively you triage bugs. That includes detecting duplicates. One big QA smell for me is opening a team's bug backlog and realizing that there are shitload of dupes in there, because it means that no one really looked at them in detail.
kristjansson 3y ago
At $website, we used to call that “swarm”. All features had to go through swarm before being released, and all product managers were made to participate in swarm. Its demise was widely celebrated.
lulznews 3y ago
>Any issues we would describe in general terms to the development team -- we didn't want them to figure out our testcases I’m sorry but this is just lol. Did the devs play back by creating bugs and seeing if your team could find them?

by pjsg 3y ago

jxramos 3y ago

that's very interesting to hide the source of the automated tests from the developers as a strategy. I can see that shifting the focus to not just disabling the test or catering to the test etc. I'll have to think about this one, there's some rich thoughts to meditate on with this one.

ansible 3y ago
It is an interesting approach I hadn't heard of before. For complex systems though, often reproducing the bug reliably is a large part of the problem. So giving the developers the maximum information is necessary. Any time a "fix" is implemented, someone needs to be asking the right questions. Can this type of problem occur in other features / programs? What truly is the root cause, and how has that been addressed?
- semireg 3y ago
  Wow, less IS more. Hear me out. How do we measure “more” information? Is it specificity or density? Because here, assuming they can reproduce, keeping the information fuzzy can make the problem space feel larger. This forces a larger mental-map.

pavel_lishin 3y ago

> When I've worked for organizations without QA teams, I introduce the concept of "sniff tests". This is a short (typically 1 hour) test session where anybody in the company / department is encouraged to come and bash on the new feature. We call those bug-bashes where we work, and they're also typically very productive in terms of defects discovered! It's especially useful since during development of small features, it's usually just us programmers testing stuff out, which may not actually reflect how the end users will use our software.

steveBK123 3y ago
A good QA person is basically a personification of all the edge cases of your actual production users. Our good QA person knew how human users used our app better than the dev or product team. It was generally a competition between QA & L2 support as to who actually understood the app best. The problem with devs testing their own & other devs code is that we test what we expect to work in the way we expect the user to use it. This completely misses all sorts of implementation error and edge cases. Of course the dev tests the happy path they coded.. that's what they thought users would do, and what they thought users wanted! Doesn't mean devs were right, and frequently they are not..
- justinator 3y ago
  This dude gets it.
- fatnoah 3y ago
  > It was generally a competition between QA & L2 support as to who actually understood the app best. So true!
  - steveBK123 3y ago
    And to clarify specifically because those who haven't experienced don't understand it... The uses of your app as intended by the authoring developers never matches the uses of your app out in the wild in the hands of human users. Over time, power users develop workflows that may go unknown by dev/product/management and are only well understood by QA / L2 support. The older the app, the more the divergence

wrs 3y ago

At Microsoft back in the day, we called those “bug bashes”, and my startup inherited the idea. We encouraged the whole company to take an afternoon off to participate, and gave out awards for highest impact bug, most interesting bug, etc.

hornban 3y ago
This is a bit of an aside, but I have a question that I'd like to ask the wider community here. How can you do a proper bug-bash when also dealing with Scrum metrics that result in a race for new features without any regard for quality? I've tried to do this with my teams several times, but ultimately we're always coming down to the end of the sprint with too much to do to implement features, and so anybody that "takes time off" to do bug bashing looks bad because ultimately they complete fewer story points than others that don't do it? Is the secret that it only works if the entire company does it, like you suggest? And yes, I completely realize that Scrum is terrible. I'm just trying to work within a system.
- debatem1 3y ago
  The team with the lowest bug bash participation this week is the victim err host of next week's bug bash.
- m4rtink 3y ago
  Seems like another data point stating sprints don't make sense in real world projects ?
- bigbillheck 3y ago
  Only assign points based on a (n-1)-day sprint instead of a n-day one.
- Shaanie 3y ago
  That's not a problem with Scrum, it's a problem with your team. If you're doing a bug bash every sprint, then your velocity is already including the time spent on bug bashes. If it's not in every sprint, you can reduce the forecast for sprints where you do them to account for it (similar to what you do when someone is off etc). If you're competing within the team to complete as many story points as possible that's pretty weird. Is someone using story points as a metric of anything other than forecasting?

jrockway 3y ago

I've always been impressed by hardware QA test teams I've worked with. On Google Fiber, they had an elaborate lab with every possible piece of consumer electronics equipment in there, and would evaluate every release against a (controlled) unfriendly RF environment. ("In version 1.2.3.4, the download from this MacBook Pro while the microwave was running was 123.4Mbps, but in version 1.2.4.5, it's 96.8Mbps." We actually had a lot of complexity beyond this that they tested, like bandsteering, roaming, etc.) I was always extremely impressed because they came up with test cases I wouldn't have thought of, and the feedback to the development team was always valuable to act on. If they're finding this issue, we get pages of charts and graphs and an invite to the lab. If a customer finds this issue, it just eats away at our customer satisfaction while we guess what could possibly have changed. Best to find the issue in QA or development. As for software engineers handling QA, I'm very much in favor of development teams doing as much as possible. I often see tests bolted on to the very end of projects, which isn't going to lead to good tests. I think that software engineers are missing good training on what to be suspicious of, and what best practices are. There are tons of books written on things like "how to write baby's first test", but honestly, as an industry, we're past that. We need resources on what you should look out for while reviewing designs, what you should look out for while reviewing code, what should trigger alarm bells in your head while you're writing code. I'm always surprised how I'll write some code that's weird, say to myself "this is weird", and then immediately write a test to watch it change from failing to passing. Like times when you're iterating over something where normally the exit condition is "i < max", but this one time, it's different, it actually has to be "i <= max". I get paranoid and write a lot of tests to check my work. Building that paranoia is key. > I like putting "" into every field and pressing submit. Going deeper into the training aspect, something I find very useful are fuzz tests. I have written a bunch of them and they have always found a few easy-to-fix but very-annoying-to-users bugs. I would never make a policy like "every PR must include a fuzz test", but I think it would be valuable to tell new hires how to write them, and why they might help find bugs. No need to have a human come up with weird inputs when your idle CI supercomputer can do it every night! (Of course, building that infrastructure is a pain. I run them on my workstation when I remember and it interests me. Great system.) At the end of the day, I'm somewhat disappointed in the standards that people set for software. To me, if I make something for you and it blows up in your hands... I feel really shitty. So I try to avoid that in the software world by trying to break things as I make them, and ensure that if you're going to spend time using something, you don't have a bad experience. I think it's rare, and it shouldn't be, it should be something the organization values from the top to the bottom. I suppose the market doesn't incentive quality as much as it should, and as a result, organizations don't value it as much as they should. But wouldn't it be nice to be the one software company that just makes good stuff that always works and doesn't require you to have 2 week calls with the support team? I'd buy it. And I like making it. But I'm just a weirdo, I guess.

JonChesterfield 3y ago

> This meant that you could use fancy statistics to estimate the actual number of bugs left. That's very clever. Precise test case in QA plus vague description given to dev. Haven't seen it before, thank you for sharing that insight.

bluGill 3y ago
If there is a precise testcase, automate it. There real value of manual tests is whene they explore to find variations you didn't think of. Your manual tests should be explore x
- JonChesterfield 3y ago
  Yes, automated tests are better than manual. The subtle point I'd missed is to consider not giving the dev team your automated test to improve the odds they fix the general pattern as opposed to your point instance. The statistical side to it is really interesting too, still thinking about that.
dredmorbius 3y ago
Generally, "The German Tank Problem": <https://en.wikipedia.org/wiki/German_tank_problem https://en.wikipedia.org/wiki/German_tank_problem> There are similar methods used in estimating wildlife populations, usually based on catch-release (with banding or tagging of birds or terrestrial wildlife) or repeat-observation (as with whales, whose fluke patterns are distinctive).

krisoft 3y ago

> I've been in these test session where we have generated 100 bug tickets in an hour. Is that like… usefull to anyone? Especially if they are duplicates. It feels to me that 10 different bugs is enough to demonstrate that the feature is really bad, after that you are just kinda bouncing the rubble?

NegativeK 3y ago
It's "free" QA. And presumably you'll do it again later until it's better.
notpachet 3y ago
As noted in the post, one characteristic of a healthy QA environment at your work is how effectively you triage bugs. That includes detecting duplicates. One big QA smell for me is opening a team's bug backlog and realizing that there are shitload of dupes in there, because it means that no one really looked at them in detail.

kristjansson 3y ago

At $website, we used to call that “swarm”. All features had to go through swarm before being released, and all product managers were made to participate in swarm. Its demise was widely celebrated.

lulznews 3y ago

>Any issues we would describe in general terms to the development team -- we didn't want them to figure out our testcases I’m sorry but this is just lol. Did the devs play back by creating bugs and seeing if your team could find them?