6 ms·

But testing by human is expensive

by alex_smart 1y ago

Anonbrit 1y ago
and horribly unreliable even when done by competent and motivated humans, let alone most IT workers
- karolinepauls 1y ago
  I've heard of good experiences using a 3rd party QA company (for frontend-heavy changes) and had pretty OK experiences doing it in-house with subject experts (though testing backend changes, so pretty much "either it works or it doesn't" in my case).
- somewhereoutth 1y ago
  A good QA person/team will (be strongly incentivized to) make it their mission to find bugs and usability issues. Likely they will be domain experts, if only in the domain of the usage of your product. Worth their weight in gold, no machine can give you that.
Retric 1y ago
Expensive isn’t unnecessary. Developer written tests can’t tell you if your UI is intuitive for novice users.
- sfn42 1y ago
  They can however tell you whether the button/form/page/whatever is working and continues working.
  - dylan604 1y ago
    Do those developer created tests include weird and random text that users will unwittingly break your form with? More than once has a form been broken because devs didn’t test for emoji or other glyphs in the text. Just because a button click calls a form with as expected doesn’t not mean the form will behave as expected with unexpected user input
    - sfn42 1y ago
      Playwright is essentially a programmable headless browser. It sends a request to a page, waits for the response to load and then for things like React rendering, secondary requests etc, then once the page is ready you can do whatever you want. You can interact with the DOM, fill in forms, scroll, click things, get a div and check its size and location, anything you want. Testing special symbols and such should be no problem at all, provided that you actually think to write that test.
      - Retric 1y ago
        Which is can be used for a range of things but doesn’t remove the need for independent testing. To use one tiny concrete example. Someone who isn’t aware of U+0022 + U+201C quotation marks will happily use U+0022 everywhere without ever considering the very slightly more aesthetically pleasing option. Independent validation is about redundancy, ideally you don’t want anything to reach that stage but people are really bad at testing for issues the’ve never considered. Usability in terms of colorblindness not just actual blindness yadda yadda good testing is really involved.
        necovek 1y ago
        There is fuzzing, FWIW. But the same also holds for people who are "independent testers" and unaware of the same issues. What I found is that software developers are bad at testing edge cases while they are in the creation mode (when they focus on happy path), but that good engineers switch to breaking mode once they try to cover things with sufficient tests. TDD also encourages breaking things first, but really, this is a mindset change that is usually skipped.
        Retric 1y ago
        Fuzzing isn’t going to find either of the issues I mentioned. It’s useful for finding the kind of issues most developers I have worked with are well aware of, though may not specifically test for.
somewhereoutth 1y ago
Perhaps cheaper than upsetting your customers when they are the ones doing the testing? No machine testing can completely (or even mostly) validate a human interface.

But testing by human is expensive

Anonbrit 1y ago

and horribly unreliable even when done by competent and motivated humans, let alone most IT workers

karolinepauls 1y ago
I've heard of good experiences using a 3rd party QA company (for frontend-heavy changes) and had pretty OK experiences doing it in-house with subject experts (though testing backend changes, so pretty much "either it works or it doesn't" in my case).
somewhereoutth 1y ago
A good QA person/team will (be strongly incentivized to) make it their mission to find bugs and usability issues. Likely they will be domain experts, if only in the domain of the usage of your product. Worth their weight in gold, no machine can give you that.

Retric 1y ago

Expensive isn’t unnecessary. Developer written tests can’t tell you if your UI is intuitive for novice users.

sfn42 1y ago
They can however tell you whether the button/form/page/whatever is working and continues working.
- dylan604 1y ago
  Do those developer created tests include weird and random text that users will unwittingly break your form with? More than once has a form been broken because devs didn’t test for emoji or other glyphs in the text. Just because a button click calls a form with as expected doesn’t not mean the form will behave as expected with unexpected user input
  - sfn42 1y ago
    Playwright is essentially a programmable headless browser. It sends a request to a page, waits for the response to load and then for things like React rendering, secondary requests etc, then once the page is ready you can do whatever you want. You can interact with the DOM, fill in forms, scroll, click things, get a div and check its size and location, anything you want. Testing special symbols and such should be no problem at all, provided that you actually think to write that test.
    - Retric 1y ago
      Which is can be used for a range of things but doesn’t remove the need for independent testing. To use one tiny concrete example. Someone who isn’t aware of U+0022 + U+201C quotation marks will happily use U+0022 everywhere without ever considering the very slightly more aesthetically pleasing option. Independent validation is about redundancy, ideally you don’t want anything to reach that stage but people are really bad at testing for issues the’ve never considered. Usability in terms of colorblindness not just actual blindness yadda yadda good testing is really involved.
      - necovek 1y ago
        There is fuzzing, FWIW. But the same also holds for people who are "independent testers" and unaware of the same issues. What I found is that software developers are bad at testing edge cases while they are in the creation mode (when they focus on happy path), but that good engineers switch to breaking mode once they try to cover things with sufficient tests. TDD also encourages breaking things first, but really, this is a mindset change that is usually skipped.
        Retric 1y ago
        Fuzzing isn’t going to find either of the issues I mentioned. It’s useful for finding the kind of issues most developers I have worked with are well aware of, though may not specifically test for.

somewhereoutth 1y ago

Perhaps cheaper than upsetting your customers when they are the ones doing the testing? No machine testing can completely (or even mostly) validate a human interface.