4 ms·

My feeling on testing: - If it is used by a machine, then it can be tested by a machine. - If it is used by a human, then it must be tested by a human.

alex_smart 1y ago
But testing by human is expensive
- Anonbrit 1y ago
  and horribly unreliable even when done by competent and motivated humans, let alone most IT workers
  - karolinepauls 1y ago
    I've heard of good experiences using a 3rd party QA company (for frontend-heavy changes) and had pretty OK experiences doing it in-house with subject experts (though testing backend changes, so pretty much "either it works or it doesn't" in my case).
  - somewhereoutth 1y ago
    A good QA person/team will (be strongly incentivized to) make it their mission to find bugs and usability issues. Likely they will be domain experts, if only in the domain of the usage of your product. Worth their weight in gold, no machine can give you that.
- Retric 1y ago
  Expensive isn’t unnecessary. Developer written tests can’t tell you if your UI is intuitive for novice users.
  - sfn42 1y ago
    They can however tell you whether the button/form/page/whatever is working and continues working.
    - dylan604 1y ago
      Do those developer created tests include weird and random text that users will unwittingly break your form with? More than once has a form been broken because devs didn’t test for emoji or other glyphs in the text. Just because a button click calls a form with as expected doesn’t not mean the form will behave as expected with unexpected user input
      - sfn42 1y ago
        Playwright is essentially a programmable headless browser. It sends a request to a page, waits for the response to load and then for things like React rendering, secondary requests etc, then once the page is ready you can do whatever you want. You can interact with the DOM, fill in forms, scroll, click things, get a div and check its size and location, anything you want. Testing special symbols and such should be no problem at all, provided that you actually think to write that test.
        Retric 1y ago
        Which is can be used for a range of things but doesn’t remove the need for independent testing. To use one tiny concrete example. Someone who isn’t aware of U+0022 + U+201C quotation marks will happily use U+0022 everywhere without ever considering the very slightly more aesthetically pleasing option. Independent validation is about redundancy, ideally you don’t want anything to reach that stage but people are really bad at testing for issues the’ve never considered. Usability in terms of colorblindness not just actual blindness yadda yadda good testing is really involved.
        necovek 1y ago
        There is fuzzing, FWIW. But the same also holds for people who are "independent testers" and unaware of the same issues. What I found is that software developers are bad at testing edge cases while they are in the creation mode (when they focus on happy path), but that good engineers switch to breaking mode once they try to cover things with sufficient tests. TDD also encourages breaking things first, but really, this is a mindset change that is usually skipped.
- somewhereoutth 1y ago
  Perhaps cheaper than upsetting your customers when they are the ones doing the testing? No machine testing can completely (or even mostly) validate a human interface.
sfn42 1y ago
Tools like playwright allow pretty nice web UI testing. You can make sure things are working properly and continue working as things change. Doesn't replace human testing but it does ease the human load and help catch problems and regressions before they get to the human testers.
- Ygg2 1y ago
  > Tools like playwright allow pretty nice web UI testing. Can they test for color blindness and myopia?
  - dylan604 1y ago
    “Doesn't replace human testing” It’s like you stopped reading to try to score internet points or something. The answer to your question was one more sentence from where you stopped reading
    - Ygg2 1y ago
      The point of the question was to both see if the bar has changed and to address the overall article's point "Can we test it".
  - sfn42 1y ago
    There are tools that can analyze a page for accessibility. I don't have experience with them but I wouldn't be surprised if you could include them in a CI/CD pipeline. I don't know whether they specifically control for color blindness and myopia but I wouldn't be surprised if they do. I know they check for contrast and stuff like that. You could also check things like colors etc using Playwright but I would say it's probably the wrong tool for that job. It's more about testing functionality - make sure a page has the right content and works correctly on a technical level. Without automated tools this type of thing can take a lot of time - in order to ensure quality you would basically have to click through the entire application for every release. Otherwise you might end up with some minor changes to one page breaking a different page and you'd never know until a tester checked it out or a user complained. With Playwright and similar tools you can catch regressions like this automatically.
__MatrixMan__ 1y ago
User acceptance testing is a good idea but doing it in cases where you can get away with cheaper testing is not.

alex_smart 1y ago

But testing by human is expensive

Anonbrit 1y ago
and horribly unreliable even when done by competent and motivated humans, let alone most IT workers
- karolinepauls 1y ago
  I've heard of good experiences using a 3rd party QA company (for frontend-heavy changes) and had pretty OK experiences doing it in-house with subject experts (though testing backend changes, so pretty much "either it works or it doesn't" in my case).
- somewhereoutth 1y ago
  A good QA person/team will (be strongly incentivized to) make it their mission to find bugs and usability issues. Likely they will be domain experts, if only in the domain of the usage of your product. Worth their weight in gold, no machine can give you that.
Retric 1y ago
Expensive isn’t unnecessary. Developer written tests can’t tell you if your UI is intuitive for novice users.
- sfn42 1y ago
  They can however tell you whether the button/form/page/whatever is working and continues working.
  - dylan604 1y ago
    Do those developer created tests include weird and random text that users will unwittingly break your form with? More than once has a form been broken because devs didn’t test for emoji or other glyphs in the text. Just because a button click calls a form with as expected doesn’t not mean the form will behave as expected with unexpected user input
    - sfn42 1y ago
      Playwright is essentially a programmable headless browser. It sends a request to a page, waits for the response to load and then for things like React rendering, secondary requests etc, then once the page is ready you can do whatever you want. You can interact with the DOM, fill in forms, scroll, click things, get a div and check its size and location, anything you want. Testing special symbols and such should be no problem at all, provided that you actually think to write that test.
      - Retric 1y ago
        Which is can be used for a range of things but doesn’t remove the need for independent testing. To use one tiny concrete example. Someone who isn’t aware of U+0022 + U+201C quotation marks will happily use U+0022 everywhere without ever considering the very slightly more aesthetically pleasing option. Independent validation is about redundancy, ideally you don’t want anything to reach that stage but people are really bad at testing for issues the’ve never considered. Usability in terms of colorblindness not just actual blindness yadda yadda good testing is really involved.
        necovek 1y ago
        There is fuzzing, FWIW. But the same also holds for people who are "independent testers" and unaware of the same issues. What I found is that software developers are bad at testing edge cases while they are in the creation mode (when they focus on happy path), but that good engineers switch to breaking mode once they try to cover things with sufficient tests. TDD also encourages breaking things first, but really, this is a mindset change that is usually skipped.
somewhereoutth 1y ago
Perhaps cheaper than upsetting your customers when they are the ones doing the testing? No machine testing can completely (or even mostly) validate a human interface.

sfn42 1y ago

Tools like playwright allow pretty nice web UI testing. You can make sure things are working properly and continue working as things change. Doesn't replace human testing but it does ease the human load and help catch problems and regressions before they get to the human testers.

Ygg2 1y ago
> Tools like playwright allow pretty nice web UI testing. Can they test for color blindness and myopia?
- dylan604 1y ago
  “Doesn't replace human testing” It’s like you stopped reading to try to score internet points or something. The answer to your question was one more sentence from where you stopped reading
  - Ygg2 1y ago
    The point of the question was to both see if the bar has changed and to address the overall article's point "Can we test it".
- sfn42 1y ago
  There are tools that can analyze a page for accessibility. I don't have experience with them but I wouldn't be surprised if you could include them in a CI/CD pipeline. I don't know whether they specifically control for color blindness and myopia but I wouldn't be surprised if they do. I know they check for contrast and stuff like that. You could also check things like colors etc using Playwright but I would say it's probably the wrong tool for that job. It's more about testing functionality - make sure a page has the right content and works correctly on a technical level. Without automated tools this type of thing can take a lot of time - in order to ensure quality you would basically have to click through the entire application for every release. Otherwise you might end up with some minor changes to one page breaking a different page and you'd never know until a tester checked it out or a user complained. With Playwright and similar tools you can catch regressions like this automatically.

__MatrixMan__ 1y ago

User acceptance testing is a good idea but doing it in cases where you can get away with cheaper testing is not.