6 ms·

Can we test it? Yes, was can [video]

81 points by zdw 1y ago

webdevver 1y ago
i thought that's what customers were for?
somewhereoutth 1y ago
My feeling on testing: - If it is used by a machine, then it can be tested by a machine. - If it is used by a human, then it must be tested by a human.
- alex_smart 1y ago
  But testing by human is expensive
  - Anonbrit 1y ago
    and horribly unreliable even when done by competent and motivated humans, let alone most IT workers
    - karolinepauls 1y ago
      I've heard of good experiences using a 3rd party QA company (for frontend-heavy changes) and had pretty OK experiences doing it in-house with subject experts (though testing backend changes, so pretty much "either it works or it doesn't" in my case).
    - somewhereoutth 1y ago
      A good QA person/team will (be strongly incentivized to) make it their mission to find bugs and usability issues. Likely they will be domain experts, if only in the domain of the usage of your product. Worth their weight in gold, no machine can give you that.
  - Retric 1y ago
    Expensive isn’t unnecessary. Developer written tests can’t tell you if your UI is intuitive for novice users.
    - sfn42 1y ago
      They can however tell you whether the button/form/page/whatever is working and continues working.
      - dylan604 1y ago
        Do those developer created tests include weird and random text that users will unwittingly break your form with? More than once has a form been broken because devs didn’t test for emoji or other glyphs in the text. Just because a button click calls a form with as expected doesn’t not mean the form will behave as expected with unexpected user input
        sfn42 1y ago
        Playwright is essentially a programmable headless browser. It sends a request to a page, waits for the response to load and then for things like React rendering, secondary requests etc, then once the page is ready you can do whatever you want. You can interact with the DOM, fill in forms, scroll, click things, get a div and check its size and location, anything you want. Testing special symbols and such should be no problem at all, provided that you actually think to write that test.
        Retric 1y ago
        Which is can be used for a range of things but doesn’t remove the need for independent testing. To use one tiny concrete example. Someone who isn’t aware of U+0022 + U+201C quotation marks will happily use U+0022 everywhere without ever considering the very slightly more aesthetically pleasing option. Independent validation is about redundancy, ideally you don’t want anything to reach that stage but people are really bad at testing for issues the’ve never considered. Usability in terms of colorblindness not just actual blindness yadda yadda good testing is really involved.
  - somewhereoutth 1y ago
    Perhaps cheaper than upsetting your customers when they are the ones doing the testing? No machine testing can completely (or even mostly) validate a human interface.
- sfn42 1y ago
  Tools like playwright allow pretty nice web UI testing. You can make sure things are working properly and continue working as things change. Doesn't replace human testing but it does ease the human load and help catch problems and regressions before they get to the human testers.
  - Ygg2 1y ago
    > Tools like playwright allow pretty nice web UI testing. Can they test for color blindness and myopia?
    - dylan604 1y ago
      “Doesn't replace human testing” It’s like you stopped reading to try to score internet points or something. The answer to your question was one more sentence from where you stopped reading
      - Ygg2 1y ago
        The point of the question was to both see if the bar has changed and to address the overall article's point "Can we test it".
    - sfn42 1y ago
      There are tools that can analyze a page for accessibility. I don't have experience with them but I wouldn't be surprised if you could include them in a CI/CD pipeline. I don't know whether they specifically control for color blindness and myopia but I wouldn't be surprised if they do. I know they check for contrast and stuff like that. You could also check things like colors etc using Playwright but I would say it's probably the wrong tool for that job. It's more about testing functionality - make sure a page has the right content and works correctly on a technical level. Without automated tools this type of thing can take a lot of time - in order to ensure quality you would basically have to click through the entire application for every release. Otherwise you might end up with some minor changes to one page breaking a different page and you'd never know until a tester checked it out or a user complained. With Playwright and similar tools you can catch regressions like this automatically.
- __MatrixMan__ 1y ago
  User acceptance testing is a good idea but doing it in cases where you can get away with cheaper testing is not.
diggan 1y ago
Seems like blog spam, the actual content (presentation/talk) is at: https://www.youtube.com/watch?v=MqC3tudPH6w https://www.youtube.com/watch?v=MqC3tudPH6w
- tomhow 1y ago
  Thanks, we updated the URL from https://antithesis.com/blog/2025/bugbash_2025/mitchell_hashimoto/ https://antithesis.com/blog/2025/bugbash_2025/mitchell_hashi...
cloogshicer 1y ago
I think what people really mean when they say "This can't be tested" is: "The cost of writing these tests outweighs the benefit", which often is a valid argument, especially if you have to do major refactors that make the system overall more difficult to understand. I do not agree with test zealots that argue that a more testable system is always also easier to understand, my experience has been the opposite. Of course there are cases where this is still worth the trade-off, but it requires careful consideration.
- MoreQARespect 1y ago
  It's often shorthand for "this cant be unit tested" or "this isnt dependency injected" even though integration tests are perfectly capable of testing non-DI code. The author's claims that we should isolate code under test better and rely more on snapshot testing are spot on.
  - diggan 1y ago
    > rely more on snapshot testing are spot on Never quite liked "snapshot testing" which I think has a better name under "golden master testing" or similar anyways. Reason for the dislike, is that it's basically a codified "Trust me bro, it's correct" without actually making clear what you are asserting with that test. I haven't found any team that used snapshot testing and didn't also need to change the snapshots for every little change, which obviously defeats the purpose. The only things snapshot testing seems to be good for, is when you've written something and you know it'll never change again, for any reason. Beyond that, unit tests and functional/integration tests are much easier to structure in a way so you don't waste so much time reviewing changes.
    - MoreQARespect 1y ago
      >I haven't found any team that used snapshot testing and didn't also need to change the snapshots for every little change, which obviously defeats the purpose I dont see how this even defeats the point, let alone obviously. If a UI changes I appreciate being notified. If a REST API response changes I like to see a diff. If somebody changes some CSS and it changes 50 snapshots, it isnt a huge burden to approve them all and sometimes it highlights a bug.
aspenmayer 1y ago
When this happens, how do you determine who gets the karma? Is it right and just and logical for OP to get karma for submitting a URL that HN readers didn't visit after being updated by mods, or for OP to get karma previously for a URL that was deemed lacking with regards to the guidelines? It seems like they should get one or the other, but not both. Just some food for thought. The reason I mention it, is that a person who has been commented upon by me previously for using scripts submitted this before OP, and if precedent holds, they should get the karma, not OP. But they have been commented upon by mods for having used scripts, but somehow haven't been banned for doing so, because dang has supposedly interacted with them/spoken with them, as if that could justify botting. But I digress. To wit: https://news.ycombinator.com/item?id=44449650 https://news.ycombinator.com/item?id=44449650
- sebastianmestre 1y ago
  They're fake internet points, it's no big deal
  - aspenmayer 1y ago
    Then why don't you email mods to delete your account? Your post adds nothing to the discussion I made with my reply, so what are you even doing here?
    - thunderbong 1y ago
      Because it's possible they didn't know about it? Because it's possible they might post something of more value another time? Let's assume good faith.
      - aspenmayer 1y ago
        Who didn't know about what now? This is HN inside baseball. If you or they don't know, they should ask somebody or lurk more, to be blunt. To phrase this in a better way, they should post better if they want a better response.
- 1y ago
j_w 1y ago
My takeaway from this is that when you have a system or feature that "can't be tested" that you should try to isolate the "untestable" portions to increase what you can test. The "untestable" portions of a code base often gobble up perfectly testable functionality, growing the problem. Write interfaces for those portions so you can mock them.
- gblargg 1y ago
  This often comes naturally for portable code, where the OS-specific things are separated from the core logic that just processes data. His keyboard handling example illustrates this.
pinoy420 1y ago
[dead]
testthetest 1y ago
The ROI on unit tests, as well as the answer to "Can we test it?" is changing fast in the age of AI. 1. AI is making unit tests nearly free. It's a no-brainer to ask Copilot/Cursor/insert-your-tool-here to include tests with your code. The bonus is that it forces better habits like dependency injection just to make the AI's job possible. This craters the "cost" side of the equation for basic coverage. 2. At the same time, software is increasingly complex: a system of a frontend, backend, 3rd-party APIs, mobile clients, etc. A million passing unit tests and 100% test coverage mean nothing in a world when a tiny contract change breaks the whole app. In our experience the thing that gives us the most confidence is black-box, end-to-end testing that tests things exactly as a real user would see them.
- necovek 1y ago
  I've talked about this in a few places, but there is a simple way to get to higher testability and trust in the automated tests being done with little to no extra effort for complex systems — this slightly tunes how you define the testing pyramid, but it's important to always test exactly enough, or you hit maintenance burden that kills you down the line. All it takes is for a component developer to build a fully functioning component by having proper fakes for all the external dependencies. If this is religiously applied through the org, then all components in the org will have all of their real business logic and integration points sufficiently tested. Then, your end-to-end tests can work against a very complex system that is extremely fast, but which uses the actual business logic, other than the very extreme boundaries to external systems not under your control. However, AI is usually not much of a help here, because most software is not written in the way to easily construct either a fake or real production component from the same component — and AI is trained on existing source code.
- YZF 1y ago
  I haven't looked at much at the quality of unit tests generated by AI but I would imagine it would suffer from the same general problems as humans. That is that most of these unit tests would be bad tests. They would include assumptions about implementation and make changes to the code harder. Yes- black box and end-to-end testing. Better forced by the nature of the black box to test the behavior of the system. It's still possible to test the wrong things (behaviors you don't care about) but that tends to stand out more. Also still difficult to make these tests reliable.
devjab 1y ago
I don't think anyone ever regretted doing a test but runtime assertions are so much better because they deal with issues when they happen, rather than trying to predict potential failures. This was probably forgotten as interpreted languages became more popular, but hopefully we're going to see a return to less "developer focused" ways of dealing with errors, so that your pacemaker doesn't stop because someone forgot to catch the exception they didn't think could happen with their 100% coverage.
- YZF 1y ago
  Deal how? Do you want run time assertions in your ABS firmware or your aircraft landing software?
  - devjab 1y ago
    Well yes, considering NASA wanted it in theirs. There are a lot of advantages in immediately failing when your software does something you don't want it to do rather than trying to continue in a corrupted or invalid state. As for how you deal with it, it depends. Erlang has a lot of "self-healing" supervisor/actor models which are widely used, but I don't have a lot of experience with those. We use a form of recovery oriented computing in our solar plant components, though to be fair, we only use a very tiny part of it since we basically reboot to the last known valid state. Which isn't exactly 100% safety, but it's not like anyone dies if an inverter or datalogger goes offline for an hour. I'm curious to know how you would deal with errors in these sort of things without runtime assertions or similar. Maybe my knolwedge is outdated?
    - YZF 1y ago
      I haven't done firmware in more than a decade but back when I did asserting was a no-no. You can't crash. Let's say I have a machine and I'm in the process of moving some actuators crashing might mean I destroyed the machine. That can't be better then letting things that can keep going going even if there's some unexpected condition. Physical safety is typically there by design and doesn't rely on software (there are some exceptions/processes for safety critical software). So what to do: - Extensive use of things like state machines where you can convince yourself that you are handling all possible inputs at every possible state. - Clean/clear/simple code. - Code reviews. - Test the heck of your software before you ship it. Automated testing and manual testing. Never fails isn't possible. Mechanical systems also fail. Electronics also fail. Software can be pretty reliable and either way you can't really tell what to do if there's an unexpected condition. Expected errors should be handled (let's say you try to move an actuator and it doesn't move, or you have a position error on your servo). Firmware can be designed to not run out of memory (so that failure mode can be eliminated by design). What's to guarantee you won't be just rebooting in a loop if you reboot? That said, I guess if you have analyzed things well enough and that's your preferred recovery then fair enough. But how much state does software for an inverter have that can't just be handled by code?
borg16 1y ago
shout out to "A tribe called Quest" for the title (my guess)

81 points by zdw 1y ago

i thought that's what customers were for?

My feeling on testing: - If it is used by a machine, then it can be tested by a machine. - If it is used by a human, then it must be tested by a human.

alex_smart 1y ago
But testing by human is expensive
- Anonbrit 1y ago
  and horribly unreliable even when done by competent and motivated humans, let alone most IT workers
  - karolinepauls 1y ago
    I've heard of good experiences using a 3rd party QA company (for frontend-heavy changes) and had pretty OK experiences doing it in-house with subject experts (though testing backend changes, so pretty much "either it works or it doesn't" in my case).
  - somewhereoutth 1y ago
    A good QA person/team will (be strongly incentivized to) make it their mission to find bugs and usability issues. Likely they will be domain experts, if only in the domain of the usage of your product. Worth their weight in gold, no machine can give you that.
- Retric 1y ago
  Expensive isn’t unnecessary. Developer written tests can’t tell you if your UI is intuitive for novice users.
  - sfn42 1y ago
    They can however tell you whether the button/form/page/whatever is working and continues working.
    - dylan604 1y ago
      Do those developer created tests include weird and random text that users will unwittingly break your form with? More than once has a form been broken because devs didn’t test for emoji or other glyphs in the text. Just because a button click calls a form with as expected doesn’t not mean the form will behave as expected with unexpected user input
      - sfn42 1y ago
        Playwright is essentially a programmable headless browser. It sends a request to a page, waits for the response to load and then for things like React rendering, secondary requests etc, then once the page is ready you can do whatever you want. You can interact with the DOM, fill in forms, scroll, click things, get a div and check its size and location, anything you want. Testing special symbols and such should be no problem at all, provided that you actually think to write that test.
        Retric 1y ago
        Which is can be used for a range of things but doesn’t remove the need for independent testing. To use one tiny concrete example. Someone who isn’t aware of U+0022 + U+201C quotation marks will happily use U+0022 everywhere without ever considering the very slightly more aesthetically pleasing option. Independent validation is about redundancy, ideally you don’t want anything to reach that stage but people are really bad at testing for issues the’ve never considered. Usability in terms of colorblindness not just actual blindness yadda yadda good testing is really involved.
- somewhereoutth 1y ago
  Perhaps cheaper than upsetting your customers when they are the ones doing the testing? No machine testing can completely (or even mostly) validate a human interface.
sfn42 1y ago
Tools like playwright allow pretty nice web UI testing. You can make sure things are working properly and continue working as things change. Doesn't replace human testing but it does ease the human load and help catch problems and regressions before they get to the human testers.
- Ygg2 1y ago
  > Tools like playwright allow pretty nice web UI testing. Can they test for color blindness and myopia?
  - dylan604 1y ago
    “Doesn't replace human testing” It’s like you stopped reading to try to score internet points or something. The answer to your question was one more sentence from where you stopped reading
    - Ygg2 1y ago
      The point of the question was to both see if the bar has changed and to address the overall article's point "Can we test it".
  - sfn42 1y ago
    There are tools that can analyze a page for accessibility. I don't have experience with them but I wouldn't be surprised if you could include them in a CI/CD pipeline. I don't know whether they specifically control for color blindness and myopia but I wouldn't be surprised if they do. I know they check for contrast and stuff like that. You could also check things like colors etc using Playwright but I would say it's probably the wrong tool for that job. It's more about testing functionality - make sure a page has the right content and works correctly on a technical level. Without automated tools this type of thing can take a lot of time - in order to ensure quality you would basically have to click through the entire application for every release. Otherwise you might end up with some minor changes to one page breaking a different page and you'd never know until a tester checked it out or a user complained. With Playwright and similar tools you can catch regressions like this automatically.
__MatrixMan__ 1y ago
User acceptance testing is a good idea but doing it in cases where you can get away with cheaper testing is not.

diggan 1y ago

Seems like blog spam, the actual content (presentation/talk) is at: https://www.youtube.com/watch?v=MqC3tudPH6w https://www.youtube.com/watch?v=MqC3tudPH6w

tomhow 1y ago
Thanks, we updated the URL from https://antithesis.com/blog/2025/bugbash_2025/mitchell_hashimoto/ https://antithesis.com/blog/2025/bugbash_2025/mitchell_hashi...

cloogshicer 1y ago

I think what people really mean when they say "This can't be tested" is: "The cost of writing these tests outweighs the benefit", which often is a valid argument, especially if you have to do major refactors that make the system overall more difficult to understand. I do not agree with test zealots that argue that a more testable system is always also easier to understand, my experience has been the opposite. Of course there are cases where this is still worth the trade-off, but it requires careful consideration.

MoreQARespect 1y ago
It's often shorthand for "this cant be unit tested" or "this isnt dependency injected" even though integration tests are perfectly capable of testing non-DI code. The author's claims that we should isolate code under test better and rely more on snapshot testing are spot on.
- diggan 1y ago
  > rely more on snapshot testing are spot on Never quite liked "snapshot testing" which I think has a better name under "golden master testing" or similar anyways. Reason for the dislike, is that it's basically a codified "Trust me bro, it's correct" without actually making clear what you are asserting with that test. I haven't found any team that used snapshot testing and didn't also need to change the snapshots for every little change, which obviously defeats the purpose. The only things snapshot testing seems to be good for, is when you've written something and you know it'll never change again, for any reason. Beyond that, unit tests and functional/integration tests are much easier to structure in a way so you don't waste so much time reviewing changes.
  - MoreQARespect 1y ago
    >I haven't found any team that used snapshot testing and didn't also need to change the snapshots for every little change, which obviously defeats the purpose I dont see how this even defeats the point, let alone obviously. If a UI changes I appreciate being notified. If a REST API response changes I like to see a diff. If somebody changes some CSS and it changes 50 snapshots, it isnt a huge burden to approve them all and sometimes it highlights a bug.

aspenmayer 1y ago

When this happens, how do you determine who gets the karma? Is it right and just and logical for OP to get karma for submitting a URL that HN readers didn't visit after being updated by mods, or for OP to get karma previously for a URL that was deemed lacking with regards to the guidelines? It seems like they should get one or the other, but not both. Just some food for thought. The reason I mention it, is that a person who has been commented upon by me previously for using scripts submitted this before OP, and if precedent holds, they should get the karma, not OP. But they have been commented upon by mods for having used scripts, but somehow haven't been banned for doing so, because dang has supposedly interacted with them/spoken with them, as if that could justify botting. But I digress. To wit: https://news.ycombinator.com/item?id=44449650 https://news.ycombinator.com/item?id=44449650

sebastianmestre 1y ago
They're fake internet points, it's no big deal
- aspenmayer 1y ago
  Then why don't you email mods to delete your account? Your post adds nothing to the discussion I made with my reply, so what are you even doing here?
  - thunderbong 1y ago
    Because it's possible they didn't know about it? Because it's possible they might post something of more value another time? Let's assume good faith.
    - aspenmayer 1y ago
      Who didn't know about what now? This is HN inside baseball. If you or they don't know, they should ask somebody or lurk more, to be blunt. To phrase this in a better way, they should post better if they want a better response.
1y ago

j_w 1y ago

My takeaway from this is that when you have a system or feature that "can't be tested" that you should try to isolate the "untestable" portions to increase what you can test. The "untestable" portions of a code base often gobble up perfectly testable functionality, growing the problem. Write interfaces for those portions so you can mock them.

gblargg 1y ago
This often comes naturally for portable code, where the OS-specific things are separated from the core logic that just processes data. His keyboard handling example illustrates this.

pinoy420 1y ago

[dead]

testthetest 1y ago

The ROI on unit tests, as well as the answer to "Can we test it?" is changing fast in the age of AI. 1. AI is making unit tests nearly free. It's a no-brainer to ask Copilot/Cursor/insert-your-tool-here to include tests with your code. The bonus is that it forces better habits like dependency injection just to make the AI's job possible. This craters the "cost" side of the equation for basic coverage. 2. At the same time, software is increasingly complex: a system of a frontend, backend, 3rd-party APIs, mobile clients, etc. A million passing unit tests and 100% test coverage mean nothing in a world when a tiny contract change breaks the whole app. In our experience the thing that gives us the most confidence is black-box, end-to-end testing that tests things exactly as a real user would see them.

necovek 1y ago
I've talked about this in a few places, but there is a simple way to get to higher testability and trust in the automated tests being done with little to no extra effort for complex systems — this slightly tunes how you define the testing pyramid, but it's important to always test exactly enough, or you hit maintenance burden that kills you down the line. All it takes is for a component developer to build a fully functioning component by having proper fakes for all the external dependencies. If this is religiously applied through the org, then all components in the org will have all of their real business logic and integration points sufficiently tested. Then, your end-to-end tests can work against a very complex system that is extremely fast, but which uses the actual business logic, other than the very extreme boundaries to external systems not under your control. However, AI is usually not much of a help here, because most software is not written in the way to easily construct either a fake or real production component from the same component — and AI is trained on existing source code.
YZF 1y ago
I haven't looked at much at the quality of unit tests generated by AI but I would imagine it would suffer from the same general problems as humans. That is that most of these unit tests would be bad tests. They would include assumptions about implementation and make changes to the code harder. Yes- black box and end-to-end testing. Better forced by the nature of the black box to test the behavior of the system. It's still possible to test the wrong things (behaviors you don't care about) but that tends to stand out more. Also still difficult to make these tests reliable.

devjab 1y ago

I don't think anyone ever regretted doing a test but runtime assertions are so much better because they deal with issues when they happen, rather than trying to predict potential failures. This was probably forgotten as interpreted languages became more popular, but hopefully we're going to see a return to less "developer focused" ways of dealing with errors, so that your pacemaker doesn't stop because someone forgot to catch the exception they didn't think could happen with their 100% coverage.

YZF 1y ago
Deal how? Do you want run time assertions in your ABS firmware or your aircraft landing software?
- devjab 1y ago
  Well yes, considering NASA wanted it in theirs. There are a lot of advantages in immediately failing when your software does something you don't want it to do rather than trying to continue in a corrupted or invalid state. As for how you deal with it, it depends. Erlang has a lot of "self-healing" supervisor/actor models which are widely used, but I don't have a lot of experience with those. We use a form of recovery oriented computing in our solar plant components, though to be fair, we only use a very tiny part of it since we basically reboot to the last known valid state. Which isn't exactly 100% safety, but it's not like anyone dies if an inverter or datalogger goes offline for an hour. I'm curious to know how you would deal with errors in these sort of things without runtime assertions or similar. Maybe my knolwedge is outdated?
  - YZF 1y ago
    I haven't done firmware in more than a decade but back when I did asserting was a no-no. You can't crash. Let's say I have a machine and I'm in the process of moving some actuators crashing might mean I destroyed the machine. That can't be better then letting things that can keep going going even if there's some unexpected condition. Physical safety is typically there by design and doesn't rely on software (there are some exceptions/processes for safety critical software). So what to do: - Extensive use of things like state machines where you can convince yourself that you are handling all possible inputs at every possible state. - Clean/clear/simple code. - Code reviews. - Test the heck of your software before you ship it. Automated testing and manual testing. Never fails isn't possible. Mechanical systems also fail. Electronics also fail. Software can be pretty reliable and either way you can't really tell what to do if there's an unexpected condition. Expected errors should be handled (let's say you try to move an actuator and it doesn't move, or you have a position error on your servo). Firmware can be designed to not run out of memory (so that failure mode can be eliminated by design). What's to guarantee you won't be just rebooting in a loop if you reboot? That said, I guess if you have analyzed things well enough and that's your preferred recovery then fair enough. But how much state does software for an inverter have that can't just be handled by code?

borg16 1y ago

shout out to "A tribe called Quest" for the title (my guess)