4 ms·

My learning from test automation: * The primary goal of test automation is to prevent regression. A secondary goal can be performance tuning your product. * T

My learning from test automation: * The primary goal of test automation is to prevent regression. A secondary goal can be performance tuning your product. * Tests are tech debt, so don’t waste time with any kind of testing that doesn’t immediately save you time in the near term. * Don’t waste your energy testing code unless you have an extremely good reason. Test the product and let me the product prove the quality of your code. Code is better tested with various forms of static analysis. * The speed with which an entire test campaign executes determines, more than all other factors combined, when and who executes the tests. If the test campaign takes hours nobody will touch it. Too painful. If it takes 10-30 minutes only your QA will touch it. When it takes less than 30 seconds to execute against a major percentage of all your business cases everybody will execute it several times a day.

jeremyjh 4y ago
Tests are not tech debt. You could have bad, brittle tests that you could consider debt but just having tests isn’t debt. Debt implies there is something you could do about it in the future to pay it down, which isn’t the case for a good test suite.
- ronnier 4y ago
  It’s debt. When you can’t add new features quickly because you have nightmarish tests to fix and you spend more time on the tests than the product, I’d say it’s debt. Especially with the insane mocking setups.
  - jeremyjh 4y ago
    Yes, those are the bad tests I was referring to. NOT having tests greatly increases the debt burden of your production code because you cannot refactor with any confidence and so you simply won’t.
    - hbrn 4y ago
      > Yes, those are the bad tests I was referring to This is the No True Scotsman issue with testing. When it fails, you just disregard the failure as "bad tests". But any company that has anything that resembles a testing culture will have a good amount of those "bad tests". And this amount is way higher than people are willing to admit. > you cannot refactor with any confidence Anecdotally, I've had way more cases where I wouldn't refactor because too many "bad tests" were breaking, not because I lacked confidence due to lack of tests. There are many things beyond tests that allow you refactor with confidence: simple interfaces, clear dependency hierarchy, modular design, etc. They are way more important than tests. Tests are often a last resort when all of the above is a disaster. When you're at a place where you need tests to keep your software stable you are probably already fucked, you're just not willing to recognize it. You shouldn't have zero tests, but tests should be treated as debt. The fewer tests you need to keep your software stable, the better your architecture is. Huge number of tests in a codebase is typically a signal of shitty architecture that crumbles without those crutches.
  - marcosdumay 4y ago
    Tests can carry tech debit, just like any code. They certainly are not identified by it. Tests are one of the ways you have to ensure your code is correct. Consequently, they are business-oriented code that exist to support your program usage, and subject to its requirements. How much assurance you need is completely defined by those requirements. (But how you achieve that assurance isn't, and tests are only one of the possible tools for that.)
    - hbrn 4y ago
      They are still debt since they don't directly contribute to product value: you can delete all your tests and your software will keep functioning. It doesn't mean it's a debt worth taking, though IME most companies are either taking way too much or way to little. Not treating tests as debt typically leads to over-testing, and it is way worse than under-testing. Also, what you're talking about (business-oriented requirements) is more akin to higher level tests (integration/e2e), not unit tests.
      - jeremyjh 4y ago
        If you can simply delete it that’s not debt, it’s just cost. My bank doesn’t let me delete my loan obligation and still live in my house.
        hbrn 4y ago
        You can simply burn the bank, no? I've never seen a team where bad tests were simply deleted. They were always "fixed" instead.
      - Rexxar 4y ago
        You can also delete the source code after compiling it and your software will keep functioning. Does it mean that code don't directly contribute to product value ?
  - jlarocco 4y ago
    I think that's projection. Your tests being a nightmare doesn't imply my tests are a nightmare.
  - Too 4y ago
    NOT having tests is dept. When you can’t fearlessly add features quickly because you introduce regressions in another end of the product that you didn’t think of, and later have to spend all your time on firefighting because it got deployed to prod. If your mocks and tests are in the way when introducing features or refactoring, they are likely not on the right level. Too much unit testing of moving internals, rather than public apis, usually being one of the culprits.
commandlinefan 4y ago
* tests are a handy way to run one single function in isolation without having to spin up an instance and navigate the UI, too.

jeremyjh 4y ago

Tests are not tech debt. You could have bad, brittle tests that you could consider debt but just having tests isn’t debt. Debt implies there is something you could do about it in the future to pay it down, which isn’t the case for a good test suite.

ronnier 4y ago
It’s debt. When you can’t add new features quickly because you have nightmarish tests to fix and you spend more time on the tests than the product, I’d say it’s debt. Especially with the insane mocking setups.
- jeremyjh 4y ago
  Yes, those are the bad tests I was referring to. NOT having tests greatly increases the debt burden of your production code because you cannot refactor with any confidence and so you simply won’t.
  - hbrn 4y ago
    > Yes, those are the bad tests I was referring to This is the No True Scotsman issue with testing. When it fails, you just disregard the failure as "bad tests". But any company that has anything that resembles a testing culture will have a good amount of those "bad tests". And this amount is way higher than people are willing to admit. > you cannot refactor with any confidence Anecdotally, I've had way more cases where I wouldn't refactor because too many "bad tests" were breaking, not because I lacked confidence due to lack of tests. There are many things beyond tests that allow you refactor with confidence: simple interfaces, clear dependency hierarchy, modular design, etc. They are way more important than tests. Tests are often a last resort when all of the above is a disaster. When you're at a place where you need tests to keep your software stable you are probably already fucked, you're just not willing to recognize it. You shouldn't have zero tests, but tests should be treated as debt. The fewer tests you need to keep your software stable, the better your architecture is. Huge number of tests in a codebase is typically a signal of shitty architecture that crumbles without those crutches.
- marcosdumay 4y ago
  Tests can carry tech debit, just like any code. They certainly are not identified by it. Tests are one of the ways you have to ensure your code is correct. Consequently, they are business-oriented code that exist to support your program usage, and subject to its requirements. How much assurance you need is completely defined by those requirements. (But how you achieve that assurance isn't, and tests are only one of the possible tools for that.)
  - hbrn 4y ago
    They are still debt since they don't directly contribute to product value: you can delete all your tests and your software will keep functioning. It doesn't mean it's a debt worth taking, though IME most companies are either taking way too much or way to little. Not treating tests as debt typically leads to over-testing, and it is way worse than under-testing. Also, what you're talking about (business-oriented requirements) is more akin to higher level tests (integration/e2e), not unit tests.
    - jeremyjh 4y ago
      If you can simply delete it that’s not debt, it’s just cost. My bank doesn’t let me delete my loan obligation and still live in my house.
      - hbrn 4y ago
        You can simply burn the bank, no? I've never seen a team where bad tests were simply deleted. They were always "fixed" instead.
    - Rexxar 4y ago
      You can also delete the source code after compiling it and your software will keep functioning. Does it mean that code don't directly contribute to product value ?
- jlarocco 4y ago
  I think that's projection. Your tests being a nightmare doesn't imply my tests are a nightmare.
- Too 4y ago
  NOT having tests is dept. When you can’t fearlessly add features quickly because you introduce regressions in another end of the product that you didn’t think of, and later have to spend all your time on firefighting because it got deployed to prod. If your mocks and tests are in the way when introducing features or refactoring, they are likely not on the right level. Too much unit testing of moving internals, rather than public apis, usually being one of the culprits.

commandlinefan 4y ago

* tests are a handy way to run one single function in isolation without having to spin up an instance and navigate the UI, too.