3 ms·

Yet it's extremely popular to have huge, inefficient test suites running on continuous integration servers hundreds of times a day. It boggles the mind.

datavirtue 5y ago
This is a good thing if you consider the price of not having them. I have worked in shops without tests. It's a great way to hand out free money to unsuspecting customers.
- amw-zero 5y ago
  Yes I agree - I have been a testing fanatic for the better part of the last 10 years, after being absolutely paralyzed at a company without tests. But, after all this time, I believe their cost-to benefit-ratio is horrendous. It’s fairly common to hear of test suites with a 2:1 ratio of test to implementation lines. That would be fine if they didn’t immensely prevent refactoring and block merges / deployments. Contrast that with something like F*, where the specification and code are right alongside each other, and the implementation gets proven. They are reporting a 5:1 ratio of specification to implementation code lines, which is much better than only a few years ago. Soon I think we’ll be striking distance to largely get rid of huge test suites which are so commonplace today.
  - exdsq 5y ago
    The one thing is a spec doesn't mean the product is correct - you still need testing, just in a different way. It'll probably replace unit tests though.
    - logicchains 5y ago
      A spec can catch anything your tests can. And if your spec overlooks something, your tests will overlook it too.
      - exdsq 5y ago
        I've seen projects using formal methods including mathematical specifications miss things caught in end-to-end tests written by domain experts rather than formal method engineers. Perhaps you're right in theory but I've yet to see it in practice.
        amw-zero 5y ago
        This just means that you thought of something in the test suite that wasn't in the spec. You can underspecify something.
    - amw-zero 5y ago
      Correct, there’s still nothing stopping us from writing the wrong spec or writing an insufficient spec, but there’s really no solution to that problem anyway. And the same problem exists with tests today - you can have an insufficient amount of test cases, or you can write the wrong cases because you misunderstand the requirements. The difference being, when you realize your tests are wrong, what do you do? Change and add more test cases. You may also need to remove some, because they don’t make sense anymore. Vs., updating logic in a specification, and that’s it.
  - satisfice 5y ago
    I’m not sure what your point is about ratios. I don’t think they mean anything, strictly speaking. Perhaps they are related to some special knowledge you have that is not conveyed by the numbers. Testing is a social act. Testing itself cannot be obsolete until complexity and misunderstanding is obsolete. This will happen never. Perhaps certain kinds of shallow automated checking will become obsolete over time.

datavirtue 5y ago

This is a good thing if you consider the price of not having them. I have worked in shops without tests. It's a great way to hand out free money to unsuspecting customers.

amw-zero 5y ago
Yes I agree - I have been a testing fanatic for the better part of the last 10 years, after being absolutely paralyzed at a company without tests. But, after all this time, I believe their cost-to benefit-ratio is horrendous. It’s fairly common to hear of test suites with a 2:1 ratio of test to implementation lines. That would be fine if they didn’t immensely prevent refactoring and block merges / deployments. Contrast that with something like F*, where the specification and code are right alongside each other, and the implementation gets proven. They are reporting a 5:1 ratio of specification to implementation code lines, which is much better than only a few years ago. Soon I think we’ll be striking distance to largely get rid of huge test suites which are so commonplace today.
- exdsq 5y ago
  The one thing is a spec doesn't mean the product is correct - you still need testing, just in a different way. It'll probably replace unit tests though.
  - logicchains 5y ago
    A spec can catch anything your tests can. And if your spec overlooks something, your tests will overlook it too.
    - exdsq 5y ago
      I've seen projects using formal methods including mathematical specifications miss things caught in end-to-end tests written by domain experts rather than formal method engineers. Perhaps you're right in theory but I've yet to see it in practice.
      - amw-zero 5y ago
        This just means that you thought of something in the test suite that wasn't in the spec. You can underspecify something.
  - amw-zero 5y ago
    Correct, there’s still nothing stopping us from writing the wrong spec or writing an insufficient spec, but there’s really no solution to that problem anyway. And the same problem exists with tests today - you can have an insufficient amount of test cases, or you can write the wrong cases because you misunderstand the requirements. The difference being, when you realize your tests are wrong, what do you do? Change and add more test cases. You may also need to remove some, because they don’t make sense anymore. Vs., updating logic in a specification, and that’s it.
- satisfice 5y ago
  I’m not sure what your point is about ratios. I don’t think they mean anything, strictly speaking. Perhaps they are related to some special knowledge you have that is not conveyed by the numbers. Testing is a social act. Testing itself cannot be obsolete until complexity and misunderstanding is obsolete. This will happen never. Perhaps certain kinds of shallow automated checking will become obsolete over time.