4 ms·

Very nice. Anyone could point me to LLVM's code coverage and whether they use TDD? "Amount of unit tests is pretty small comparing to regression tests. One rea

by a_imho 11y ago

Very nice. Anyone could point me to LLVM's code coverage and whether they use TDD? "Amount of unit tests is pretty small comparing to regression tests. One reason behind that decision is that LLVM internals constantly change all the time. Supporting tests under such conditions is very time consuming." Isn't the whole promise of testing is easier maintenance in the long run? Also, regression tests seem to run much faster, is it because of the Google Test?

mining____ 11y ago
I suspect they mean that unit testing internal stuff that changes regularly means rewriting unit tests to use the new internal implementation details. In contrast, the regression tests don't have the dependency on implementation detail, just the interface.
pygy_ 11y ago
> Isn't the whole promise of testing is easier maintenance in the long run? It depends on the kind of maintenance. It makes it easier to add or refactor code without changing the semantics of what exists, but it makes it harder to modify the semantics. A test suite that's too tight can be an impediment to change.
wyldfire 11y ago
> Isn't the whole promise of testing is easier maintenance in the long run? Well, it's not as if the point is regarding testing vs not, it's regarding the balance of what kind of tests LLVM should have. LLVM and Clang have extraordinarily well-defined interfaces at their boundary (C/C++/LLVM IR|executables), so writing regression tests is bound to reap great value over time. Writing unit tests, less so. The benefit of unit tests over ones with larger scope is that it's often easier (or possible at all) to write tests that cover uncommon use cases/environments/errors. A less tangible benefit of unit tests is that it helps guide sufficient decomposition of the design-under-test. From what I've seen, LLVM doesn't need any help in this area, they're extraordinarily well-disciplined here.
jjoonathan 11y ago
Yes, but if you're not wise about how you test it's easy to not get the bang for your buck. Cargo cult testing is a very real problem. Example: I was extending some code the other day that had 3kloc of thin wrapper code and 30kloc of mocks and unit tests which effectively did nothing but ensure that the 3kloc passed things through unaltered. Meanwhile, the code is widely acknowledged to be frustrating to interface with because it uses a string-based API that isn't documented (and no, those unit tests won't tell you semantics), versioned, or maintained with an eye towards compatibility. Naturally, there are no integration tests with the layer above or below it, and those interfaces are where all the actual bugs happen. So the tests increased cost by a factor of 1000% and had 0 return. Also see: cargo cult comments. You know, these guys: /** * Gets the foo. */ Foo getFoo() {return foo;} Anyway, my personal rule of thumb is to start with integration tests and only drop to the granularity of unit tests if I've got a "logic hairball" on my hands where the interfaces are much simpler than the internal logic (parsing and algorithm implementations, usually). I'd be happy to hear other opinions though!
- lfowles 11y ago
  My experience with integration tests first is that a handful of use cases get stuffed into these test cases, but it's harder to figure out what you are actually testing. It's good to test that everything comes together and works in this one cherry picked case, but I'd rather be able to give the more granular guarantee that all of my components hold up their individual contracts.
  - ZenoArrow 11y ago
    Doesn't it make sense to focus on contracts at a higher level of abstraction though? Wouldn't it be better to put the contracts at the level of the user rather than setting contracts for most/all of the functions in your code? If the mocked user input resulted in the correct output and there were no other side effects, wouldn't that be sufficient in many cases?
    - lfowles 11y ago
ZenoArrow 11y ago
> "Isn't the whole promise of testing is easier maintenance in the long run?" Yes, but the structure of the tests matters. Generally speaking there are two major problem areas to avoid with unit tests: 1. Unit tests that are too close in design to the specific implementation. What this means is that instead of testing the desired behaviour in an abstract way, you mirror the design decisions made in the code being tested. When the code being tested changes, this can break the unit test, even if the new code has the desired behaviour. 2. Test-induced design damage. Unit testing relies on being able to isolate the code being tested. This can lead to an increase in code complexity, as well as a less flexible design. Both of these issues can be overcome, but it does mean that you have to consider what to test and how to test it, as badly applied tests can cause maintenance issues.

by a_imho 11y ago

mining____ 11y ago

I suspect they mean that unit testing internal stuff that changes regularly means rewriting unit tests to use the new internal implementation details. In contrast, the regression tests don't have the dependency on implementation detail, just the interface.

pygy_ 11y ago

> Isn't the whole promise of testing is easier maintenance in the long run? It depends on the kind of maintenance. It makes it easier to add or refactor code without changing the semantics of what exists, but it makes it harder to modify the semantics. A test suite that's too tight can be an impediment to change.

wyldfire 11y ago

> Isn't the whole promise of testing is easier maintenance in the long run? Well, it's not as if the point is regarding testing vs not, it's regarding the balance of what kind of tests LLVM should have. LLVM and Clang have extraordinarily well-defined interfaces at their boundary (C/C++/LLVM IR|executables), so writing regression tests is bound to reap great value over time. Writing unit tests, less so. The benefit of unit tests over ones with larger scope is that it's often easier (or possible at all) to write tests that cover uncommon use cases/environments/errors. A less tangible benefit of unit tests is that it helps guide sufficient decomposition of the design-under-test. From what I've seen, LLVM doesn't need any help in this area, they're extraordinarily well-disciplined here.

jjoonathan 11y ago

Yes, but if you're not wise about how you test it's easy to not get the bang for your buck. Cargo cult testing is a very real problem. Example: I was extending some code the other day that had 3kloc of thin wrapper code and 30kloc of mocks and unit tests which effectively did nothing but ensure that the 3kloc passed things through unaltered. Meanwhile, the code is widely acknowledged to be frustrating to interface with because it uses a string-based API that isn't documented (and no, those unit tests won't tell you semantics), versioned, or maintained with an eye towards compatibility. Naturally, there are no integration tests with the layer above or below it, and those interfaces are where all the actual bugs happen. So the tests increased cost by a factor of 1000% and had 0 return. Also see: cargo cult comments. You know, these guys: /** * Gets the foo. */ Foo getFoo() {return foo;} Anyway, my personal rule of thumb is to start with integration tests and only drop to the granularity of unit tests if I've got a "logic hairball" on my hands where the interfaces are much simpler than the internal logic (parsing and algorithm implementations, usually). I'd be happy to hear other opinions though!

lfowles 11y ago
My experience with integration tests first is that a handful of use cases get stuffed into these test cases, but it's harder to figure out what you are actually testing. It's good to test that everything comes together and works in this one cherry picked case, but I'd rather be able to give the more granular guarantee that all of my components hold up their individual contracts.
- ZenoArrow 11y ago
  Doesn't it make sense to focus on contracts at a higher level of abstraction though? Wouldn't it be better to put the contracts at the level of the user rather than setting contracts for most/all of the functions in your code? If the mocked user input resulted in the correct output and there were no other side effects, wouldn't that be sufficient in many cases?
  - lfowles 11y ago

ZenoArrow 11y ago

> "Isn't the whole promise of testing is easier maintenance in the long run?" Yes, but the structure of the tests matters. Generally speaking there are two major problem areas to avoid with unit tests: 1. Unit tests that are too close in design to the specific implementation. What this means is that instead of testing the desired behaviour in an abstract way, you mirror the design decisions made in the code being tested. When the code being tested changes, this can break the unit test, even if the new code has the desired behaviour. 2. Test-induced design damage. Unit testing relies on being able to isolate the code being tested. This can lead to an increase in code complexity, as well as a less flexible design. Both of these issues can be overcome, but it does mean that you have to consider what to test and how to test it, as badly applied tests can cause maintenance issues.