4 ms·

In the aerospace industry (commercial and defense are similar here because the software for commercial has to be written according to government rules, see FAA

by rplst8 11y ago

In the aerospace industry (commercial and defense are similar here because the software for commercial has to be written according to government rules, see FAA DO-178B) it's actually verboten. Testing independence is part of the flight certification of the software if I'm not mistaken.

thaumasiotes 11y ago
This has all the enforceability of prohibiting your workers from thinking about what they're working on, or practicing in their spare time. Tests you write for your own benefit are not a deliverable. They're part of your work process. As I said before, it's not ridiculous at all to require that deliverable tests be written by some other party than the developer whose code needs to pass them.
- EdHominem 11y ago
  Actually I think this is one of those things that sounds so obvious, and yet isn't right in practice. I'l roughly segment projects into those that can fail a bit and those that can't. Now obviously, for things that can't fail even a little you can't just take someone's word. But why can you take the word of two people, or ten? The failures (in programming and code review) are absolutely correlated. If one person fails at something another likely will fail in the same place or when reviewing it. When people die if there's a bug, or you're even losing a lot of money, you should not be trusting best-effort human anything. This is where imperative programming against a test suite breaks down and just can't cope. You need to switch out your internals for something provable in its domain. (eg, the math to land a space shuttle in a fixed-time infrastructure (ie before it lands itself)). And for everything else, it's a $/$ calculation. How much do you want to spend to have some unknowably smaller amount of risk? And usually the best value for the dollar is having the original engineering team work on the tests, with outside oversight and good metrics. If an integrated team is having problems getting full branch coverage (the only worthwhile coverage metric...) in a given method, they rewrite the method. External teams have to test what's given (or waste a ton of time in communication delays) and that usually ends up with suboptimal tests and suboptimal coverage. What you can't do though is simply ask a developer if their work is properly tested and trust their answer. You need real metrics and to know what they mean for you. (Like benchmarks.)
  - RyanZAG 11y ago
    The tests are written independently of the original code. If someone has made a logic error or misunderstood some part of the spec, there is a good chance that the independent test implementation is not going to have the identical error. It's definitely a higher chance of finding bugs than having 1 developer write both, as long as the testing is written independently based on an API.
    - EdHominem 11y ago
      There's a difference between code tests though and spec verification. Spec verification is (should be...) black-box integration testing. And yes, this is a good chance to get a second set of eyes on the spec to make sure nothing was missed in the implementation. (unit-tests can't catch missing code!) Code tests should be more white-box and should be measured against line/code coverage instead of spec coverage. These are what I think the original devs should write, and should be 95%+ of the total test volume. The problem with expecting testing to pick up spec errors is that it takes comes at the end of the design/build phase instead of the beginning when you can make changes easily.
      - thaumasiotes 11y ago
        > Code tests should be more white-box and should be measured against line/code coverage instead of spec coverage. These are what I think the original devs should write Nobody's stopping them.
      - RyanZAG 11y ago
        Well you're not going to pick up spec errors with same-developer unit tests, so either you have someone else write those spec tests or you wait until the customer picks them up on initial delivery / QA.

by rplst8 11y ago

thaumasiotes 11y ago

This has all the enforceability of prohibiting your workers from thinking about what they're working on, or practicing in their spare time. Tests you write for your own benefit are not a deliverable. They're part of your work process. As I said before, it's not ridiculous at all to require that deliverable tests be written by some other party than the developer whose code needs to pass them.

EdHominem 11y ago
Actually I think this is one of those things that sounds so obvious, and yet isn't right in practice. I'l roughly segment projects into those that can fail a bit and those that can't. Now obviously, for things that can't fail even a little you can't just take someone's word. But why can you take the word of two people, or ten? The failures (in programming and code review) are absolutely correlated. If one person fails at something another likely will fail in the same place or when reviewing it. When people die if there's a bug, or you're even losing a lot of money, you should not be trusting best-effort human anything. This is where imperative programming against a test suite breaks down and just can't cope. You need to switch out your internals for something provable in its domain. (eg, the math to land a space shuttle in a fixed-time infrastructure (ie before it lands itself)). And for everything else, it's a $/$ calculation. How much do you want to spend to have some unknowably smaller amount of risk? And usually the best value for the dollar is having the original engineering team work on the tests, with outside oversight and good metrics. If an integrated team is having problems getting full branch coverage (the only worthwhile coverage metric...) in a given method, they rewrite the method. External teams have to test what's given (or waste a ton of time in communication delays) and that usually ends up with suboptimal tests and suboptimal coverage. What you can't do though is simply ask a developer if their work is properly tested and trust their answer. You need real metrics and to know what they mean for you. (Like benchmarks.)
- RyanZAG 11y ago
  The tests are written independently of the original code. If someone has made a logic error or misunderstood some part of the spec, there is a good chance that the independent test implementation is not going to have the identical error. It's definitely a higher chance of finding bugs than having 1 developer write both, as long as the testing is written independently based on an API.
  - EdHominem 11y ago
    There's a difference between code tests though and spec verification. Spec verification is (should be...) black-box integration testing. And yes, this is a good chance to get a second set of eyes on the spec to make sure nothing was missed in the implementation. (unit-tests can't catch missing code!) Code tests should be more white-box and should be measured against line/code coverage instead of spec coverage. These are what I think the original devs should write, and should be 95%+ of the total test volume. The problem with expecting testing to pick up spec errors is that it takes comes at the end of the design/build phase instead of the beginning when you can make changes easily.
    - thaumasiotes 11y ago
      > Code tests should be more white-box and should be measured against line/code coverage instead of spec coverage. These are what I think the original devs should write Nobody's stopping them.
    - RyanZAG 11y ago
      Well you're not going to pick up spec errors with same-developer unit tests, so either you have someone else write those spec tests or you wait until the customer picks them up on initial delivery / QA.