3 ms·

Test suites can make you less careful because you start relying on the tests flagging any bugs. And then when the test suite does find a bug, you just hack the

by ssp 16y ago

Test suites can make you less careful because you start relying on the tests flagging any bugs. And then when the test suite does find a bug, you just hack the code until the test suite passes instead of fixing the underlying problem. This is the same problem as using microbenchmarks for performance work: you end up tuning to the benchmark instead of to the real world.

jerf 16y ago
By "you", I think you may mean "I". If ssp is hacking the test suite to bypass a failing test, ssp has the problem, not the test suite. jerf does not have much trouble with that, jerf has experienced "the one failing test that turns out to reveal a major underlying problem followed by a real fix that it would have taken him multiple customer-losing hard-to-reproduce bugs in field to learn about" multiple times. Not that this is perfect, I've got just such a bug out there right now that simply refuses to be reproduced by anyone once a developer is looking at it, but I sure have far fewer of those than I would without the tests.
- ssp 16y ago
  If ssp is hacking the test suite to bypass a failing test I'm talking about hacking the application to pass the test suite, not hacking the test suite. You can often "fix" a failing test by doing if (condition that failed) whack the application state so that the test suite will pass. without understanding what the actual bug were. And it's not always obvious to yourself that this is what you are doing. ssp has the problem, not the test suite Tuning to benchmarks is not some unique character flaw of mine. When you measure some aspect of people's behavior, they will optimize to that measurement. If the measurement is a boolean PASS/FAIL from the test suite, then they will optimize their behavior to get a PASS. But the actual desired outcome is not PASS, it is "bug free program".
  - lutorm 16y ago
    The answer to the problem of optimization by proxy is not no optimization at all.
    - deleted 16y ago
      [deleted]
  - jerf 16y ago
    Hack the app, hack the test suite, I meant either equally. Which I think if you read my post is pretty clear that the main dichotomy is between "hack" and "real fix" and where the hack goes hardly matters. Nevertheless, you are arguing that because people sometimes program to benchmarks, you are apparently better off without the benchmarks. I say that's nonsense. The solution is to use the benchmarks better. Are we programmers or automatons? (Or managers?) If you're going to be that defeatist about programming you're not going to be able to be a successful programmer under any circumstances, the entire field is a minefield of superficially-appealing optimization opportunities!
    - ssp 16y ago
      Nevertheless, you are arguing that because people sometimes program to benchmarks, you are apparently better off without the benchmarks. No, I am not.

by ssp 16y ago

jerf 16y ago

By "you", I think you may mean "I". If ssp is hacking the test suite to bypass a failing test, ssp has the problem, not the test suite. jerf does not have much trouble with that, jerf has experienced "the one failing test that turns out to reveal a major underlying problem followed by a real fix that it would have taken him multiple customer-losing hard-to-reproduce bugs in field to learn about" multiple times. Not that this is perfect, I've got just such a bug out there right now that simply refuses to be reproduced by anyone once a developer is looking at it, but I sure have far fewer of those than I would without the tests.

ssp 16y ago
If ssp is hacking the test suite to bypass a failing test I'm talking about hacking the application to pass the test suite, not hacking the test suite. You can often "fix" a failing test by doing if (condition that failed) whack the application state so that the test suite will pass. without understanding what the actual bug were. And it's not always obvious to yourself that this is what you are doing. ssp has the problem, not the test suite Tuning to benchmarks is not some unique character flaw of mine. When you measure some aspect of people's behavior, they will optimize to that measurement. If the measurement is a boolean PASS/FAIL from the test suite, then they will optimize their behavior to get a PASS. But the actual desired outcome is not PASS, it is "bug free program".
- lutorm 16y ago
  The answer to the problem of optimization by proxy is not no optimization at all.
  - deleted 16y ago
    [deleted]
- jerf 16y ago
  Hack the app, hack the test suite, I meant either equally. Which I think if you read my post is pretty clear that the main dichotomy is between "hack" and "real fix" and where the hack goes hardly matters. Nevertheless, you are arguing that because people sometimes program to benchmarks, you are apparently better off without the benchmarks. I say that's nonsense. The solution is to use the benchmarks better. Are we programmers or automatons? (Or managers?) If you're going to be that defeatist about programming you're not going to be able to be a successful programmer under any circumstances, the entire field is a minefield of superficially-appealing optimization opportunities!
  - ssp 16y ago
    Nevertheless, you are arguing that because people sometimes program to benchmarks, you are apparently better off without the benchmarks. No, I am not.